← Devlog
devonboardscannertestingprocess

A hundred green tests, zero runtimes detected

The DevOnboard scanner's first slice shipped with 100 passing tests. Then it scanned DevOnboard's own repo and detected zero runtimes.

Three weeks of work since the last entry, and the shape of it was: async job infrastructure merged, the scanner re-sliced before any code got written, and the first slice of the scanner landed. Along the way the lesson from my last post got tested twice, and both times it held up better than I wanted it to.

The async job layer merged first. Scanning a repo is a multi-step, long-running operation, so it needed real job infrastructure rather than a request that blocks: step-by-step progress, per-step error isolation, and a status the frontend can poll. It merged with 72 backend and 56 frontend tests green.

But it merged after a manual walkthrough, not when the tests passed. That walkthrough found four bugs, and the ugliest was architectural: the job runner was sharing a single database session across every step, which is exactly the thing the async session pattern exists to prevent. Also missing was any way to recall scan state when you revisited a repo, so a completed scan looked like it had never happened, and a sidebar icon that never once reflected scan status. Tests verify the pieces. None of those four are piece-level failures.

Then I re-sliced the scanner before writing a line of it. The original plan was one “build the scanner” group, which is the kind of scope that sounds fine in a planning doc and becomes a three-week branch in practice. I cut it into a scaffold-first Slice 0 (rate-limit gate, repo-tree fetch, coordinator storing a result, per-file error isolation, a partial-result flag, and exactly one detector wired end to end to prove the loop), then one thin slice per detector after that. The point of Slice 0 was never the detector. It was proving the whole path works before anything gets built on top of it.

Slice 0 landed with 100 passing tests. Then I ran it against three real repos.

Two worked. This site’s repo returned node >=22.12.0. An old C# project of mine returned dotnet netcoreapp2.2 from a .csproj sitting three directories deep, which is precisely why I picked it, since it proves the scanner reads paths out of the actual repo tree instead of guessing.

The third repo was DevOnboard itself, and it returned nothing at all.

The scanner was matching every source file as a literal path at the repo root, so backend/pyproject.toml never matched pyproject.toml, and frontend/package.json never matched package.json. A repo that is plainly Python and Node detected zero runtimes. Worse, it did it silently, because a path that isn’t there is a skip, not a warning. Nothing failed. It just found nothing and reported success.

The part that stung: my own architecture decision record uses “Node + Python in a monorepo” as its illustrative multi-runtime example. The spec explicitly described the case the implementation could not handle, and a hundred tests written against that implementation all agreed it was fine.

The fix was to resolve source filenames by basename anywhere in the tree, with two constraints that mattered more than the fix itself. Source priority still beats depth, because the decision record orders sources, not locations, and making depth the primary key would have quietly reinterpreted the spec. And only one path gets fetched per filename, the shallowest match, because resolving every match turns a bounded scan into one that scales with repo size. A monorepo with 200 package.json files would become 200 API calls, and rate limiting is the stated top risk for this phase.

Re-scanning the real repo confirmed it: an empty result became python >=3.12, read from backend/pyproject.toml.

The last decision of the stretch was to stop. I had one 90-minute block left before leaving town and a stretch goal queued up: a viewer for scan results, so verification doesn’t require me to open psql and read JSON by hand, which is genuinely how I verified all of the above. It’s the right next thing to build. I called it a no-go anyway. A half-finished feature sitting on a branch for a month is worse than not starting it, and an unresolved go/no-go quietly taxes attention I needed elsewhere. Slice 0 was designed to be a clean stopping point, so I let it be one.

Outside the editor

The other reason the calendar mattered: three weeks ago I ran the Teton Crest Trail as a single-day traverse, 40 miles and 9,101 feet of climbing, in 12:24. It’s the biggest day I’ve put together in the mountains, and it went the way long days usually go: comfortable early, long stretches without seeing anyone, views that justify the whole thing, and eventually embracing the suck at 30 miles down with 10 to go and climbing still ahead. But like every long run, the gratitude I feel for my body and the pride at the end trump all of it.

That’s become the training block for The Rutt 50k in September, 11,105 feet of gain over 31 miles, so the volume stays high through August. I’ve noticed the two halves of this log rhyme more than I expected them to. You can plan a route or a slice carefully, and the plan is still worth making, but nothing tells you what you actually built until you go run it.

Next up when I’m back: the scan result viewer, then the detectors one thin slice at a time.

For now I’m pausing the work. I’m heading back to the East Coast to spend a month with my family, and I’d rather be present with them than half-present with a branch open. The repo will be exactly where I left it, which was the whole point of stopping where I did.