Editorial writing · not a catalog record
Send someone who wasn't there
Code review usually finds style. The reviews that found real bugs in this catalog had one thing in common, and it was not being clever.
The review that mattered took ten minutes
I built a routine last week that reads a queue of candidate projects, gathers evidence about each, and sorts them for a human to judge. Fifteen tasks. Four batches. Every batch reviewed before the next began. Two hundred and thirty-five tests, all passing.
Then a final reviewer looked at the whole branch and reported that the routine could not run. Not "had a bug." Could not run, at all, ever, against the actual data.
Twelve of the fifty-five real candidates have no GitHub repository — Obsidian, NotebookLM, Amazon Q Developer, Jules — and the harness treated a missing repository as a fatal error. Eight of them fell inside the first batch it would process.
I want to be precise about how that reviewer found it, because the method is the whole point of this post. It did not reason about the code. It loaded directory/candidates.json, ran my selection function against the real fifty-five records with a stub that returned perfect responses, and pasted the output:
error: https://docs.aws.amazon.com/.../what-is.html:
['candidate has no GitHub repository']
run_build exit code with a perfectly healthy GitHub: 1It went and looked. I had written the code, written its tests, and reviewed it four times, and I had never once pointed it at the queue it existed to read. My fixtures all had repositories, because I wrote the fixtures, and I imagined the data instead of opening it.
Every review that worked ran something
That is the pattern, and it held across the week without exception.
Reviews that read a diff and reasoned about it found real things — a magic number, a missing guard, a docstring that promised a behaviour the function did not have. Useful, cheap, worth doing.
Reviews that executed something found the bugs that would have shipped.
Asked whether a guard could be defeated, one reviewer did not answer "it looks correct." It built twenty adversarial mutations against the real catalog — changed a confidence score, deleted a record, added one, renamed a repository, mutated an existing block, nulled fields that were not permitted to be null — ran each through the guard, and reported which were rejected and which were accepted. It could not break it. That sentence means something. "It looks correct" does not.
The difference is not diligence. It is that a claim about behaviour can only be settled by producing the behaviour.
Refactoring 796 lines without hoping
The catalog's validator had grown into a single function of 796 lines — three quarters of its file — enforcing every editorial invariant in the project. It needed decomposing, and decomposing the thing that guards everything is exactly where a quiet regression does the most damage.
The tests would not have caught a subtle change. They assert on error message substrings, so a rule that silently stopped firing, or fired in a different order, could pass.
So the refactor was verified differently. I kept a copy of the original implementation, then generated forty-two catalogs: the real one, plus forty-one mutations each designed to trip a different rule — a taxonomy group that is not a list, a project whose score does not match its weighted dimensions, evidence citing a blob SHA that does not match its immutable URL, a candidate already curated, a web copy out of sync. Both implementations ran against all forty-two. The harness compared the complete result: every error string, in order, and any exception raised.
42 catalogs compared, 0 divergence(s)Error ordering matters more than it sounds. It is the cheapest available proof that the rules still run in the same sequence, which is the thing a decomposition is most likely to disturb and the thing no test asserts.
One of the forty-two revealed something else. Both implementations crashed identically on a malformed record — a non-object in projects.json was reported and then killed the run before any error could be returned. A pre-existing bug, invisible to the test suite, surfaced by a harness built for an unrelated purpose. Fixing it later produced exactly one divergence out of forty-two, which is what a deliberate behaviour change should look like.
Proving a rewrite changed nothing visible
The same problem, in a different shape: the web application had four near-identical rendering paths, one per collection. Collapsing them into one engine plus four descriptors is the obvious cleanup and a genuine risk, because rendering has no unit tests worth the name — it is DOM, and the tests are end-to-end.
So before touching it, a browser harness captured every rendered surface of the live page: all five grids, all twenty-five filter option lists, and all 269 record dialogs, opened one by one. About 1.4 MB of HTML. Then the refactor. Then the same capture again.
299 surfaces compared — BYTE-IDENTICALAn intermediate version was not byte-identical. It reported 269 whitespace-only differences and zero real ones, because moving template literals into a method had shifted their indentation, and that indentation is inside the emitted HTML. Whitespace between block elements is almost never visible. Almost. Rather than reason about which of 269 cases might be the exception, I restructured so the templates kept their original indentation, and got the stronger guarantee for free.
That is the useful instinct: when you can cheaply upgrade "probably fine" to "provably identical," do it, because "probably fine" is where the surprises live.
The fix that made it worse
Here is the part I would most like to skip.
The final review of that routine returned sixteen findings. I fixed all sixteen in one pass, and a re-review confirmed every one addressed. It also found that my fix had introduced a new defect worse than several it repaired.
One finding was that the routine re-verified every citation it had ever written, so a README edited weeks later would deadlock the run permanently. A real problem. My fix scoped verification to blocks written today, using a date field — which the agent being verified writes itself, and which nothing pins.
A block dated yesterday, citing https://example.invalid/anything with a content hash of sixty-four zeros, passed every guard with the network never touched. I had turned the fabrication guard into something the thing it guards could switch off, and I did it while fixing sixteen other problems, feeling productive.
Two lessons, and the second is the expensive one. A fix wave needs its own review, always. And the moment you scope a security check by a value the untrusted party controls, you have not scoped it — you have removed it. The repair was to scope by comparison against the reference branch instead: decided by something the agent cannot write.
The uncomfortable arithmetic
Of the findings that mattered across that routine, four came from code that I had specified. Not implementation slips — my design, transcribed faithfully:
- A rule forbidding an agent from naming a classification, written case-sensitively against lowercase identifiers, so
Coding_Agentwalked through it. - A tolerance meant to let a first run proceed, written so it also swallowed a genuine failure and continued against a worktree that did not exist.
- Three git commands whose exit codes went unchecked, so a failed branch switch let a commit land unreferenced while the run reported success.
- A guard reading only the working tree, so an agent that committed its out-of-scope edit left nothing for it to see.
I reviewed that plan before handing it over. I found nothing. I am not able to review my own design, and neither, I suspect, is anyone — the same model of the problem that produced the gap is the one you use to look for it.
This is the entire argument for sending someone who wasn't there.
What made the reviews work
Four things, none of them subtle:
- No shared context. Reviewers got the diff, the requirements, and the constraints — never my reasoning, never my summary of what I had done. A reviewer who inherits your framing inherits your blind spot.
- Adversarial framing, explicitly. Not "check this over." Try to defeat this guard. Can a fabricated citation still pass? What input makes this abort? The twenty-mutation review happened because someone was asked to break something, not to inspect it.
- Never pre-judging a finding. The instruction that would have destroyed this is "don't worry about X" — because X is where the bug is. When I disagreed with a finding, I ruled on it in writing and recorded why, so the decision was auditable rather than invisible.
- Scoped re-reviews with a cap. Each fix round is re-reviewed against only the fix, so it verifies rather than wanders. Five rounds maximum, then the remaining findings get adjudicated in writing instead of looping forever.
It is worth saying plainly: this cost more than the implementation. Reviewing the routine consumed more effort than building it. That ratio sounds wrong until you weigh it against shipping a fabrication guard that a date field could disable, into a catalog whose only real asset is that its evidence can be trusted.
The short version
If you take one thing: the reviews that found real bugs were the ones that ran something against real data. Not more careful reading. Not a bigger model. Someone who wasn't there, told to break it, who opened the actual file.
I now know the specific feeling of having reviewed my own work thoroughly and found nothing, and I have learned to treat that feeling as no evidence at all.