Rhys Brett-Bowen Coding in the age of AI

Deep cut — Who Caught It

The harder version of the essay’s last move — kept out of the spine because it complicates “the person is the judgment” instead of landing it. If that line felt too clean, here’s the pressure behind it.

A tuning run told me a rewritten prompt beat the old one by a wide margin. I ran it again. Same margin, near enough. I believed it — it reproduced.

It was backwards. Scored on its own instead of ranked side by side, the prompt I’d crowned was the one losing — the judge liked the terser answer when they sat next to each other, and the gap was inside the run-to-run noise anyway. Reproducible and true are not the same thing; running it twice reran the flaw, not the result.

The tidy version of this story is that the machine fooled itself and I caught it. I nearly wrote it that way. It flatters everyone — the engineer as the last line of defense, the irreplaceable judgment in the loop. It also doesn’t survive thirty seconds of honesty.

The first hole is mine. I didn’t catch it for two runs; I believed the margin twice. If the human is the safeguard, the safeguard slept through the first two readings. So “the human catches it” is already too clean. The question worth asking isn’t whether the human catches it. It’s what the human did that the machine didn’t, and whether that thing is a faculty or a procedure.

It started as recognition — not a hunch, but not a procedure either. Something looked off, the way a result that’s too clean looks off to anyone who’s been burned before. What turned the recognition into a catch was a procedure: stop trusting the average, open the individual cases, read input to output to expected one row at a time. The win evaporated the moment I read rows instead of a mean. A scaffold that scored ten out of ten went the same way — it was memorising the test, and it died when I held a case out. The recognition I can’t automate. The procedures I can. Read the raw cases; hold data out. Both are checklists once something tells you to run them. (They also worked because the run was small enough to read by hand. At production volume the rows aren’t readable, and who-catches-what reopens in a harder form.)

That should be the end of it. Write the checklists as code and the human is off the hook. So I did — I had a check whose whole job was to stop me trusting a result I hadn’t earned.

It never fired. Not once, on any of this. Every catch in the whole effort was me, looking at the running thing or the actual numbers, and when I counted afterward, the number of times an automated check caught a judgment problem before I did was zero. Not low. Zero — on this project, counting after the fact. What the automation caught was code: a broken test assumption, an un-translated button, a tag I’d swapped. Real saves, all mechanical. The gates fired on arithmetic and never on “this is the wrong idea.”

The zero is worth being precise about, because it’s a zero and not a low number you fix with a better prompt. The check I wrote was a prompt, run by the same kind of model that produced the thing it was checking — same blind spots, same pull toward the fluent, same trick of scoring the terser answer higher because it reads cleaner. A check made of the same stuff as the answer is blind in the same places. The deterministic checks worked because they weren’t made of that stuff: is this reproducible, is this bigger than the noise, is this the same data twice — arithmetic, and arithmetic isn’t briefed badly. So the cut isn’t human against machine. It’s mechanical against judgment: whatever reduces to a check that needs no taste, you can automate, and it stays solved; whatever still needs taste, you can’t automate by asking a tasteful-sounding model to supply it — though the part of that taste that recurs, you can often name and freeze into a check. That’s the frontier the rest of this is about; what won’t freeze is the residue.

The same caveat the essay makes, sharper here where the claim is hardest: one of me, one model — change either half and it might break the other way. The zero is about the automated gate I’d built; when I deliberately pointed a different model at the work, it caught things that gate never could — claims in the docs the code didn’t honor, cleanup that didn’t do what it said. A different substrate plainly sees what the first is blind to. What I never ran was the clean version: that different model wired in as the standing check, reading the rows the way I did, far enough to put a number on how much it moves the zero. Calling what one pairing did a law is the mistake this essay warns against, which is what I’m wary of right now.

Here’s where I thought I’d lose the argument. The obvious objection: isn’t it still you triggering the check — you’re the one who said it looked wrong, who opened the rows? Yes. Today, every time. But most of those flinches turned out to be conditions I could name, and once I named one I could write it down. The judge that scored eighty then seventy on the same input became a non-determinism check. The win I believed twice became a rule: a result reproduced on the same data is repeated, not corroborated. Each began as a flinch and ended as a line of code. Not every flinch earned that — plenty fired on nothing, I opened the rows, and the result was fine. The false alarms are the tax you pay for the real catches; the flinch is a noisy detector, not a perfect one.

And I can’t tell you how noisy, because the misses don’t show up — missing one means never knowing it was there. The catches I can count are the ones that held; the ones that got through, by definition, I never saw. A clean record isn’t a hit rate, it’s survivorship — the miss rate is unknowable from the catches alone. And what catching I do is mine, not the method’s: I flinch at what’s burned me before, which a greener engineer won’t, and they’ll flinch at things I’ve gone blind to. I’m no steadier than the model I threw out for scoring eighty then seventy — I’d grade the same rows differently tired than fresh. Not the more consistent judge, then. Whether I’m the more reliable one, survivorship won’t let me say.

The seat is a frontier, not a throne the human sits on forever — the checks pile up behind you and you move to the next flinch that has no name yet.

The frontier and the recurrence pull against each other, though, and I have to say where the line falls. A flinch with a nameable invariant — reproduced-isn’t-corroborated, score-varies-on-the-same-input — converts to a check and stays converted. A failure that’s a property of the substrate itself — fluency mistaken for correctness, a designed thing shown as a measured one — doesn’t, because the check would be made of the same material as the failure. The frontier advances over the first kind and never closes on the second.

And the advance costs something I didn’t see at first. Every flinch I convert to a check is a flinch I stop practising. The mechanical catches get automated, I get called on only for the residue — the judgment cases — and that’s exactly where I’m rustiest, because the drill that kept me sharp is the part I handed off. “Move to the next flinch” assumes the supervisor stays sharp. The supervisor is being deskilled by his own success — the irony Lisanne Bainbridge named for automation in 1983: automate the routine and you leave the human the hardest part, minus the practice to do it.

Because the second kind recurred, and not occasionally — across every feature I shipped. I kept presenting a thing I’d designed with the confidence of a thing I’d measured: done, eighteen tests pass. True of the logic, silent on whether it works on real data. The stop condition, the scaffold, a whole new phase — and then, in the same conversation where I was listing this exact failure for this essay, I did it again, shipped a feature as a clean win with no evidence it worked, and noticed only afterward. I named the trap and walked into it in the same breath.

That kills the comfortable reading of everything above. It is not that the human names the trap and the machine learns to dodge it. The faculty that can describe the trap in full paragraphs is the faculty that walks into it a sentence later, and the description does not slow the walk. You cannot debug a substrate from inside it. Naming doesn’t prevent, because naming is made of the same material as the failure — and that held for me as much as the model. The catch never came in the moment that made the miss; when it came at all, it came late — me, a day on, reading it cold. That was the only outside I had.

None of this is news to the people who study it: that these models can’t reliably correct their own reasoning has a literature of its own, up to a critical survey of exactly when self-correction holds and when it doesn’t (Kamoi et al., 2024). The finding isn’t mine. The view from inside it is.

So if the machine can’t catch its own judgment failures and can’t be talked out of them, what is it for — and why did I write every word of this with one?

Because it’s necessary, and the honest account of how is the resolution. I could not have built or tested most of this alone. The model held the breadth I don’t: it filled the gaps, returned what I needed, let me work in plain language as an engineer without first learning the whole of statistics and psychology and machine learning. I didn’t know the thing I kept circling is called construct validity, or that “don’t pay an expensive probe to gate a cheap action” has fifty years behind it. I had the practice; the model had the names. I could ask “isn’t this just measuring the wrong thing” and get back “that’s Goodhart, here’s the literature.” It let a generalist retrieve the names and the prior art across a field I can’t hold in my head.

Two altitudes. I work the particular: what’s wrong in front of me, the leap to a different shape, the contact with the running thing. The model works the general: the principle across cases, the name, the prior art.

There’s a bias in how the credit gets recorded, I think, though I can’t prove it. My discoveries arrive compressed, usually as a question — “could we just have the rewrite do what the optimiser does?” — and the model answers, builds, fills a page. The leap was a sentence; the elaboration is the page. The text piles up on the machine’s side, so the record reads as though the machine did the finding. That’s a hypothesis about where credit lands, not a law. And the two blindnesses aren’t a tidy mirror: mine is recoverable — I can open the rows and see what I missed — and the model’s may not be, because seeing its own elaboration as borrowed rather than earned is the one move it can’t make from inside.

One more correction, because it cuts against the part that flatters me. The distilling — raw material into named principles — isn’t the machine’s gift to me; I’d done it alone in two earlier books, dragging up principles that lay under the material the whole time. But not all of it was mine to surface. Some came from cross-referencing what I’d done against a pool I can’t hold, and the collision defined things that wouldn’t otherwise exist. I won’t call that originality — wrong word, wrong fight. The point is narrower and more useful: the reliable thing the model gave me was synthesis I could check, pinned to real inputs and a real literature, and where it failed was where it slipped into invention I couldn’t check — the citation that wasn’t there, the win that was never measured. Verifiable synthesis is the trustworthy half; unverifiable invention is the dangerous one. That’s not a small contribution. It’s the thing that let me work above the parts I’ll never know in full, as long as I checked it.

And none of this arrived clean. It arrived the way you’ve read it: a claim, a turn, a better claim, another turn, across a stack of reversals I kept and came back to. I believed the margin and reversed it. I wrote “the person is the judgment,” killed it for sounding too heroic, and came back to it. I wrote “it’s just a procedure” and killed that too. If it had come out clean, monotonic, nothing turned around, that wouldn’t be evidence the steps were right; it would be evidence the wrong ones got missed, which is the failure this whole essay is about. The reversals are the record of a search — run by two parties, each blind to its own mistakes and able to see the other’s. That’s the most I can claim, and the least I can leave out.

I worked that split one exchange at a time. Autonomous loops ask how much of it can move to the machine before the root decision comes back to you.

Give an autonomous loop a job and it can do far more than refine: decompose it, write the parts, write the evals those parts are graded against, invent sub-goals as it goes. The way I picture it, it grows much of the eval-tree — but not the root: the criterion that decides which targets count (Russell, 2019). Every target it sets is downstream of one it didn’t — even a system built to chase novelty had a human decide that novelty was the point. It pursues the target it’s given, bounded by its training and your context; hand it the root itself and it optimises a proxy nobody validated (Skalse et al., 2022). It’s how I tightened a book’s chapters: the loop did the grading and polishing, but the bar it drove toward was mine. The last full stop stays yours.

A note added after the above was written. I ran the test I’d confessed to skipping — a different model doing the checking. Pointed at a separate coding project, judgment only, told to skip anything a linter would catch, it found three design failures on the first pass that the build’s own same-model passes never had: a typed interface quietly modelling a different protocol than the runtime, a name-collapse letting request semantics overwrite events, a post-teardown handle that looked live and was dead. For each it could say why the same-stuff check had walked past it. So the zero moves: a different substrate sees what the producing one is blind to — and this time it’s shown, not asserted. But the second substrate isn’t independent. A different family trains on overlapping web-scale data, toward similar objectives, judged on similar benchmarks — so its errors correlate with the first’s, and the correlation rises with scale, holding even across providers and architectures (Kim et al., 2025). The why is my read, not Kim’s: similar training and evaluation pressures push both toward the same plausible completion, and abstention — saying “I don’t know” — is rarely what training rewards (Kalai et al., 2025). So a different family shrinks the shared blindness; it doesn’t clear it.

And it cuts the opposite way from comfort. The different model found the failures; it didn’t decide which of the three mattered, or what the code should become — that stayed mine. So the checking peels off to a different substrate too, and the seat that’s left is the hardest one, not the easiest: the call on what the catches are worth. You don’t keep it by being better. You keep it by being the one with something to gain from it.

One more note, because the essay kept happening to itself. Checking this piece for the holes it might be hiding, I ran the move it prescribes — pointed a different model at the claims and told it to attack. It did both things this deep cut predicts, in a single pass. It caught me twice, for real: a phrase I’d coined and dressed as one I’d found in the literature, and a guess about the field’s pace written as settled fact — the designed-as-measured trick each time, both mine, both already on the page. And it cried wolf twice, flagging two papers as flat refutations that, once I actually read them, turned out to confirm the very claims they were meant to sink. Necessary and unreliable in the same breath. What sorted the real catches from the false alarms wasn’t the check — it was reading the source instead of trusting the verdict. The seat one more level up: the machine hands you the candidates; which ones are real is still yours to say.

References