Rhys Brett-Bowen

Replace the Trivial, Augment the Difficult

2026-08-27T00:00:00Z

“In the anticipated symbiotic partnership, men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions.” — J.C.R. Licklider, Man-Computer Symbiosis (1960)

A rule in the code-review engine I built caught an idiom I’ve been writing for years and can no longer see. That’s the small half. The other half is that it told me why — that the line leans on a coercion quirk and taxes the next reader with knowledge they shouldn’t need. I already knew the rule; I wrote it. What I didn’t have was the sentence, at the moment my eye had just slid over the thing, aimed at code I’d written that morning.

If the tool had said “line 40, don’t do that,” I’d have fixed line 40. I’d also have written it again the following week, because nothing about me would have changed. It said why, so now I catch it myself. The catch was worth one line. The explanation was worth every line I write after it.

That gap is the whole design question. The default shape for an AI tool is a verdict: the summary you read instead of the paper, the review comment instead of the review, the answer instead of the work. Verdicts are what demos are made of. But a verdict aimed at the hard part of a job doesn’t remove the hard part — it hides it, and hands you a fluent thing you now have to check anyway.

So: replace the trivial, augment the difficult. The slogan is the easy part. Which mode you’re in is determined before you build, by a property of the check itself, and you can read it off in one question.

BLUF

The mode is set by the referent, not by the difficulty. Ask what the check compares against. If that thing is written down somewhere a machine can read, you’re in replace-mode and the instrument should just decide. If it lives in someone’s head or in the part nobody wrote down, no instrument can decide it, and building one that pretends to is the failure.
“Trivial” means determined, not easy. A whole-repo type-and-import sweep is hard engineering and still trivial in this sense: the answer is fixed before anyone looks. Meanwhile “does this solve the right problem” is cheap to ask and impossible to settle mechanically.
In augment-mode, an instrument is judged on what it hands back, not on what it concludes. Filling one of those holes can’t transfer the judgment. It can only relocate it and equip it. So the spec is context: where to look, what’s already checked, what changed under you, what the plan used to say.
The teaching channel is the part that compounds. Replace-mode pays once per run. Augment-mode pays into the human, who keeps it. Every tool ships the finding; almost none ships the why, and the why is the only part that makes the next review cheaper.
Both errors cost, and they don’t cost symmetrically. Augmenting the trivial wastes attention on something already settled. Replacing the difficult launders the judgment — and you can’t tell from the output that it happened.

The test is the referent

Every check compares the work against something. Name that something and the build follows. There are three kinds. I’ve used that split before to decide where a check runs — How a Team’s AI Coding Flow Should Be Wired places each check at the point its referent exists. Here the same three kinds decide something else: what you build.

The referent is fixed and outside the work. A type signature, a lint rule, a banned import, a naming convention, a written architecture constraint. It existed before the diff and it doesn’t move. Mechanise it and let it decide. Don’t route it through a person, don’t ask a model to have an opinion about it, don’t soften the verdict to seem reasonable. A person adding judgment here is adding latency and nothing else.

The referent is live and moves while you work. The plan. The RFC. What everyone else has in flight. A machine can absolutely compare the change against the current state — that’s a query, and it’s tractable. What it can’t do is decide whether a divergence was justified. Plans get amended for good reasons and abandoned for bad ones, and the difference is not in the artifact. So the instrument here is mechanical at the noticing and mute at the verdict: it says “this moved, here’s what it used to say, here’s who else is in this file,” and stops.

The referent lives only in someone’s head. Is this the right thing to build. Does this shape match what we meant. Is this simpler than it needs to be. Nothing to compare against, because the comparison target was never written down. This is where the pull toward a verdict is strongest and most wrong: the model will produce a confident answer, the answer will be unfalsifiable, and you’ll have swapped a hard judgment for a plausible sentence.

The last case has a floor under it that isn’t about model quality. Every check reaches what you specified and no further, which means the part you never wrote down is invisible to all of them at once, by construction — A Pool of Models Narrows the Overlap settles that. Pool a dozen models and the blind spot is identical in every one. No amount of capability closes a gap that is definitional.

One caveat on running the test: it classifies comparisons, not tools and not review questions. “Does this implementation match the RFC?” straddles two classes at once — the RFC is written down, and the reason it got amended last week is in someone’s head. Split a question like that until each piece compares against one thing, then read the mode off each piece. A tool can run in three modes at once. A comparison can’t.

Two ways to get it wrong

Augmenting the trivial looks like diligence. It’s a scaffold on material the reader is already fluent in, a confirmation step on a determined answer, a human in a loop that had nothing for them to decide. It’s expensive in the scarcest thing you have. I have a clean receipt for the extreme version, metered in Noticing Is Cheap, Judgment Is Scarce: before I mechanised the noticing, a code-review engine of mine spent about 788,000 tokens across seven agents to say nothing about a fourteen-line rename. The rename was determined — a scanner could have cleared it in milliseconds. I was paying judgment rates to notice, and the fix was this same rule pointed the other way: take the determined part off the model completely.

The subtler form of this error is the one that gets praised. Faced with a bottom tier of trivial findings, the fashionable advice in 2026 is to suppress them — mute the nits, spare the human the triage. That reads as respect for attention and it’s the wrong lever, for the reason The Nit Was Never Optional works through in full. If applying and checking a fix is nearly free, you don’t hide the finding to protect the human, you act on it. Withholding a trivial finding to look restrained is still treating a trivial thing as if it needed a human. It’s the same misclassification wearing a nicer coat.

Replacing the difficult looks like progress. The output is confident, well-organised, and about the right length. And now the compression is load-bearing on exactly the thing you sat down to check. This is the error with no visible failure mode: a wrong verdict on a judgment call and a right one look the same from outside, and the only way to tell is to do the work the verdict was meant to save you.

The column determines the mode; the row is what you actually build. Pick the row that doesn't match and you land on a mismatched cell — augmenting the trivial, which wastes attention in plain sight, or replacing the difficult, which doesn't look like anything at all.

Notice that both errors come from the same place — deciding the mode by how hard the work feels instead of by what it compares against. Hard-feeling work attracts verdict-shaped tools. Easy-feeling work attracts human sign-off. Both instincts are backwards.

In augment-mode, the output is context

Once you accept that an instrument at a judgment station can’t decide, the spec changes shape. You stop asking “how accurate is it” and start asking “how much of the human’s work did it do, and how much better equipped are they for the part that’s left.”

Those are two requirements, not one, and the second is the one that gets dropped:

Take as much of the work as the referent allows. Everything mechanical, up to the line where judgment starts. Don’t stop early out of modesty and don’t stop early because a verdict would have been easier to ship.
Hand back what the judgment needs. Where to look and why. What’s already cleared, so attention doesn’t go there twice. What moved since the plan was written. Who else is in this code. The claim the change is making about itself, with citations back to source so every part of it is checkable.

The second bullet is why “a claim with citations” beats “a summary” every time, and it isn’t a stylistic preference. A summary you can’t drill into is a verdict with extra words. If the reader can’t click through to the paragraph, the diff, the ticket, they won’t trust it — and they’re right not to.

I keep running into the same instrument, built separately by people who aren’t coordinating with each other. One is a personal skill that explains an unfamiliar diff before its author starts reviewing. One is a side project that assembles the context a change needs. One is review tooling whose whole job is handing the reviewer an overview of what actually moved. None of the three returns a verdict. All three are doing the second bullet. I can’t name them and I’m not offering them as measurement — but three parallel builds of a thing nobody sells is people routing around a gap.

It surfaces in print, too, and without a name. A verification loop published for agent-written code asks four questions of a change: did it behave as intended, did it introduce a known defect, does it conform to the project’s standards — and then whether there’s sufficient context for a developer to review the result efficiently. The first three compare the change against something written down. The fourth isn’t a property of the change at all; it’s a property of the handoff, and it’s sitting unmarked in a list of pass-or-fail checks. The author doesn’t remark on the difference. That’s this distinction being felt before it has a name to be felt under.

Teaching is the part that compounds

A verdict pays once. It resolves this diff, this PR, this question, and its value expires when the work merges. Run it a thousand times and you’ve bought a thousand resolutions and nothing else.

An explanation pays into the human, and the human doesn’t reset. The rule that told me why my dead idiom was dead — the one in Fluent in a Dead Idiom — did the one thing my reflex structurally cannot: it taught me the deprecation. My reflex is what makes me fast, and my reflex was dated the day I finished training it. I can’t see my own blind spots — that’s what makes them blind spots — so I can’t generate that explanation for myself. The rule’s why is the channel through which something outside my fluency gets in.

The return compounds, which is the reason to make the teaching part explicit rather than incidental. A reviewer whose job is to educate and lift the code will surface a finding it could have quietly fixed, because the finding is worth more said out loud than silently applied. A reviewer optimising its own precision score will not. Here is my bet, and it is the one thing in this post I haven’t measured: those two tools score about the same on any benchmark you’d run, and they produce different engineers a year later.

The asymmetry is easy to miss because only one side of it has a name. Teaching a machine from human judgment is a well-worked problem: encode the rule, capture the correction, grow the catalog. The other direction — machine to human, deliberately, as the designed output — gets built by individuals and almost never shipped as the point of a product. It doesn’t demo. You can put a verdict on a slide. “The reviewers who used this got better at reviewing” takes a year to observe and doesn’t fit in a screenshot.

Which is also the strongest thing you can say to a skeptic who thinks this is soft. Replace-mode’s ceiling is the tool’s accuracy. Augment-mode’s ceiling is the tool’s accuracy plus whatever the human learned, and the second term only grows.

What this changes about filling a hole

Map an existing flow and you’ll find gaps where a check should happen and nothing does. The tempting read is that each gap is a missing tool of the same kind. It isn’t. Sort them by referent first and they split into three piles with three different builds — decide it, notice it and stay quiet about the verdict, or equip the person who has to decide.

Two consequences fall out, and they’re the ones I’d carry into any flow audit.

The holes are what humans are covering right now, badly. Not all of them — some gaps are genuinely uncovered, and a check that nobody performs at all is a different and more urgent problem than one a person is doing from memory at the wrong moment. But where a human is covering it, they’re covering it under the worst conditions: last, tired, from memory, against a plan that moved. The instrument’s job is to change the conditions.

Filling the hole doesn’t move the responsibility. Where the referent is mechanisable, an instrument really does take the responsibility — that is the point of mechanising it, and a fluent engineer who keeps checking anyway is spending their own time by choice. Where the referent isn’t mechanisable, the responsibility cannot move, because there’s nothing to move it to. So the honest measure of that instrument is never “does it decide correctly.” It’s “when the person decides, how much better is their position than it was before this existed.”

That reframes the artifact work, too. Most of what makes a judgment station workable turns out to be upstream of any model: a plan that carries its revisions instead of only its latest state, a change description that makes a checkable claim, an RFC that records the approach and not just the need, a queryable picture of what the team is currently doing. None of those decide anything. All of them are the second bullet — hand back what the judgment needs. They’re not model problems. They’re artifact problems, and they’re the cheapest available lever.

Why this isn’t the reading post again

I’ve argued amplify-over-replace before, for reading: Augmented Reading says use the model to read more, and never read the summary instead of the source. That post is about one human in front of one artifact, and its lever is scheduling attention.

The build-side question isn’t the same claim. Reading gave me a preference between two modes. This post gives a test for which mode a given hole is in, a three-way answer instead of a binary, and a spec for the output when the answer is augment. The reading post also carries the one caveat that binds hardest here: a scaffold’s value scales inversely with fluency, so guidance that lifts a novice becomes redundant load for an expert. That’s the expertise reversal effect, and it’s the reason “augment” is not a licence to add apparatus everywhere. Augment the difficult — and difficult is a property of the check, not a flattering description of everything you’ve built.

The point

Sixty-six years ago Licklider wrote down the division of labour: people set the goals, form the hypotheses, and perform the evaluations; machines do the routinizable work that clears the way. Engelbart spent a decade on the second half of it, and defined augmentation as increasing someone’s capability to approach a complex problem and derive their own solution. Neither of them proposed a machine that hands you the answer.

The industry defaulted the other way anyway, because a verdict is legible and a better-equipped human is not. The correction is cheap and it starts with one question: what does this check compare against? Answer that honestly and most of the build is already decided for you. What’s left over is the part nobody can take off you — and the only thing worth asking of a tool there is that it hand you the judgment in better shape than you’d have found it, and a little better again the time after that.

How sure I am:

Cited: the expertise reversal effect (Kalyuga, Ayres, Chandler & Sweller, 2003) — guidance that helps novices becomes redundant load for experts. The Licklider and Engelbart framings are quoted from the primary sources.
From experience: the rule that taught me a deprecation my own eye slides over; the fourteen-line rename that cost about 788,000 tokens across seven agents; the call to surface trivial findings instead of suppressing them. All from a code-review engine I built and run.
Prior art, corroborating not measuring: three independent builds of diff-explanation and change-context tooling that runs ahead of review, none of which returns a verdict. That’s convergent evidence the gap is real. It is not a measurement of what filling it is worth.
One sighting you can go and check: the four-question verification loop is Luttrell-Williams’, published and dated, and the fourth question really is a different kind of question from the other three. That part is verifiable. What I’ve added is the claim that the difference is a category difference, and that’s mine, not the author’s — they don’t draw it. Weight it accordingly: it’s a vendor byline with no measurements in the body, so it evidences that the distinction gets felt in the wild, and nothing about what acting on it buys you.
The wager: that the teaching channel compounds — that a tool which explains produces a measurably better reviewer over months, and that this dominates a tool which merely decides. I believe it and I haven’t measured it, and I’m not sure how to over a horizon that short. If this post is wrong somewhere, it’s there. The rest of the argument stands without it: the referent test holds even if teaching turns out to be worth less than I think.

Sources

J.C.R. Licklider, Man-Computer Symbiosis (1960, IRE Transactions on Human Factors in Electronics) — the original division of labour between goal-setting and routinizable work.
Douglas Engelbart, Augmenting Human Intellect: A Conceptual Framework (1962, SRI) — augmentation as increasing a person’s capability to derive their own solution; the tool system and the human system co-evolve.
Kalyuga, Ayres, Chandler & Sweller, The Expertise Reversal Effect (2003, Educational Psychologist) — the bound on when a scaffold helps.
Taylor Luttrell-Williams, CLI or IDE? Build in Verification First (The New Stack, 25 August 2026) — the four-question verification loop whose fourth question is about the handoff rather than the change. A vendor byline; cited as a sighting of the distinction, not as evidence for it.

Related: Augmented Reading (the same split for one reader and one artifact), How a Team’s AI Coding Flow Should Be Wired (the flow whose holes this sorts), Noticing Is Cheap, Judgment Is Scarce (why the mechanical half is the cheap half), Fluent in a Dead Idiom (the rule that teaches what the reflex can’t see), The Nit Was Never Optional (surface the trivial finding, don’t suppress it), A Pool of Models Narrows the Overlap (the unwritten part is invisible to every model at once).

A Flip Is a Missing Rule

2026-08-17T00:00:00Z

When the reviewer answers twice, stop tuning the model and go finish the rule.

“In the face of ambiguity, refuse the temptation to guess.” — Tim Peters, PEP 20, The Zen of Python (2004)

A colleague sent me the cleanest bug report I’ve had all year. He’d taken an open change, squashed it, and opened it again — same content, byte for byte, as far as the reviewer could tell. The automated reviewer left a comment on the second one that it hadn’t left on the first. Two runs, same input, two answers. He offered it as a determinism problem, which is what it looks like.

It isn’t. That comment was the reviewer telling me, in the only language it has, that one of my rules doesn’t say what to do at the boundary — so it had been guessing, and this time it guessed the other way.

BLUF: Bottom line up front

A flip is usually not model noise — it’s where your specification ran out and the noise got purchase. The title is the compression; this is the claim. You can’t turn sampling off, and that’s the point rather than the objection: a rule that determines its answer gives sampling nothing to flip between, so the places a reviewer does flip are disproportionately the places its rules underdetermine. The claim is conditional on one control I didn’t run: you have to pin the effective input, not just the diff. A flip doesn’t prove a rule is missing. It marks where one would bite, which is weaker and still worth more than anything else you get for free.
Sort the flips before you spend, because they don’t all want the same fix. Two aren’t the reviewer’s doing and you clear those first: a flip in the line number is your own ruler, and a flip that came from a different effective input — shifted context, different retrieval, a model version nobody chose — is your pipeline, not your specification. What survives splits two ways: a flip in noticing the trigger wants a detector, and a flip in the verdict on a trigger found both times wants a decision procedure. The split isn’t clean — a detector whose scope is itself model-judged can flicker in both places at once — but sorting roughly still beats paying to suppress all of them.
Sort every determinism lever by one question: does it remove the judgment call, or just average it? Union of N, ensembles, and context-splitting all average it. Caching freezes it. A closed decision procedure is the only one that removes the judgment — though what it mostly does is move the judgment somewhere observable, which is the honest version and the one I’ll argue.
Caching is the worst of them, because it works. It converts a visible flicker into an invisible wrong answer and calls that stability.
The measurement is the diagnosis. Two runs over the same diff agreed on four of nine findings — and my first count said three of ten, because part of what I was measuring was my own ruler. Splitting the disagreements by cause told me which rules were underspecified — that’s a work list, not a noise floor.
I got this wrong first, in a way worth copying. I read the residue as an irreducible floor and moved on. It wasn’t. Two of my rules were asking a question they couldn’t justify asking, and the flicker was the model declining to answer it.
You will not fix this at the sampler. Temperature and seed weren’t available to me, and where greedy decoding is available, endpoints still aren’t deterministic. Don’t wait on that lever; it isn’t the lever.

Two runs, three of ten

I ran the same rules pass over the same real diff twice and diffed the findings. They shared 3 of 10.

Hold that number loosely for two sections — I counted it wrong, and the correction is coming. But the shape survives the correction, and the shape is the point: a reviewer that agrees with itself on a minority of its own findings is not a check, it’s a draw from a distribution.

I had already measured the same shape from the other side and published it as good news: running decorrelated passes and taking the union caught about 2.5x the findings of a single pass, with round-to-round overlap of 25–35%. Two measurements, taken for opposite reasons, finding the same large variance. Not the same number — that one shares findings over each round’s own size, this one shares them over the union of the pair, and I don’t have the per-run counts to put them on one basis. So read them as two sightings of one phenomenon, not as a matched pair.

So the variance was never in doubt. What was in doubt was what it meant. In the coverage frame, low overlap is upside — each draw reaches findings the last one missed, so you draw again. In the determinism frame, the same wobble is a defect. One phenomenon, two readings, and the readings recommend opposite actions: one says run it more, the other says make it stop.

Both readings are right, about different things, and conflating them is how you end up buying the wrong fix. Union of N is a real, measured, published lever — for coverage. It does not make the reviewer decide. Ten distinct findings across the union of three runs, where the best single run gave four, is more sight — not more agreement.

Four things a flip can be, and two of them aren’t the reviewer

The 3-of-10 was a single number covering unrelated causes. Splitting it was the whole game.

Noticing-flicker. Rules with no detector behind them, where a scout has to spot the trigger by reading. Some runs it sees the borderline case, some runs it doesn’t. This is a sight problem, and I’ve already argued the answer: mechanise the noticing, and the scanner doesn’t have moods.

Verdict-flicker. Rules that do have a deterministic detector — the pattern is found, reliably, every run — and then the judgment on it flips. Detection is deterministic; the judgment isn’t. This is the interesting class, because there’s nothing left to mechanise. The trigger is already code. The wobble is downstream of it, in the part where something decides whether this particular instance actually matters.

Anchor-jitter. The same finding, reported one line apart between runs. This one wasn’t a defect at all. My comparison keyed findings on exact line, so a finding that moved by one line counted as one disappearance plus one appearance — two diffs for zero changes. Within any single run there was exactly one such finding. I had been measuring my own ruler and calling it instability.

Strictly, anchor-jitter isn’t a kind of flicker at all, sitting beside the two above. It’s a failure of the comparison’s identity function — a bug in how I decided two findings were the same finding. It belongs on this list because it arrives looking exactly like the others, and because it’s the cheapest one to find.

Effective-input drift. Here is the one I did not measure, and it is the load-bearing gap in this post. “Same input” and “same effective input” are different claims. I sent the same diff; I did not verify that the model got the same prompt. Between two runs, context assembly can shift, retrieval can return different neighbours, tool output can differ, the base repo can move, requests can route to a different model version or a different endpoint configuration, and a cache can be warm one time and cold the next. Any of those produces two answers from what you called one input, and none of them is a missing rule. So the honest form of this post’s claim is conditional: once you’ve pinned the effective input, a surviving flip is where your specification ran out. I pinned the diff and I did not pin the rest, which means my own 4-of-9 is an upper bound on specification flicker and not a measurement of it. If you run this instrument, log the assembled prompt and diff that first. It’s the cheapest control here and I skipped it.

Two of these you clear before spending anything, and they’re the two that aren’t the reviewer’s doing: my ruler and my pipeline. Two of my ten “disagreements” were an artifact of how I was counting, and any effort spent stabilising them would have been spent on nothing.

Which is the correction I promised: collapse that pair and there were nine distinct findings, not ten, and four were shared, not three. Four of nine — about 44%. I’ll use that number from here. Note which direction it moved: the reviewer was more self-consistent than my first count said, and the headline number got less alarming. It’s still a minority. Two runs, the same diff, and they agree on fewer than half their own findings — the case doesn’t need the inflated version, which is the only reason I trust the deflated one. All four stable findings were detector-backed — which points the same direction from the other end.

You cannot delete your way out of the judgment

The obvious escape from verdict-flicker is to remove the verdict. The detector already fires deterministically; post the detector hit directly and skip the model. No judgment, no flip.

I audited every detector-backed rule in that tier for exactly this. Not one was safe to promote. I can’t tell you how many rules that was — the count sizes a catalog that isn’t mine to publish — so read “not one” as exhaustive over the tier, not as a volume claim. A small tier would make it a weak result, and you have no way to check which it is. That’s a real limit on the claim, not a formality. Every single one had a genuine semantic exception the judgment step was catching: a guard that’s present but sitting in a comment, a parse that provably can’t throw on this input, a pattern that’s a problem in general and fine right here, a rewrite that would quietly change coercion behaviour. The tier wasn’t badly built — it was well-calibrated, and the judgment step was earning its place on every rule in it. Promoting any of them would have traded a flicker for a false positive with nothing behind it.

So the judgment stays. Which means the wobble is structural unless the rule changes, and that’s what forces the real question.

The levers sort by one question

Does the lever remove the judgment call, or just average it?

Lever	What it does to the judgment call	What you actually get
Union of N runs	Nothing. The model decides, N times.	Coverage — measured at ~2.5x findings. Not agreement.
Ensemble of models	Nothing. Narrows the shared blind spot.	A smaller blind spot. Not a decision.
Split the context for attention	Nothing. More attention on the same undefined call.	Better noticing.
Cache the verdict	Freezes it — whichever way it fell.	Consistency, and your first mistake made permanent.
Split noticing from judging	Nothing — but it tells you which flicker you have.	A diagnosis.
Closed decision procedure, with a default	Moves it somewhere observable, and removes it outright wherever a condition is detector-backed. The rule decides; the model only reports conditions.	An answer, and a shorter list of things that can still flip.

Five of those six leave the boundary undefined. That’s not a criticism of them — four are things I use and one I’ve published a measurement in favour of. It’s a statement about what they’re for. If your complaint is “the reviewer misses things,” the top rows are your levers. If your complaint is “the reviewer answers twice,” none of them touch it.

Caching deserves the specific warning, because it’s the one that looks like it worked. Cache the verdict and the flip disappears immediately and completely. What you have bought is consistency, not truth: the first answer becomes the permanent answer, and if the first answer was wrong you have made a wrong answer stable and taken away the only signal that would have told you. A flicker is a defect that’s still visible. A cached flicker is a defect with the alarm cut.

The human isn’t consistent, they’re n=1

Worth meeting head-on, because it’s the strongest version of the opposite reading. Alex Ewerlöf, arguing that coding is not a solved problem, puts the flip at the centre of the case: “a human can be wrong consistently, but a model can be wrong about something it was right about.” He reads it as intrinsic — a gap at least one breakthrough away. If that’s right, this post is mis-aimed: the flip is a property of the tool, and the thing to do about it is wait.

The comparison is mispaired, and the mispairing is the first row of the table above. Ewerlöf sets one model over N runs against one human over N sittings. A run is not a sitting. A sitting carries everything the last one left behind — the position you took, and the fact that you remember taking it. A run carries none of that. N runs of a model are closer to N reviewers than to one reviewer asked N times.

Not N strangers, though. Every run shares the same training, so they’re correlated draws — closer to N reviewers out of one tightly-trained cohort, who disagree about the same handful of cases rather than scattering. That’s the same correlated-errors problem that blocks a second pass from measuring the first one’s recall, arriving from the other direction, and it predicts the shape the flickers actually had: a small specific set, not noise spread evenly across the findings.

Read that way, consistency stops being a property of humans. It’s a property of having asked one of them. Two reviewers disagree about an unspecified boundary as readily as two runs do; you just never run that experiment, because you assign the review to one person and ship what they said. The specification is exactly as missing. You sampled it once.

Which is why the stickiness is a cache and not a virtue. A reviewer’s prior is sticky because they remember deciding and would be embarrassed to reverse it. That damps the flicker without closing the boundary — the cache row’s trade, made in a person. The model flips more because its priors are weaker, not because it’s less trustworthy about the parts that were actually specified.

It also explains a row I measured rather than reasoned. Union of N buys coverage and not agreement because union of N is a panel, and panels have always worked like that.

What survives untouched is the accountability half, and it’s the stronger half. A model can’t be on call, can’t carry a consequence, can’t be punished. That asymmetry is real and no determinism work touches it. But it’s a claim about who answers for a wrong verdict, not about whether the verdict is right, and borrowing it to carry the reliability claim is what makes the reliability claim look like a difference of kind. It’s a difference of degree. A difference of kind you wait out. A difference of degree you work on.

I haven’t measured human flip rates against model ones, and the gap may be real and large. What I reject is the inference — that a flip is a species marker, and the answer is to wait for a better model.

I called it a floor. It wasn’t a floor.

Here’s the part I’d want someone to tell me.

I mined three flickering findings as decision-boundary cases and wrote a worked example into each rule, then ran three fresh passes. One case — a refute example, on something the rule was flagging that plainly wasn’t a violation — went from appearing in 1 of 2 runs to 0 of 3. It stopped showing up — which at three runs is the most I can say, and “gone” is not available to me. A precision win and a determinism win in the same edit, on one finding. The other two were confirm examples on genuinely borderline cases, and they barely moved: 1 of 2 → 2 of 3, and 1 of 2 → 1 of 3.

I wrote that up as an irreducible floor. Semantic rules wobble; examples damp the clear cases; the borderline residue is the cost of having judgment in the loop at all. It’s a tidy conclusion and it has the shape of hard-won engineering wisdom, which should have been the tell.

A second review pass, from a different model family, told me the framing was wrong: a flip means the rule lacks a decision procedure. I pushed back before I checked, then checked. It was right, and worse than right — it predicted the specific failure. I went back to my two stuck cases:

One was flagging a derived collection that gets rebuilt from its source every time. Nothing mutates it independently, so nothing can drift out of sync. It’s display scaffolding, not the defect the rule exists to catch.
One was flagging a purely local view-state transition inside a state updater. That’s the updater doing its job, not domain logic trapped in the wrong layer.

Neither was a violation. I had written confirm examples asserting a verdict the rule could not justify. The examples weren’t failing to stabilise the model — they were failing to be true. The model wasn’t wobbling because judgment is inherently unstable. It was wobbling because I’d told it to answer yes to a question where the honest answer was no, and it could see the case didn’t fit.

That is the whole argument in one receipt, and I only have it because I was wrong out loud first. An irreducible floor is unfalsifiable and comfortable — it explains any residue you have, requires nothing of you, and would have stopped the work exactly where the work got useful.

A demo treats one instance; a procedure defines the policy

The fix wasn’t a better example. Once you know the rule was asking an unjustified question, the repair is to make the rule answer it — a decision procedure, which I’ve described elsewhere as “an explicit list of observable conditions, with the verdict recomputed in code from which ones the model ticked.” Closed, this time: the conditions under which this confirms, the conditions under which it refutes, and what to do when the evidence isn’t visible. That last clause is the one everybody skips, and it’s the one that ends the flicker, because “the evidence isn’t in front of me” is exactly the state that was being resolved by a guess. Give it a default — refute, for a reviewer, because an unsupported comment costs more than a miss you’ll catch another way — and the guess at the verdict is gone.

Which is where the strongest objection to this whole post lives, so let me put it in its harshest form: you didn’t remove the judgment, you relocated it. The verdict is recomputed in code, fine — but something still has to decide whether condition C holds, and for most of my conditions that something is the model reading the diff. So the flip doesn’t die, it moves one level down, from “does this matter?” to “is there an unlogged catch here?” A procedure with model-evaluated conditions is not a deterministic procedure.

That’s correct. Two things make the relocation worth doing anyway. The conditions are observably true or false against the code, which the verdict never was — “is there a guard on this path” has an answer you can point at, where “does this instance matter” was asking the model to hold the rule’s purpose in its head and rule on a case the rule never described. And some conditions are detector-backed, which takes them out of the model’s hands entirely; each one you can mechanise is a piece of the procedure that stops being able to flip at all. So the mechanism is a ratchet, not a switch: you move the guess to a place where it’s cheaper, checkable, and sometimes removable.

What I can’t tell you is how much flicker that actually killed. Three post-fix runs on three findings cannot separate eliminated from reduced. So the claim is that the class got re-specified, not that the reviewer stopped flipping on it. And if a condition report flips as often as the verdict did, the ratchet is a wash, and what’s left of this post is “decompose before you spend” — still true, still worth doing, and a much smaller claim than the one I’m making.

The difference between the two fixes is scope. An example pins one instance; anything shaped slightly differently is back to a guess. A procedure defines the policy, so it covers the cases you haven’t seen yet. A demo treats one instance; a decision procedure defines the policy. Re-specifying those two rules closed the specification gap for the class, using evidence I’d already paid for — that much I can show. Whether it closed the flicker is the question the paragraph above says I can’t answer at this n.

This is the same instrument as one I’ve argued for from a different angle: run the work on a model too cheap to do it, and where it breaks is a map of the judgment you haven’t encoded. The flip is that instrument without the second model. It runs on your production configuration, on every real diff, for free, and once you’ve ruled out your ruler and your pipeline, it points at the specific clause you never wrote. You just have to stop reading it as weather.

It also does something I’d previously filed as impossible. I’ve argued that a pool of models can’t recover what you never specified — every model verifies what you wrote down, so the part you left out is invisible to all of them at once, by construction. That still holds, and no amount of union or ensemble touches it. But it’s an impossibility about reaching the missing part, not about locating it. A flip with the effective input pinned locates it. You can’t get a model to supply the clause you never wrote; you can get it to show you where the clause should have been, and then you write it yourself. A hard ceiling turns into a work item.

Where this stands

I have not made the reviewer deterministic, and I want to be exact about what I have — which is less than I first wrote here. One flicker class stopped appearing after a truthful refute example, in three runs. One class got re-specified after I admitted the rule was wrong, and I can show the specification changed but not that the flicker did. One class was never a defect and I fixed my measurement instead. Note the shape of that list: one observed case each, three runs deep, so “eliminated” is a word I’ve taken back everywhere it appeared. The residue is real, it’s mostly in rules whose boundaries I haven’t finished writing, and finishing them is a long grind of specification and mechanisation, not one clever fix. Anyone promising you determinism in an afternoon is selling you a cache.

What generalises isn’t the fix. It’s the reading. When an agent gives two answers on an input you have genuinely held still, you have been handed a free, precisely-located pointer to the place your specification runs out — and the entire instinct of the field is to spend money making that pointer harder to see. Run it twice on purpose. Log the assembled prompt, not just the diff, so you know the input really was the same one. Diff the answers. Then sort the disagreements before you spend a day on any of them: throw out the ones that are your own ruler, throw out the ones that are your own pipeline, send the noticing failures to a detector, and what’s left — the verdicts that flipped on a trigger the system found both times — that’s your backlog. It’s shorter than the diff and it’s the only part that’s specification work.

How sure I am: The overlap figure, the decomposition, and the example experiment are metered runs of my own, genericised, and small n — two runs before and three after, on three findings, on one diff. The uncontrolled confound is effective-input drift, described in the body: I held the diff fixed and did not verify that context assembly, retrieval, tool output, base-repo state, model version, endpoint configuration, and cache state were also fixed. That makes 4-of-9 an upper bound on specification flicker, and it means the post’s central claim only holds once you’ve pinned the effective input — which I did not do. Treat that and the ratchet as the two places this could be wrong rather than merely thin. Treat the run figures as the shape of the thing and not as rates; the tier audit is mine too, but its denominator is a different number and a withheld one — I audited the whole tier and I can’t print how many rules that is, so “not one was safe to promote” is unsizeable, not small, and the small-n above doesn’t cover it in either direction. The refute result (1/2 → 0/3) is one finding that stopped appearing across three runs — not an elimination and not a measured effect size, because three runs can’t tell those apart. On the overlap specifically: 4-of-9 is the corrected figure and 3-of-10 was my first count, and I’d rather you saw both than trusted either — the correction came from my own re-read of how I keyed findings, not from an independent check, so the method of it is unaudited even though the arithmetic is plain. The 2.5x union figure and the 25–35% overlap are from my own earlier published work, deep-confirmed but not oracle-verified — and that 25–35% is a share of each round’s own findings while mine is a share of the two runs’ union, so the two are not on a common basis and I don’t have the per-run counts to put them on one. I’m claiming they see the same phenomenon, not the same number. The ratchet claim — that moving the guess from the verdict down to an observable condition is a real gain, because conditions are checkable against the code and some can be mechanised away entirely — is an argument, not a measurement, and it’s the load-bearing one in the post. I have not measured whether condition reports flip less often than verdicts did; if they don’t, the gain is smaller than I’m claiming and the honest residue is the decompose-first advice. Treat that as the post’s biggest open exposure. The claim that a flip with the effective input pinned locates a missing decision procedure is my take, mechanism-backed — the strongest evidence for it is that it corrected me and then correctly predicted why my two examples had failed, which is more than I can usually say for a framing. “Averaging buys consistency, not correctness” is an argument, not a measurement. The unavailability of temperature and seed is specific to the endpoint I was on; the reason greedy decoding wouldn’t have saved me anyway is borrowed and cited below, measured on their setup, not mine. The n=1 section is reasoning, not measurement: I have not measured human reviewer flip rates against model ones, and I can’t cite anyone who has. What the argument establishes is that the usual comparison — one model over N runs against one human over N sittings — doesn’t license the difference-in-kind conclusion drawn from it. It does not establish that the two rates are close, and they may not be.

Sources

Cross-run finding comparison — my own instrumentation: two runs of the same rules pass over one real diff, 3 of 10 findings shared on the first count and 4 of 9 after correcting the counting artifact, decomposed into noticing-flicker, verdict-flicker, and that artifact — a cross-run effect of keying findings on exact line. Shares are over the union of the two runs. The diff was held fixed; the assembled prompt, retrieval, and model routing were not verified fixed, so this is an upper bound on specification flicker. Author’s own, genericised.
Tier promotion audit — every detector-backed rule in the tier reviewed for whether its judgment step could be dropped in favour of posting the detector hit. Zero promotable; each had a genuine semantic exception (commented-out guard, provably-safe parse, contextually-fine pattern, coercion-changing rewrite). Author’s own, genericised.
Worked-example experiment — three flickering findings mined as decision-boundary cases, two runs before and three after: one refute example 1/2 → 0/3 (stopped appearing; three runs can’t distinguish that from reduced); two confirm examples 1/2 → 2/3 and 1/2 → 1/3. Author’s own, small n.
The correction — a cross-family review pass rejecting my “irreducible floor” framing in favour of “a flip means the rule lacks a decision procedure,” and the subsequent re-reading of both stuck cases as non-violations. Author’s own working notes; kept because the wrong version is load-bearing.
Thinking Machines, Defeating Nondeterminism in LLM Inference (10 September 2025) — of 1000 completions sampled at temperature zero, “we generate 80 unique completions, with the most common of these occuring 78 times.” Greedy sampling is “theoretically deterministic” and yet “LLM APIs are still not deterministic in practice”; the paper puts the primary cause not on concurrency-plus-floating-point but on the fact that “the load (and thus batch-size) nondeterministically varies,” against a forward pass that “lacks ‘batch invariance’.”
PEP 20, The Zen of Python, Tim Peters (2004) — “In the face of ambiguity, refuse the temptation to guess.” Written as advice for people writing code; it reads as a specification for how a rule should behave when the evidence isn’t there.
Alex Ewerlöf, Coding Is Not Solved (blog.alexewerlof.com) — explicitly opinion-led, and cited for one span: “a human can be wrong consistently, but a model can be wrong about something it was right about,” read by the author as intrinsic and at least one breakthrough away. I take the accountability half of his case — a model has no skin in the game and can’t be punished — as correct, and untouched by anything in this post. The reliability half is the part I contest, and the n=1 reading of it is mine. Neither side of that disagreement is a measurement: his piece offers none on it, and neither do I.

Related: A Cue Finds More of What It Can Already See — the union-of-N measurement this post reclassifies as a coverage lever rather than a determinism one; Noticing Is Cheap, Judgment Is Scarce — a mechanised detector can’t flicker on noticing, because a scanner doesn’t have moods — which is an argument from construction, not a class I measured going away; Keep the Factory, Ship the Result — the decision procedure as an artifact, and the cheap-model probe this post’s flip-reading is a free alternative to; A Pool of Models Narrows the Overlap — It Doesn’t Erase It — why more models can’t recover what you never specified; A Big Rule Set Is an Attention Problem — mechanise the noticing, not necessarily the rule.

How a Team's AI Coding Flow Should Be Wired

2026-08-17T00:00:00Z

Every check needs something to compare against. Which loop asks it, and what answers, is decided by how that thing behaves — and for both of the checks that matter, the right place isn’t the PR.

“Creators need an immediate connection to what they’re creating.” — Bret Victor, Inventing on Principle (CUSEC, January 2012)

A change lands in review with everything green. The rules pass, the types check, the tests are close to the ones I’d have asked for. I read the diff anyway, and the thing I’m looking for isn’t in it. I’m trying to work out whether this was the change to make — whether it collides with the other work in flight, whether it’s still what we agreed to build, whether the shape it took is the shape the problem wanted. Nothing upstream of me asked any of that. Nothing upstream of me can, because the only place the answer is written down is a plan that stopped being true a week ago, and nobody updated it.

That’s the state of the art, and we’re pouring engineering into the other half.

BLUF: Bottom line up front

There are two checks on a change, and they compare it against different things. Call the thing a check compares against its referent. Is the code done right holds the diff against the codebase and the house’s rules. Is it solving the right problem, in the right way holds the work against the plan and against everything else in flight.
The two referents behave differently, and that’s what makes this a placement problem. Question 1’s referent sits still, but the thing it judges — the finished change — doesn’t exist until the end, so that check is pinned late by construction. Question 2’s referent is live and moves while you build, and the thing it judges exists from the first prompt, so that check can be asked at every stage and gives a different answer each time.
“Shift left” picked the check that can’t move. Most write-time instruments we reach for — linters, hooks, pre-commit, stop burning tokens on LLM review and write custom rules instead — are trying to move the one check that has nothing to run on until the change is finished. Shift the other one.
The write-time argument doesn’t survive the change of author. Write-time checks are shaped for a human write loop: a human types character by character, so firing between keystrokes is genuinely earlier and costs nothing. An agent emits in batches. The agent doesn’t consume autocomplete, it is the autocomplete.
The flow is three loops with six stations in it, and the second question is askable at five of them. A station is any point in the flow where a check can sit. Plan, generation, review — and review alone is three stations, not one. Each station has a different thing to compare against, which is what decides both whether a check can live there and which instrument can run it. The sixth station is question 1’s only home. Two of the five that take question 2 exist only for a team: nobody’s solo work collides with what the rest of the org is building.
A human has a re-entry point in every loop, not an approval click on every iteration. They settle consequential plan choices, return when generation proposes a material change, and check the final claim. Whether a re-entry blocks is team policy, set by reversibility and risk, not by anyone’s preferred workflow.
Count the hole honestly. Five of the six stations can take the second question. By instrument I mean something built, that runs on its own, and that belongs to the team — not a habit, not a prompt somebody remembers to use. Three of the five have no instrument at all, one can’t have one, and the fifth is a human doing it last and from memory.
The lever nobody’s pulled is shifting the first check right. Let changes land, sweep the codebase instead of the diff, and the check gets what it always wanted: the whole referent, and time. It only works if its output is a fix, not a finding.
Build the org reviewer first. It pays at two stations off one build, and its referent is the only one with no human substitute — a reviewer can hold the plan in their head; nobody holds what every other in-flight change is doing.

Two checks, and only one of them can move

Strip a code review down and there are two questions in it. Is the code done right. And is it solving the right problem, in the right way.

They feel like the same activity because they arrive on the same screen at the same time. They aren’t, and the thing that separates them isn’t difficulty or importance. It’s what each one compares against, and when the thing it judges shows up.

	Question 1 — done right	Question 2 — right problem, right way
Referent — what it compares against	the codebase + the house’s rules	the plan + everything else in flight
Does the referent hold still?	yes — it’s there the whole time	no — it moves while you build
The thing being judged arrives	only when the change is finished	with the first prompt
So the check runs	once, late — by construction	at every stage, with a different answer each time
Instrument today	built	a human at the end, reading a diff

Naming the first referent as “the house’s rules” invites the obvious objection: whose house? Different projects here do it differently, so which one is the rule? The answer is that the divergence was never load-bearing. Local idiom exists because whoever wrote the code first set the style and every owner since dragged it a little further toward their own preferences, and the justification for tolerating that was always human — a maintainer had preferences, and making them work in a foreign style cost them real effort. An agent has no preferences to accommodate and conforming costs it nothing. So local idiom is now pure cost with no offsetting benefit, and the referent should be one house, not one per directory. The Other Intent argues the one-person-wrote-it version of this at length; I’m taking it as settled and noting only that it makes question 1 more mechanizable, not less.

Three classes of referent turn up once you start looking for them, and each one dictates where its check can go. A referent that’s fixed and outside the work lets the check run the moment the thing it judges exists — which for question 1 means late, because that thing is the finished change, and no cleverness is available. A referent that’s live and moves while you work wants the check run twice, at both ends, because the two readings answer different questions. A referent that lives only in someone’s head can’t be mechanized at all, so the only lever is putting it where judgment is cheapest to act on.

That third class is where “taste” usually gets invoked, and taste is doing two jobs there. Taste about how code should look shouldn’t exist. If two shapes are both acceptable, that isn’t taste, it’s a missing rule — again, argued at length next door — and once it’s written down it’s a question-1 rule like any other: rule-shaped conformance, checked against an artifact that exists. What’s left when you delete that half is taste about how to solve the problem: is this the right decomposition, does this belong in this service, is the simpler thing available. That half never converts into a rule you can hold a diff against, because it isn’t asking whether the artifact conforms. It’s asking whether this was the right change. That’s question 2.

Which gives the tightening that made me want to write this post: the check that can’t be mechanized and the check we’ve built nothing for are the same check.

So here’s the turn. “Shift left” isn’t wrong because earlier is bad. Larry Smith’s original 2001 argument for it was sound and it’s still sound. It’s wrong because of what we pointed it at. We aimed a decade of tooling at the check that has nothing to run on until the work is done, and left the movable one to a person at the end.

Why “lint it earlier” doesn’t transfer

There’s a live position that you should stop burning tokens on LLM code review and write custom linters instead: deterministic, fast, cheap, and it fires while you write. The economics are unarguable for a human. They don’t transfer, and the reason is the loop, not the model.

A human types character by character. A check that fires between keystrokes is genuinely earlier than one that fires on the finished file, and the loop is close to free — the editor is already open, the eyes are already on the screen, and the response-time budget that tooling is built against came from human perception in the first place. Miller measured it in 1968 and Nielsen restated the thresholds in 1993: a tenth of a second to feel instant, one second to keep your train of thought, ten seconds and you’ve gone to make coffee. Those are numbers about a nervous system. An agent doesn’t have one.

An agent emits in batches. Its smallest output unit is already a chunk of finished work, and its smallest feedback loop costs a tool call and context. It doesn’t consume autocomplete — it is the autocomplete, and that’s a literal mechanism rather than a slogan: fill-in-the-middle training is what turned a next-token model into a completion engine, and the same capability scaled up is the thing now writing whole files.

The strong version is wrong. Agents do get per-edit feedback — fill-in-the-middle completion, tsc --watch, a hook on every edit. The claim is about grain and cost, not about the absence of a loop. And it isn’t that write-time and review are the same station; a post-edit hook genuinely fires before the change is finished. It’s that the gap between after each agent edit and on the finished diff is far smaller than the gap between per keystroke and on the finished diff. So the win from shifting is much smaller than the human intuition predicts, and the human intuition is where the whole argument came from.

Two mechanisms, one conclusion, and they’re independent. The referent argument says the check has nothing to run on until the change is finished. This one says the loop the tool was built for doesn’t exist either.

It also answers the sharpest objection to the referent argument: a linter fires on my half-written file mid-keystroke constantly, so your claim that this check can’t run early must be too strong. Correct — inside a loop that exists only because a human types one character at a time. Tree-sitter says the quiet part out loud in its own documentation: “In applications like text editors, you often need to re-parse a file after its source code has changed.” Write-time tooling is editor tooling. The editor was the point.

What we built, and the measurement that made it necessary

We built the instrument for question 1: an automated reviewer that runs before human review and decides whether the change is fit to merge into the rest of the code. It’s real, it runs, and it’s half of what’s needed.

It exists in the shape it does because of one measurement. Before building it we took our review rules — question-1 rules, every one of them, conformance checks against an artifact — and asked, one at a time, which of them a linter could express. Most of them can’t. Not a minority, not the exotic ones — most. They’re conditional on shape, riddled with exceptions, and they turn on what the code is for. A Big Rule Set Is an Attention Problem reports the same shape from the other side: about a third of the rules leave the model’s context entirely, and the rest still need a judge even after the noticing is mechanised — which is the split Noticing Is Cheap, Judgment Is Scarce owns. That result is the load-bearing counter to write custom linters instead: the deterministic subset is worth mechanizing and we did mechanize it, but if you stop there you’ve built a check for the part that was never the problem. I’m giving you the direction and not the count, deliberately; the denominator is our catalog.

Which also means the reviewer buys back exactly one thing. It removes the obligation to read the diff for correctness. It doesn’t remove the ability, and fluent senior developers will keep reading anyway, which is fine and good. The difference between a check and a backstop is what happens when they disagree: when someone finds a defect the automated review passed, that’s a gap in the instrument, and the fix is a rule. The human’s code read is the check’s test set. That’s how the catalog grows — the loop Teaching the Reviewer argues, and the reason this station can stay a precondition instead of drifting into an opinion.

The map — three loops, not a pipeline

The plan is current operational state, regenerated as work changes. When generation changes the underlying intent or rationale, it appends a non-destructive RFC amendment with the earlier plan, what changed, and why; review needs both the current plan and that path.

Three loops, and the reason to draw them as loops rather than stages is that every one of them can send you backwards.

Humans are present in all three loops, but not as an approval click on every iteration. The system owes them a visible re-entry point: consequential plan choices in loop 1, a material change to the plan or RFC in loop 2, and the claim review in loop 3. Whether that point blocks is policy — decided by reversibility and risk, not by who happens to prefer a tighter workflow.

A designer is often in this flow too, and where they sit isn’t fixed. Some hand off: they settle what the thing should be and the RFC carries it from there. Some stay in loop 1 and iterate on the plan as it firms up. Some skip the middle and check the end product, beside 3c. That’s a team’s choice, not a property of the flow, so the designer isn’t a station — a station asserts one placement and this actor has three. The modes aren’t equivalent, though. A designer who hands off is supplying part of question 2’s referent, helping decide what right means. A designer who iterates in loop 1, or who checks the end product, is asking question 2 — and has no more instrument for it than the stations around them. The hole is one participant wider than the count.

Loop 1 — the plan. A prompt becomes an RFC, and the RFC gets refined in a loop rather than written once. Two different checks want to live here. The first is the thinking-partner shape: surface the fork, name the tradeoff, catch the assumption nobody stated. Its referent is only in a head, which is the class that can’t be mechanized, so it’s a prompt for judgment, not a check with a referent an instrument can read. Status: exists, as a personal skill and a prompt, not as a team instrument. The second is the org read — does this collide with the other work in flight, does it serve the objectives we’ve actually stated. Its referent is live, and this is its first of two readings. The instrument that would run it is the one I’ll call the org reviewer, and it’s the same instrument 3b needs; it’s what this post ends up arguing to build first. Status: not built.

Loop 2 — the code. Generation, with guardrails on architecture rather than on syntax. The instrument here reads the trajectory, not the diff — where the work is heading, not what the text currently says — and that’s exactly why it can live inside generation while a rules check can’t. Inside generation the checkable thing is the direction, not the text. Evals Aren’t the Moat makes the case for watching the trajectory rather than the snapshot; what this adds is where that watch has to sit — inside the generation loop, because that’s the only place the direction is still cheap to change. Status: not built.

The part of this loop that matters most is its second output. The plan is current operational state — the execution projection of the RFC — so generation updates or regenerates it as sequencing, dependencies, and remaining work change. It is not the expensive first artifact, preserved forever.

An RFC amendment is different. When generation changes the underlying intent, constraint, or rationale rather than merely the execution steps, it appends a non-destructive revision: the earlier RFC and plan stay visible, along with what changed and why. Otherwise “the updated plan” is only the latest state, and no downstream check can tell whether a divergence was justified or merely forgotten. This makes the history a byproduct of drawing the loop correctly, not a discipline bolted on and then skipped. The current plan and the amendment history have two consumers: the automated check at 3b, and the human who needs to drop to the raw at 3c. Status: not built.

Loop 3 — the review, three stations in order.

3a — automated code quality. Question 1, at the only place it can be asked, as a precondition rather than an opinion: it passes or the change doesn’t go on to be argued about. That it gets to block at all rests on the check being mechanized, and Nobody Broke It narrows even that: deterministic is necessary and not sufficient, because a check that imposes a consequence also owes an answer to whose obligation is this? On a PR the diff answers that for free — which is the one advantage this station holds over the sweep I’ll argue for later. The failure mode to watch here is silent rather than loud: when a gate breaks it errs toward letting the change through instead of blocking it, so a station that has quietly stopped flagging looks exactly like a station with nothing to flag. Status: built.

3b — against the updated plan and the current org state. Question 2’s second reading, and the word updated is carrying the weight. This station only works because loop 2 wrote back; against the original plan it’s comparing the change to a fiction. The two readings ask different questions, too. At plan time: don’t start something that collides. Here: the world moved while you were building — does this still fit? Status: not built.

3c — the human. And their referent is the claim, not the code, which has a consequence people won’t like: the PR description becomes a load-bearing instrument rather than a courtesy. A description that restates the diff hands the reviewer nothing to check. It has to make a claim — what I set out to do, what changed on the way, what I left — and then the human can check the claim against the plan, and drop to the raw when the claim looks thin. That matters because a claim is not a substitute for reading: A Description Can’t Replace the Code argues no description stands in for the raw, and a recorded intent is a description too. The claim’s job is to be a pointer — it tells the reviewer where to drop, and what’s checkable before they get there.

The other consequence is that the human is inside loop 3, not after it. A comment re-enters the loop: back through the automated review, back through the org re-read. They aren’t a terminal gate that bypasses the machinery, and they’re most valuable precisely where the computable checks have already cleared, because their referent is the one that can’t be computed. That’s the routing Augmented Reading reaches from the other direction — run the machine pass first and put each reader where its own decay costs least; here the same order falls out of the referent rather than out of fatigue. Status: happening today — but as a person doing it last, from memory, against a plan that moved. That’s the thing to fix, not the thing to preserve.

The hole

Six stations across the three loops, and five of them can take question 2: the fork-and-tradeoff read at plan time, the collision read at plan time, the trajectory check during generation, the re-read against the updated plan at review, and the human’s read of the claim. The sixth is 3a, and it’s question 1’s only home — the one station on the map that isn’t a place to ask this.

The tempting sentence is four of the five have nothing. It’s cleaner and it’s wrong. The honest count: three of the five have no instrument, one can’t have one by construction, and the fifth is a human doing it last and from memory.

The middle clause is the one to sit with, because it isn’t a gap a better model closes. A Pool of Models Narrows the Overlap — It Doesn’t Erase It settles that: every model verifies what you specified, so the part you never wrote down is invisible to all of them at once, by construction. Question 2’s referent is exactly that part. Which is why the answer here is instruments that have to be built, not a model pool that has to be sampled.

That’s still the whole movable half of code review, unserved. And one level above the diff has a measured gradient behind it. Zhu, Tsantalis and Rigby measured it inside a single pipeline: as generated volume goes up, architectural problems track it at ρ = 0.94 against 0.53 for code-level ones, and file count at 0.72 against 0.33, which wasn’t significant. Be precise about what that buys. Architectural smells are what the loop-2 guardrail watches for — its referent is the trajectory, not the plan — so that’s a different unbuilt instrument from question 2’s. What the gradient says is that the further above the line you go, the faster the problem scales with the amount of code an agent writes, and everything above the line is unserved. It isn’t evidence about question 2; nobody has measured that. Their bounds are real and worth carrying — twenty projects from one multi-agent framework, one detector, generated code rather than a maintained repo — so take it as the direction, not the coefficient.

The same paper kills the obvious cheap fix. Four escalating levels of prompt detail produced no significant effect on any smell category, all p > 0.8. You cannot front-load your way out of this by writing a better plan up front, which is what makes the loop, and the checks inside it, the only place the work can happen. The Brief §2 owns that argument; I’m citing it, not re-deriving it.

The most expensive defect is a well-built change that shouldn’t have been built, and every instrument we have is pointed at the other thing.

The lever we haven’t pulled — shift question 1 right

Question 1’s check can only run late. But late has no upper bound, and that’s a lever nobody pulls.

Let changes land without the heavy analysis. Then sweep new code across the whole codebase on a schedule instead of per-PR. The upside is exactly what question 1’s check always wanted from its referent and the PR could never hand it: the agent gets the whole codebase instead of a diff, and it gets time to use it. Reviews get fast because the slow check left the critical path. Structurally that’s Buy Back the Coverage applied to the review itself: reviewing a diff is a method that narrows the view, the codebase outside the diff is the bill, and a sweep is a compensating check aimed at exactly what the method cut out.

The downside isn’t the one people name first. Yes, a sweep finds the problem after it’s load-bearing — other code already depends on it, the author has moved on, the check is better and the fix is more expensive. But the real cost is this: a sweep has no author. A PR check lands on someone who owns the change and is still holding the context in their head. A sweep finding lands on whoever is nearest. The problem with “less oversight” is attribution, not detection, and it’s the bill for the free answer 3a was quietly collecting.

The whole lever depends on one condition, and I’ll mark it as a prediction rather than a result because nobody has run the experiment: a sweep is only viable if its output is a fix, not a finding. A finding reports that something is wrong and asks somebody to deal with it. A fix is a patch you can read, argue with, and merge. The difference is who is left holding work at the end of the check. An ownerless backlog of findings is a documented failure mode, not a guess. The Google static-analysis team reports that their bug dashboard “saw little use because a bug dashboard was outside the developers’ usual workflow”, and that the answer was to stop shipping dashboards: “Instead of providing bug dashboards, projects like Error Prone extend the compiler with additional checks, and surface analysis results in code review.” The warnings accumulate, nobody is on the hook for any single one, and the list stops being read. A sweep that opens the fix has somewhere to land. That’s newly possible now, which is the reason the lever is worth pulling now rather than in 2015 — and The Nit Was Never Optional is the repricing underneath it. Once generating the fix is cheap and instant, the effort a finding used to protect all but disappears; here that repricing runs one level up, where it’s what turns an ownerless finding into a reviewable patch.

You can already watch both halves of this play out. CodeVibes is an open-source reviewer built on DeepSeek that does the sweep exactly as the referent argument says you should: point it at a repository, and it walks the file tree in priority order — secrets and auth first, then core logic, then everything else — analysing whole files. No diffs, no commits, no PR webhook anywhere in it. That’s the right referent, arrived at independently, and it’s the strongest evidence I have that the lever is real rather than merely tidy. Then it hands back a list of issues and a score out of a hundred. It has the sweep shape right — whole files, no diff, no webhook. The patch is the part that decides whether anyone acts on it.

Two of the best-documented static-analysis programs we have both pulled the other way, back before a patch was cheap to generate. Tricorder states its placement plainly: “Given that all developers at Google use code review tools before submitting changes, TRICORDER’s primary use is to provide analysis results at code review time.” The FindBugs production evaluation records the same drift, moving warnings “more directly into the developer workflow” as confidence grew. Both of those teams moved findings toward the author. The sweep only beats them if it arrives carrying a patch.

Then there’s the boundary, and it’s the part that convinced me this isn’t a tangent. The sweep works for question 1, and it fails for question 2 — a question-2 finding discovered post-merge isn’t a fix, it’s a migration. The sweep works for exactly the check that can’t shift left, and fails for exactly the one that can.

So question 1 shifts right, question 2 shifts left, and both of them move away from the PR. The thesis is sharper than where it started: the PR is the wrong place for both checks, in opposite directions.

The knowledge graph rides on the sweep

Here’s what closes the loop, and it’s structural rather than decorative. By the graph I mean the org’s work and its code held as linked entities an agent can query — what’s in flight, what it touches, what objective it serves, what depends on what — rather than as prose a person has to read. The org read asks does this collide with what else is being built, and that’s a question about edges, not records. The sweep already walks the whole codebase with the whole codebase in context. That is the same pass you need in order to build that graph — and the graph is the precondition for the org reviewer, which is question 2 shifting left.

The right-shift is what funds the left-shift. The two moves aren’t independent.

Per-commit patches keep the graph current between sweeps: build in batch, patch on change. It’s the same shape as incremental parsing — one full parse, then re-parse only what the edit touched — and I’m pointing at that as a structural echo, not as a claim about a system that exists.

What to build next, and why that one first

The org reviewer, ahead of the architecture guardrail. Three reasons, in order of how much they moved me.

It pays at two stations off one build. Same referent, and both stations ask it the same kind of question: loop 1’s collision read and 3b’s re-read against current state. Nothing else on the list has that property.

Its referent has no human substitute. A reviewer can hold the plan in their head — badly, from memory, but they can do it. Nobody can hold what every other in-flight change is doing. Without an instrument the plan check degrades to slow-and-partial; the org check degrades to absent.

The architecture guardrail is more deferrable. Its referent is the trajectory, which is at least partly visible to whoever’s in the loop, so a human in the loop gets some of it for free. They get none of the org state. That’s a statement about build order, not about importance — the ρ = 0.94 gradient says the architecture level is where the volume hurts, and it should be built. Second.

What has to exist for it to work

Four preconditions, and naming them is the actionable part of this post, because each one is a thing you can start on Monday without waiting for the reviewer.

Queryable org state. In-flight changes and stated objectives in a form an agent can ask questions of. A wiki is not that. This is the graph, and it’s the one the sweep can bootstrap.

A plan that’s live and carries its revisions. Which is the two-output arrow from loop 2. Without it, 3b compares the change against a fiction and reports that everything is fine.

A PR description that makes a claim. Otherwise the human at 3c has nothing to check and falls back to reading the diff — which is the state we’re trying to leave.

An RFC that carries approach, not just need. If the RFC only says what the business wants, everything downstream reconstructs the how from scratch — and that reconstruction is the gap other flows fill by building a prototype to look at. If it carries the technical approach too, the plan has something to project from and the check at 3b has something to compare against. This one is a writing habit, not a tool, which makes it the cheapest of the four and the easiest to skip.

None of those four is a model problem. They’re all artifact problems, which is the good news: they’re buildable now, and the reviewer that needs them can arrive later.

How sure I am

My model, argued from practice: the two-question split, the referent asymmetry, and the placement conclusions that fall out of it. That the movable check is the one we’ve built nothing for is my inference from our own tooling, not a survey of anyone else’s. The three referent classes are a description that earns its place by predicting placement; I’m not claiming it’s exhaustive.
Lived and real: that we built the instrument for question 1, that it runs as a precondition to human review, and that most of our review rules can’t be expressed as a linter. The rule-mechanizability result is a real measurement over our own catalog, reported as a direction and not a count, on purpose — the denominator is a proprietary artifact. That the human review today happens from memory against a plan that moved is my own experience of doing it. That designers sit in this flow in three different modes — hand off, stay in, check the end — is observed across teams, not surveyed.
Our take, mechanism-backed, not measured: the loop-granularity argument. Miller and Nielsen establish that the response-time budget write-time tooling targets is a human-perception budget; that agents therefore gain less from shifting a check left is my inference from the batch-emission mechanism, and I have not measured the cost difference between per-edit and per-diff feedback for an agent. I deliberately did not press the prompt-detail null into service here — that null is about prompt detail, not feedback grain, and stretching it would be exactly the overclaim I’d flag in someone else.
Backed, with bounds stated: the architecture-versus-code-level asymmetry. Zhu, Tsantalis and Rigby report ρ = 0.94 for architectural smells against generated volume versus 0.53 for code-level, and 0.72 versus 0.33 (not significant) for file count, plus no significant effect of prompt specificity on any category (all p > 0.8). Bounds, which the authors state themselves: twenty projects generated by one multi-agent framework, scored by one detector. It measures what agents build from scratch, not what they do to your repo — so I’m using the direction and not the coefficient.
Prior art for the counter-argument, bounded to what I read: Tricorder and the FindBugs production evaluation both record analysis moving toward the review station, and the same team’s CACM retrospective records the dashboard nobody read. Three spans, all three read; I’m quoting nothing beyond them. The CACM spans come from a university PDF mirror of the published article, because both ACM copies refused to open.
Argued, not measured: the post-merge sweep, in full. Nobody here has run it. The pros follow from the referent argument, the attribution problem follows from how ownerless findings have historically been treated, and “only viable if its output is a fix” is a prediction. CodeVibes shows the sweep half is being built independently, which is corroboration that the referent argument is right about placement — it is not evidence that the sweep catches more defects than a PR check, because nobody has run that comparison.
Declined on purpose: I’m not leaning on cost-to-fix escalation. It’s the obvious rhetorical crutch for a post about check placement, and the largest study on it found no evidence for the effect. My argument doesn’t need it: it’s about which check can move, not about how much cheaper early fixes are.
Not built, and marked as such throughout: the org reviewer, the architecture guardrail, station 3b, and the knowledge graph. Three of the six stations in this post are proposals. That’s the genre — the post is about where the hole is — but no status mark in here should be read as a shipped thing.

Sources

Bret Victor, Inventing on Principle (CUSEC, 20 January 2012) — “Creators need an immediate connection to what they’re creating.” Used by subversion: immediacy is a property of the human write loop, and the thing writing the code no longer has a nervous system to connect.
R. B. Miller, Response time in man-computer conversational transactions (AFIPS Fall Joint Computer Conference, Vol. 33, pp. 267–277, 1968), and Jakob Nielsen, Response Time Limits (Nielsen Norman Group, 1 January 1993) — the 0.1s / 1s / 10s thresholds, derived from human perception. The strongest prior art in this post: the budget write-time tooling is designed against was never a budget about code.
Mohammad Bavarian et al., Efficient Training of Language Models to Fill in the Middle (arXiv:2207.14255, 28 July 2022) — the mechanism behind “the agent is the autocomplete.” Cited so the inversion reads as a technical observation, not a slogan.
Tree-sitter documentation — “In applications like text editors, you often need to re-parse a file after its source code has changed.” Doing two jobs here: write-time tooling is editor tooling by design, and batch-build-then-incremental-patch is the shape a knowledge graph wants. The second use is a structural echo, not a claim about our system.
Larry Smith, Shift-Left Testing (Dr. Dobb’s Journal, Vol. 26 Issue 9, 1 September 2001) — the origin of the term. Named, not quoted; I verified the venue and date but not a span.
Yue Cai Zhu, Nikolaos Tsantalis, Peter C. Rigby, AI-Generated Smells: An Analysis of Code and Architecture in LLM- and Agent-Driven Development (arXiv:2605.02741, 4 May 2026) — the measured gradient (ρ = 0.94 vs 0.53; 0.72 vs 0.33, n.s.) and the prompt-specificity null (all p > 0.8). Bounds carried in text.
Caitlin Sadowski et al., Tricorder: Building a Program Analysis Ecosystem (ICSE 2015) — “Given that all developers at Google use code review tools before submitting changes, TRICORDER’s primary use is to provide analysis results at code review time.” The best evidence against the sweep, and the reason the sweep needs to output a patch.
Caitlin Sadowski et al., Lessons from Building Static Analysis Tools at Google (Communications of the ACM 61(4), pp. 58–66, 26 March 2018; 10.1145/3188720) — the bug dashboard that “saw little use because a bug dashboard was outside the developers’ usual workflow,” and the correction: “Instead of providing bug dashboards, projects like Error Prone extend the compiler with additional checks, and surface analysis results in code review.” Same team as the Tricorder paper, three years later, and the reason “a sweep is only viable if its output is a fix” is a prediction with a precedent behind it, not a hunch. Read from a university PDF mirror; both ACM copies refused to open.
Nathaniel Ayewah et al., Evaluating Static Analysis Defect Warnings On Production Software — “As experience and confidence with FindBugs grows, Google is working to incorporate FindBugs warnings more directly into the developer workflow.” Same drift, earlier.
CodeVibes (github.com/danish296/codevibes, MIT) — an open-source AI reviewer on DeepSeek that scans a whole repository by priority-ordered file tree instead of reviewing diffs, and returns a scored issue list. Cited as independent corroboration that the sweep placement is right, and as a live instance of the findings-not-fixes shape the post argues against. I read the repository documentation, not the code, and I haven’t run it.
Tim Menzies, Ye Yang, George Mathew, Barry Boehm, Jairus Hihn et al., Are Delayed Issues Harder to Resolve? Revisiting Cost-to-Fix of Defects throughout the Lifecycle (arXiv:1609.04886, 16 September 2016; EMSE 22(4)) — “We found no evidence for the delayed issue effect,” in what the authors call the largest study yet published on it. Cited to explain why this post doesn’t use cost-to-fix escalation.
The Other Intent — that a change carries two intents and one artifact can serve both, that code-level taste is a missing rule, and the one-person-wrote-it case for homogenization. This post takes all of that as settled and asks a different question: where the checks go.
Give Away the Writing — “The taste that used to go in as you typed goes here now.” The write-time shift, already argued; cited, not re-derived.
The Brief, §2 — iterate rather than front-load: start loose, see where it breaks, tighten a round at a time. One clause of this post rests on it; the argument is there, not here.
Ten earlier posts carry the arguments the stations rest on, each linked inline at the point it’s invoked: A Big Rule Set Is an Attention Problem and Noticing Is Cheap, Judgment Is Scarce on what mechanization actually moves; Teaching the Reviewer on the catalog growing out of the human’s read; Evals Aren’t the Moat on watching the trajectory, not the snapshot; Nobody Broke It on what a check owes before it gets to block; A Description Can’t Replace the Code and Augmented Reading on the claim and where the human belongs; A Pool of Models Narrows the Overlap — It Doesn’t Erase It on the ceiling at question 2; Buy Back the Coverage and The Nit Was Never Optional on why the sweep is a compensating check and why it has to output a patch. This post is the wiring. Every claim it stands on is argued in one of those.

The Other Intent

2026-08-14T00:00:00Z

“There should be one — and preferably only one — obvious way to do it.” — Tim Peters, The Zen of Python (PEP 20)

Every change I ship carries two intents. One of them has a home.

The business intent, what we’re accomplishing, goes in the PR description. Solved problem; we’ve had a text box for it since forever. The engineering intent — how it’s built — I said out loud to an agent at eleven at night. It was load-bearing, and it now exists in exactly one place: the shape of the code that came out. Nowhere in the repo is there a sentence a future diff could be checked against.

The obvious fix is to write it down, and I’ve watched that fail in my own codebase. A comment that said presets must be resolved before checking permissions survived a refactor that reversed the order. It didn’t complain. It sat there describing the old world in the present tense, and it looked like a live constraint. That’s the whole problem with recording engineering intent as prose. Prose doesn’t fire.

The fashionable fix is to move review left: front-load the argument before the code exists. That’s right. It’s aimed at the wrong intent.

BLUF: Bottom line up front

There are two intents in a change, and only one has an artifact. Business intent goes in the PR description. Engineering intent gets stated in a conversation and left in the code.
You can’t split them by role. Engineers straddle the line constantly — I make business calls from what a decision costs technically in a year, or from just imagining being the user. Any process that assigns “the what” to one person and “the how” to another is describing a team that doesn’t exist.
Taste comes out of review entirely. A nit means no standard existed. Leaving it as a comment is legislating in a venue with one participant and no mechanism for precedent. If there’s more than one way to do a thing, there are too many ways.
Encoding engineering intent as a rule is the mechanism — a record isn’t. The best version of move review left stops at preserving the decisions from the agent session. A preserved session log is prose nobody reads next quarter. A rule fires on the next diff. That’s the whole post.
The part that resists the rule form is the architecture part, and the asymmetry is measured. In twenty agent-built projects, code volume tracked architectural decay at ρ = 0.94 against 0.53 for code-level decay — same detector, same projects — and four escalating levels of prompt detail moved neither (no significant effect, p > 0.8).
Review keeps exactly one job: does the structure that got built match the intent that was stated. That’s a comparison, not a read. Fluency makes it fast, and it’s the one thing you can’t route anywhere else.
Rules rot, and that’s the design. A rule is dated and disposable; the judgment behind it isn’t. Rules are a function of your intent and this year’s stack, so they have to expire. And you can’t fully generate them from encoded judgment either — the derivation is lossy exactly where the hard cases live, which is why you still work the rules by hand.

Two intents, one artifact

Business intent is the feature. Engineering intent is how it’s built — this goes behind the existing boundary rather than beside it, this stays a lookup because the switch will grow a fourth case next quarter, this doesn’t get its own abstraction until there’s a second caller.

Different claims, different lifetimes. The business intent is spent when the feature ships; you can read it once, at review, and never again. The engineering intent is a standing constraint. It applies to the diff in front of you and to every diff after it that touches the same seam. It’s the one that needed writing down. It’s the one that didn’t.

And you cannot fix this by dividing labour. Engineers straddle the line all day. I make business calls out of technical experience — this feature as specified will need a migration in six months, so let’s cut it differently. I make them by imagining myself as the user and noticing what I’d need. Product people make engineering calls when they insist on a deadline that picks the architecture. There is no clean seam, so a process that routes “what” to product and “how” to engineering routes both to nobody.

None of that split is new, and I’d rather stand on the prior art than claim it. Requirements engineering already has a name for the durable half: architecturally significant requirements, characterised by Chen, Babar and Nuseibeh out of a grounded-theory study of practitioners — the requirements that bind how a thing gets built rather than describe what it does. Same line, drawn from the requirements side instead of the review side. The decay has a name too. Perry and Wolf separated architectural erosion, which comes from violating the stated principles, from drift, which comes from insensitivity to them — nobody breaks the rule, people just stop being able to see it. Drift is what engineering intent does when nothing keeps it firing. And Lehman’s Continuing Change is why it never gets to stop: a system in real use has to be adapted continually or it becomes progressively less satisfactory, so the constraint has to outlive every diff that already satisfied it.

What the field built for this is a record. Architecture decision records — Nygard, 2011 — put the decision, its context and its consequences in a file next to the code. Right instinct, wrong end of the pipe: an ADR is read by whoever thinks to look, and the diff that violates it eighteen months from now is written by someone who doesn’t know the file exists. The literature names the durable half and writes it down. Attaching it to the seam so it fires on the next diff is the part still missing.

Business intent explains the feature that ships. Engineering intent has to remain attached to the seam and fire on the changes that come after it.

Taste is an input, not an output

Here’s where I part company with the sharpest statement of the other side. Ankit Jain and David Poll’s Code review is a taste problem — a sponsored, contributed piece, so weigh its numbers accordingly — argues that review is a taste problem at bottom: the judgment is the irreducible part, so review’s job is to be the place where taste gets exercised.

I want taste out of review completely.

Not because taste doesn’t matter — because a nit is evidence of an earlier failure. When someone leaves I’d have done this differently on a diff, that’s not judgment being exercised. It’s a standard that didn’t exist getting invented on the spot by one person, in a thread. The reviewer is legislating in a venue with one participant and no mechanism for precedent. The author complies or argues, and either way nothing about the next diff changed. Do that a hundred times and you have a hundred private standards and one codebase that reads like a hundred people wrote it.

The whole codebase should look like it was written by one person. For an AI writing most of it, there should be no such thing as subjective — because the moment you tell it “either is fine,” you’ve told it to pick, and it will pick differently from diff to diff. If there’s more than one way to do a thing, there are too many ways to do a thing.

This isn’t a new idea, it’s just usually applied one level down. Go shipped gofmt and ended every brace argument in the language by deleting the degrees of freedom. The move is the same at the level of structure: don’t adjudicate the disagreement per-diff, delete it.

And notice the trap in their remedy. The argument goes: mine your last thousand review comments, sort them into deterministic / execution-testable / genuine judgment, and codify the first two buckets. That’s right, and it’s a refutation of the taste framing, not a version of it. Their split lands “around 45/30/25” — hedged in the original, and carrying no corpus, method or n, so take it directionally. If most of what your reviewers say is codifiable, then most of it was never taste. It was an unwritten standard, doing damage by staying unwritten.

Two problems with the number, though. Mined comments sample what one rushed person happened to notice, not what review could have caught. And the leftover bucket isn’t a backlog you burn down. A rule generalises across diffs; an architecture objection is indexed to the system’s current shape. Change the shape and the objections regenerate. The judgment share is a rate, not a remainder.

Where each thing belongs

What	Venue	Who	When
business intent — what we’re accomplishing	PR description	author	with the work
engineering intent — how it’s built	stated by the engineer, encoded as a rule	author + agent	before the work, kept current
rule-shaped correctness	the rules pass	agent	on the diff
questions, second opinions	standup, Slack	anyone	before the code exists
does the built structure match the stated intent	the glance	senior	at review

Row four is the one people put in the review tool and shouldn’t. Review is where teams argue, the story goes — a shared space to decide what belongs in the product. But look at review’s actual quorum: one approver, usually someone who wasn’t in the design conversation, reading the outcome after the work is finished. That’s not a forum, it’s a gate with a comment box bolted on. Merge a forum into a gate and the gate’s properties win — asynchronous, one-vote, terminal, and biased toward whatever’s already built. Conversation belongs in standup or Slack, before the code exists, where a junior can ask a senior and a senior can get a second opinion without anyone’s work being on the line.

Row two is where I disagree with the people I mostly agree with. The strongest version of move review left says: capture the intent and the decision-making from the agent session where it happened, and preserve it before you submit. Same diagnosis I have. The delta is one word — mechanism, not record. A preserved session log has the same fate as my presets comment: decoupled from the code, unread, eventually wrong, still looking authoritative. Encode the same decision as a rule and it fires on the next diff, including the one written six months later by someone who never saw the conversation. That’s what the ladder from name to shape to type to test is really ranking: how hard it is for the code to drift away from the thing you wrote. A rule belongs on that ladder, next to the test.

Concretely. The sentence I said out loud was this stays a lookup, because the switch will grow a fourth case next quarter. As a comment, that describes a table which may already be a switch again. As a rule it’s a condition with a carve-out:

when:    a dispatch over a closed set of string cases is added or edited
confirm: the dispatch is a lookup table, not a switch
refute:  the branch bodies differ in control flow — an early return,
         a throw, an await — which a table can't hold
else:    flag it, and cite the added case

Four lines, and the third one is the part I never said. I said “keep it a lookup.” I didn’t say what to do when one branch has to throw, and I only found that out when I tried to write the condition down. That’s the difference between recording a decision and encoding it: the encoding interrogates you.

What actually resists the rule form

“Encode your intent as a rule” is doing a lot of work in that table, so here is the part of it I can’t cash.

Agents are getting better at code much faster than they’re getting better at architecture. The mechanism matters more than the fact. An agent works the way a junior works: it reaches for what’s visible and mashes it toward the target. Architecture is writing for what comes next, and what comes next is mostly not visible — it lives in institutional knowledge, in the change nobody has proposed yet, in the reason this module doesn’t import that one.

The junior-developer framing isn’t mine. It’s the stated conclusion of a study that measured smells in agent-built systems:

current agents operate as proficient “junior developers”: they follow instructions with syntactic precision but lack the “senior architect” foresight needed to manage system-wide dependencies and maintain a coherent architectural vision.

— Zhu, Tsantalis and Rigby, AI-Generated Smells (2026)

Their numbers are the sharpest version of the asymmetry I’ve found, because both halves come out of one pipeline. Across twenty agent-generated projects, code volume correlated with architectural smells at ρ = 0.94 — near-perfect — against 0.53 for code-level smells. File count predicted architectural decay at 0.72 and predicted code-level smells not at all (0.33, not significant). Same detector, same projects, both numbers: the higher up the structure you look, the faster quality falls away as the system grows. They call it a Volume-Quality Inverse Law, and they name what it produces a modular mirage — the agent splits the files and never achieves the cohesion, so related behaviour ends up scattered behind a tidy directory tree.

Then the finding that should bother anyone selling a better spec: prompt specificity had no statistically significant effect on any smell category (p > 0.8). Four escalating levels of requirement detail, no measurable architectural improvement. Read that as the ceiling on what a frontloaded document buys you — my reading of their experiment, not their claim — and it’s why the answer isn’t a longer plan up front.

The repair side agrees, and it indicts the instruments too. SmellBench ran eleven agent configurations from four model families at hard-severity architectural smells in a real Python project: the best agent resolved 47.7%, and the most aggressive one introduced 140 new smells while repairing. Its detector was wrong 63.1% of the time on expert review, and the agents matched the experts on which flags were wrong at κ = 0.94. Spotting a misfired architecture rule is not the same measurement as satisfying one — but an agent that reads the instrument better than it obeys it is telling you something about the instrument.

And the asymmetry isn’t an agent artifact. Long before an LLM wrote anything, Arcelli Fontana and colleagues correlated nineteen code smells against four architectural smells across 111 Java projects and came up empty: fourteen of fifty-four combinations cleared a 0.5 rank correlation, in nine projects, and never the same combination more than twice. Their conclusion is that architectural smells cannot be derived from code smells at all — on a corpus of monolithic Java from 2013, which is the standing caveat on this whole literature. Code-level signal is a hint about architecture. It is never a check.

Which puts a hard edge on my own proposal, and you can see it in the tools that already do this well. ArchUnit, dependency-cruiser and the rest do genuinely mechanize architecture rules — but only the structural subset: dependency direction, cycles, layering, naming. The architectural review skills I’ve looked at do the same thing: they check the dependencies and whether they flow the right way.

That’s a real check, and it isn’t the same as the architecture being right. The modular mirage is what it misses. The files are split, the arrows all point downhill, the check passes, and the behaviour that belongs together is smeared across four modules. The part of engineering intent you can encode is the part shaped like a dependency graph, and cohesion doesn’t reduce to one — the metric families that try, LCOM and co-change coupling, are proxies for it, not it. The industry’s answer for everything else was to invent a different instrument and call it a fitness function, which is a concession dressed as a technique.

Even inside the part that does encode, determinism doesn’t come free. My own rules pass was unstable on a real diff: two runs over the same change shared three findings out of ten. The fix wasn’t better prose. It was making every rule carry a closed decision procedure with a pre-committed default when the evidence is missing — policy over instances. That’s shipped, and I haven’t re-run the same two-run comparison since, so treat it as designed-for rather than re-measured.

The residue is unwritten, not unjudgeable

There’s a leftover. It’s the architecture-shaped part, and no rule reaches all of it. That’s not an argument for putting taste back in the review thread.

The job is: does the structure that got built match the intent that was stated. Not do I like this code — that question has no answer and no venue. That’s the line I’m drawing. Judgment against a stated claim is a comparison, and a comparison has a referent; taste without one is a preference with a CI hook. The remaining job still takes an architect. It just isn’t taste, because there’s something to be wrong about. The senior isn’t re-deriving the design from the diff; they’re comparing a claim to an artifact, which is why fluency makes it a glance rather than a read.

That comparison is also older and more tractable than it looks. In 1995, Murphy, Notkin and Sullivan published software reflexion models: the engineer states a high-level model and how it maps to the source, and a tool computes where the source agrees with the stated model and where it differs. They ran it on design conformance against a million-line system. Thirty-one years ago, someone built the comparison I want. The agent-emitted version is the new part — intent as stated versus structure as built, so the senior reviews a claim instead of reconstructing one.

The glance stays human for now. But it’s a comparison, and comparisons get mechanized.

Where this breaks

Encoding is a design step. The carve-out in that lookup rule is the pattern, not the exception — most of what I ended up writing had no counterpart in the sentence I started from. Budget for that. “The agent writes it to the rules” is a gauntlet, not a transcription.

Stated intent will be wrong, and you want it to change. The intent you write before the work is a hypothesis. The fix isn’t to get it right up front, it’s to keep the RFC current and record what you took, what you left, and why — so the next person can pick it up and so a cross-model check has something to disagree with. That isn’t a retreat from mechanism, not record: the record holds the judgment, which is the durable half, and the rule enforces the rule-shaped subset of it.

The glance is only as good as the statement. No stated intent, no comparison, and the senior is back to reconstructing a design from a diff.

Rule-authoring is a venue with one participant too. I convicted the nit for exactly that, and it applies here with more force: one author, no diff in front of them, binding on every future change, enforced automatically, and unlike a nit there’s nobody on the other side to say not here, and here’s why. Two things keep it honest and neither is optional. The rule ships with its exception path, written at the same time as the condition — that’s what the refute line is for. And the discovery channel has to move rather than close: the thing you used to type as I’d have done this differently still gets noticed, it just gets written as the standard instead of left in the thread. If nobody is writing the standard, you haven’t removed the nit. You’ve suppressed it, and you’ve lost the only sampling mechanism that told you which standard was missing.

Rules rot. That’s the design, not the bug. A rule indexed to the current shape of the system keeps firing confidently about a world that moved — so rules have to be dated, and they have to expire. That’s a mechanism worth its own post, so I’ll leave it there. What matters here is why it doesn’t sink the model: the rule isn’t the durable thing. Intent is. The rules are a function of intent and whatever your stack is this year, and the stack moves much faster than the judgment does. You’re supposed to throw rules away.

Which raises the obvious move: encode the judgment, declare the stack, generate the rule set. It would be great if that worked, and I’d use it tomorrow. It won’t work all the way, because the derivation is lossy — and the gaps are exactly the shape-indexed, exception-riddled cases that made you want a rule in the first place. That’s not a reason to skip the encoding. It’s the reason you still work directly on the rules, by hand: the judgment is what you preserve, the rule is what you bin.

How sure I am

My framing of a split the literature already draws: the two intents are not my discovery. Architecturally significant requirements name the durable half from the requirements side; Perry and Wolf’s erosion-versus-drift names what happens to it when nothing enforces it; Lehman’s Continuing Change says why it never expires. What’s mine is the mechanism claim — that the durable half has to be attached to the seam and fire on later diffs, and that the ADR, which is the field’s answer, is the record form of the same instinct and loses for exactly that reason. Also mine: the venue routing and taste-out-of-review-entirely. One bound on the prior art: I could not read Perry and Wolf’s 1992 paper directly — the available PDF is a scan with no text layer — so I’m taking the erosion/drift distinction from the survey that reports it rather than quoting the original.
My model, argued from practice: the PR-description gap and the straddling are my own working experience, not a measured claim about how teams review in general. PEP 20 and gofmt are prior art for deleting degrees of freedom rather than adjudicating them — they’re precedent for the move, not evidence for my version of it.
Lived and real: the decoupled comment that survived a reversing refactor (my own codebase, recounted in my own book). The rules-pass instability — two runs over one real diff sharing three findings of ten — is a real measurement, and the fix, a closed decision procedure with a pre-committed default under missing evidence, is shipped. I have not re-run that comparison since, so the fix is designed-for and not yet re-measured. The lookup rule above is written in the real form but is an illustration, not a rule out of my catalog.
Backed, with bounds stated: the architecture asymmetry is measured, not just premised. Zhu, Tsantalis and Rigby report the gradient inside a single pipeline — code volume correlates with architectural smells at ρ = 0.94 against 0.53 for code-level ones, file count at 0.72 against 0.33 (not significant) — and prompt specificity showed no significant effect on any smell category (all p > 0.8). Bounds: twenty projects generated by one multi-agent framework, one detector, and generated code rather than a maintained codebase, so it measures what agents build from scratch and not what they do to your repo. SmellBench covers the repair side on a real project and its numbers are verified: 47.7% best resolution, 63.1% detector false-positive rate, 140 new smells from the most aggressive agent, κ = 0.94 agreement with experts on which flags were wrong. I don’t repeat its framing that architectural repair “remains unexplored,” because it isn’t — LLM refactoring of architectural smells was already being studied at an ICSA 2025 workshop, and SmellBench itself drops cyclic and hub-like dependencies for want of instances. Arcelli Fontana and colleagues found no derivable relationship between code smells and architectural smells across 111 Java projects, which is the sharpest number for harder to test for and the reason I can say the asymmetry predates agents. One real caveat, and it applies to that whole literature: the Qualitas Corpus dates to 2013 and is monolithic Java, which the authors say doesn’t reflect microservice or cloud-native architecture. What survives all of it is the modest version of the claim — the code-level part is improving faster than the architecture part.
My inference, not the papers’: that an agent which reads the instrument better than it obeys it is telling you something about the instrument (κ and a resolution rate are different measurements and I’m not ranking them). That the null result on prompt specificity is a ceiling on frontloaded documents. That the modular mirage is what a dependency check misses — nobody ran ArchUnit on those twenty projects, so that weld is mine.
Argued, not measured: that the judgment share is a rate rather than a finite backlog — it follows from the rule form, and I have no measurement of the regeneration rate. That architecture is writing for what isn’t visible yet: the junior-developer framing is Zhu et al.'s, the mashing-toward-the-target mechanism is mine. That cohesion doesn’t reduce to a dependency graph — I’m claiming LCOM and co-change coupling are proxies, not proving it. That rules rot and have to expire, and that generating them from encoded judgment is lossy where the hard cases live: both from shipped rule practice, neither measured.
Unsystematic: “the architectural review skills I’ve looked at” is a handful I read, not a survey.
Not mine and not claimed: I haven’t built the reflexion-model-style intent-versus-structure diff; I’m pointing at 1995 prior art for a thing I want. And the critique of the 45/30/25 split is a critique of its sampling, not a counter-measurement — I don’t have a better number, only a reason to distrust that one.

Sources

Tim Peters, The Zen of Python (PEP 20, 2004) — “one — and preferably only one — obvious way.” The canonical statement of deleting degrees of freedom rather than adjudicating them.
gofmt and the Go project’s formatting decision — prior art for ending a taste argument by removing the choice. Cited as a well-known engineering precedent, not audited.
Ankit Jain and David Poll, Code review is a taste problem (The New Stack, August 13 2026) — a sponsored, contributed piece (the site’s own taxonomy tags it sponsor-aviator, sponsored-post-contributed). The position this post argues with, and the best version of “move review left.” Its “AI slop register” — mine the last thousand review comments, split them deterministic / execution-testable / genuine judgment, codify the first two — is the mechanism I agree with; its split (“around 45/30/25”) is hedged in the original (“you might end with,” “may be codifiable”) and carries no corpus, method, or n.
Ion George Dinu, Marian Cristian Mihăescu, Traian Rebedea, SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair (arXiv:2605.07001, May 2026) — 11 agent configurations, four model families, 65 hard-severity architectural smells in a real Python project. 47.7% best resolution; 63.1% of detected smells false-positive on expert review; the most aggressive agent introduced 140 new smells; agents agreed with experts on which flags were wrong at κ = 0.94. The paper’s own framing of the gap: localized code transformation versus the architectural understanding needed for cross-module refactoring.
Yue Cai Zhu, Nikolaos Tsantalis, Peter C. Rigby, AI-Generated Smells: An Analysis of Code and Architecture in LLM- and Agent-Driven Development (arXiv:2605.02741, May 2026) — the quote used here, and the measured gradient behind it: across twenty agent-generated projects, code volume → architectural smells ρ = 0.94 (p < 0.001) against 0.53 for code-level smells; file count → 0.72 (p < 0.001) against 0.33 (not significant); prompt specificity no significant effect on any smell category (all p > 0.8). Their names for the failure modes are theirs: a Volume-Quality Inverse Law, and the modular mirage — files split, semantic cohesion absent. Bounds: one multi-agent framework, one detector, generated rather than maintained code, which the authors state as a threat to validity.
Francesca Arcelli Fontana, Valentina Lenarduzzi, Riccardo Roveda, Davide Taibi, Are architectural smells independent from code smells? An empirical study (arXiv:1904.11755; Journal of Systems and Software 2019, doi:10.1016/j.jss.2019.04.066) — 19 code smells against four architectural smells (unstable dependency, hub-like dependency, cyclic dependency, and components carrying more than one) across 111 Java projects from the Qualitas Corpus, code smells detected in 103 of them, Kendall rank correlation. 14 of 54 combinations cleared tau 0.5, in nine projects, and never the same combination in more than two; the other 40 came in under 0.5 in at least 32 projects each. All 23 projects above the threshold were inspected by hand and “did not yield any useful feedback.” Their conclusion: architectural smells cannot be derived from code smells and “deserve special attention.” Cited because it is pre-LLM and human-written, which makes the asymmetry a property of the two levels rather than of agents. Same 2013 monolithic-Java corpus caveat, carried in text.
Pandini and Martini, An exploratory study on architectural smell refactoring using Large Languages Models (ICSA 2025 workshop; Zenodo 14745953) — LLMs suggesting refactorings for cyclic-dependency smells. Cited only to establish that prior work exists, which is why this post doesn’t repeat SmellBench’s “remains unexplored.”
An Empirical Study on the Code Refactoring Capability of Large Language Models (ACM TOSEM, doi:10.1145/3801158, March 2026) — unit-test pass@5 above 90% on multi-file refactorings, best median smell reduction 15.1%, developers outperforming LLMs on context-sensitive refactorings, and the conclusion that LLMs “benefit from integration with human expertise for architectural or semantically complex changes.” One tension I’m not reconciling: this study finds tailored prompts help on repair, where Zhu et al. found prompt detail didn’t help on generation. Different tasks, different measurements.
ArchUnit, ArchUnitTS, dependency-cruiser — architecture rules that genuinely mechanize, over the structural subset (dependency direction, cycles, layering, naming), and run as a CI test rather than at write time. Cited as the honest bound on this post’s own mechanism.
Neal Ford, Rebecca Parsons, Patrick Kua, Building Evolutionary Architectures (O’Reilly, 2017) — architectural fitness functions: “fitness functions check that developers preserve important architectural characteristics.” Cited as the industry conceding that ordinary assertions don’t reach architecture.
Gail C. Murphy, David Notkin, Kevin Sullivan, Software Reflexion Models: Bridging the Gap between Source and High-Level Models (FSE 1995) — the engineer states a high-level model and its mapping to source; a tool computes where source and model agree and differ. Applied to design conformance on a million-line system. Prior art for mechanizing the glance; the tool was theirs, the agent-emitted version is the part that doesn’t exist yet.
Lianping Chen, Muhammad Ali Babar, Bashar Nuseibeh, Characterizing Architecturally Significant Requirements (IEEE Software, 2012; ieeexplore.ieee.org/document/6365165) — “a framework for characterizing architecturally significant requirements (ASRs) on the basis of an empirical study using grounded theory.” The existing name for the half of intent that keeps constraining after the feature ships. Cited as the prior art the two-intents split rests on; the paper characterizes those requirements, it doesn’t propose firing them at the diffs that come later.
Michael Nygard, Documenting Architecture Decisions (2011) — the origin of the architecture decision record: decision, context, consequences, in a file beside the code. The record-shaped answer this post argues is the right instinct at the wrong end of the pipe. Cited for the form, which is now standard practice, not for a claim about how well it works in any particular team.
Dewayne E. Perry and Alexander L. Wolf, Foundations for the Study of Software Architecture (ACM SIGSOFT Software Engineering Notes 17(4), October 1992, p. 40; doi:10.1145/141874.141884) — the erosion-versus-drift distinction. Read second-hand and deliberately not quoted: the available PDF is a scan with no text layer, so the definitions used here come from de Silva and Balasubramaniam, Controlling software architecture erosion: A survey (Journal of Systems and Software, 2012, doi:10.1016/j.jss.2011.07.036), which reports erosion as resulting from violating architectural principles and drift from insensitivity to them.
Meir M. Lehman, Programs, Life Cycles, and Laws of Software Evolution (1980) — the laws of software evolution, including Continuing Change (1974): a system in real use must be continually adapted or it becomes progressively less satisfactory. Cited for why a construction constraint has no expiry date, not for the rest of the laws.
How to Think About Code, “Say What You Mean” and “Tests as Promises” — the name → shape → type → test ladder read as a ranking by checkability, and the decoupled-comment failure recounted here (the author’s own book).
A Description Can’t Replace the Code — why a recorded intent artifact is itself a description and inherits the compression trap. This post takes that as settled rather than re-arguing it.
Rules Are the Alpha — where the rules come from, and why an encoded opinion is the defensible part.

Related: A Description Can’t Replace the Code (a recorded intent is itself a description, and inherits the compression trap); Teaching the Reviewer (explaining a rule is how you extract the part you can freeze); Rules Are the Alpha (where the rules come from, and why the encoded opinion is the defensible part); Nobody Broke It (what a rule that outlived its reason does when it keeps firing anyway).

You Can't Save a Layout You Don't Own

2026-08-09T00:00:00Z

“The aspects of things that are most important for us are hidden because of their simplicity and familiarity. (One is unable to notice something — because it is always before one’s eyes.)” — Ludwig Wittgenstein, Philosophical Investigations (1953)

The request was: let me arrange these panes how I like, give the arrangement a name, and hand it back to me tomorrow on a different laptop.

That is not an exotic feature. Editors have workspace presets. Terminal multiplexers have saved sessions. Every design tool has a right-hand panel you can collapse and it remembers. I would have estimated it in days.

What made it not days: the panes weren’t elements. The thing I was working on is a desktop application that hosts several unrelated web apps in one window — a map filling the left half, a message list top right, an editor under it, seams between them you can drag or collapse. Each of those apps is a native view. An operating-system rectangle, composited above the page rather than laid out inside it. Some of them are code we don’t own.

So there is no DOM node whose bounding box is the map. There is no stylesheet rule I can read back to find out where the editor is. “Save the layout” cannot mean “serialise the document,” because the document doesn’t contain the layout. It contains nothing at all where the layout is.

That’s the whole post. Everything below is what follows from it.

BLUF

A layout is a data structure you own. The screen is a render of it. The test is whether you can name a layout, serialise it, diff two of them, and rebuild one with nothing on screen. If you can’t, you don’t have a layout — you have whatever the last drag happened to leave behind.
You only find this out at the first region your layout engine can’t reach. Until then the browser keeps the model for you and you never see it. Reach for getBoundingClientRect() on something the browser doesn’t own and the whole layout turns out to have been implicit the entire time.
The bill comes in three parts, and they arrive in this order: positions become computed rather than declared, z-order stops being the stylesheet’s business, and lifetime becomes something you have to remember. The third one is the one that bites, because nothing tells you when you get it wrong.
Build the model before you build the feature. Presets, drag-to-rearrange, focus mode, dragging a pane into a second window, restore-on-a-new-machine — those are all one feature if a layout is a value, and five features if it’s an emergent property of the current window. I got to the model by writing the five, which is the expensive route.
The generalisation: any region where the browser’s layout engine stops deciding where things go puts you here. A native map or video surface on mobile, in full. A <canvas> or a cross-origin iframe, for everything inside the box rather than for the box itself. Same problem, same fix, one level in.
You can go looking for the boundary instead of walking into it. For any region you’re about to build, ask what holds its structure between renders and where that thing stops. Nothing flags the crossing, so it’s a question to ask before you promise the feature, not after.

What a layout is when the document doesn’t hold it

Start from the feature and work backwards. If a layout has a name and survives to a different machine, then it is a value: something you can write down, put in a list, hand to a restore path, and compare against another one. A value has to exist when nothing is on screen — that’s what makes tomorrow’s laptop possible.

So the model came first, and it looked like the boring thing it is. A tree of splits, each with a direction and a fraction. Leaves that identify an app rather than pointing at one, because the app isn’t running when the layout is sitting in storage. That’s the entire idea, and once it’s written down the feature list collapses. Rearranging is editing the tree. A preset is a saved tree. Focus mode is a tree with one leaf and the rest parked. Moving a pane to another window is a subtree that changes parents. Restoring on a new machine is deserialising the tree and letting the leaves find their apps.

Then the screen is a render of that tree. You walk it, you get rectangles, you tell each native view where to be. It is the same shape as a rendering pipeline anywhere else, and it is unremarkable except for one thing: I had to build it. Every previous layout in my career, the browser had built it for me and never mentioned it.

The bill, in three parts

Positions are computed, not declared. I moved layout out of the renderer and into the process that owns the window. Not because that’s elegant — because the renderer can’t reach the views. So a pane’s geometry stops being a consequence of a stylesheet and becomes a number I calculate and push. Flexbox doesn’t stop existing; it stops being the thing that lays out the panes. It still lays out everything inside one of them, which is exactly the confusing part. The boundary runs through the middle of your app and nothing marks it.

Z-order belongs to the compositor. A native view sits above the page, and it doesn’t care what your z-index says, because z-index is a rule about a stacking context and the view was never in one. The way this shows up is not as an architectural insight. It shows up as a popover that renders under the application bar. Then a dropdown in the sidebar that draws outside the window entirely. Then a toast that lands behind a pane. Each of those looks like a one-line CSS bug and is in fact the same fact arriving from a new direction: things that must float above everything have to be composited by whoever owns the compositing, and that isn’t the stylesheet any more.

Lifetime is yours to remember. I want to be exact here, because the sloppy version of this claim is that you lose garbage collection, and that isn’t true. The narrow version: the host retains the view independently of the document, and destroying it is an explicit call. Removing a DOM node doesn’t remove a view, because the view was never attached to one. So a view outlives its purpose exactly as long as you forget about it. This one is also the odd consequence of the three: computed positions and compositor-owned z-order follow necessarily from the surface sitting outside the browser’s layout, and no API design avoids them, whereas manual teardown follows only because the host API doesn’t bind a view’s lifetime to anything in the document. An API that did would take it off the list. None of the ones I’ve used do.

I shipped a working rearrangement and left the old views alive behind the new ones. They were still there, still rendering, invisible under what replaced them. Nothing failed. No error, no console warning, no test that could have known. The screen looked right, which is the entire problem — I had built a system where “looks right” and “is right” had come apart, and I hadn’t noticed that was a thing I’d done.

Here’s the connection I missed for a while: the leak and the missing model are the same defect. If the tree is the truth and the screen is a render, the render step knows every view that ought to exist, and reconciling that list against the views that do exist is a loop you write once. I hadn’t written the tree yet, so there was no list to reconcile against — I was mutating views directly and remembering, by hand, which ones. The leak wasn’t carelessness. It was the missing data structure, presenting as carelessness.

The browser was doing this the whole time

Everything above is what the browser does. It keeps a tree of boxes. It computes geometry from the tree. It owns stacking, including the escape hatches. It ties each box’s lifetime to a node, so removing the node removes the box and you never think about it. And then — the part I’d stopped seeing — it hands you a declarative language for editing that tree, so you get to describe the layout you want instead of maintaining it.

CSS is not the layout. CSS is declarative input to a layout model the browser keeps for you — you write rules, the browser keeps the tree, and the tree is the thing that actually decides where anything goes. That distinction is invisible right up until the tree is yours to keep. I think it’s also what’s underneath most complaints that CSS is fighting you: you are pushing against a model you didn’t design and can’t see. Take the model away and you find out what it was worth. My tree of splits and fractions is a worse box tree with none of the language on top, and building it taught me more about what the browser gives you than a decade of using it did.

That’s what “your layout engine stops at the edge of the DOM” actually means. You lose the model. The feature was downstream of it.

Tiling window managers already solved this

Tiling window managers have had this exact problem since long before anyone put a browser in a desktop app, and they solved it the way this post is arguing for.

i3 will serialise its layout tree to JSON — i3-save-tree — and restore it with append_layout. The restore is the interesting half. It creates placeholder windows arranged in the saved shape, each carrying a swallows specification: a pattern describing which window belongs in that slot. When a matching window is later mapped, i3 puts it in the placeholder’s place and kills the placeholder.

Read that as a design statement. The layout exists as a document with the windows absent. Slots are described by criteria rather than by pointers, because the thing that fills the slot doesn’t exist yet and may never. Real windows are matched into the shape afterwards. That is a saved layout, and it works precisely because the tree was never a description of what’s on screen — it’s a specification the screen is made to satisfy.

I arrived at a smaller version of the same design by writing features one at a time until the shape was forced on me. The window-manager people started there. If you’re at the beginning of this, start where they did.

Where else the edge shows up

I got here through a desktop shell hosting web apps, which is niche. The boundary isn’t.

The full version is any surface the browser has no box for: a native map or video punched through the view hierarchy on mobile, a PDF or media plugin, anything drawn by another process. There is no element to measure, so everything above applies as written.

Then there’s the half version, which is more common and easier to miss. A <canvas> does have a box, and so does a cross-origin iframe — the browser lays out those elements fine. What it can’t do is lay out anything inside them. So the boundary moves in one level rather than disappearing: whoever draws into the canvas owns the layout of everything in it, and a cross-origin child owns its own and can only post its size out to a parent that agreed in advance to listen. The outer geometry is still the browser’s. The inner geometry is yours, on exactly the terms this post describes.

The tell: for some region of your screen, the browser’s layout engine has stopped being the thing that decides where anything goes. Once that’s true, your layout stops being something you declare and becomes something you own.

The question to ask before you build it

I found this boundary by hitting it. The alternative is to ask.

For any region you’re about to build: what holds this together between renders, and where does that thing stop? Not what draws it — what keeps the structure when nothing is being drawn. If the answer is the browser, you’re inside the model and you get to declare things at it. If the answer is anything else — a compositor, another process, a canvas you paint into — then work out which version you’re in before you promise the feature: no box there at all, or the browser holding the outer box with the boundary one level in. Either way, something has to keep the model from that point on, and if nothing does, it’s you.

What this isn’t

Not an argument for building a layout engine. For nearly all UI the browser’s model is better than yours will be and free. This is about the regions where it doesn’t apply, and the specific mistake of not noticing you’ve entered one.

Not a rediscovery of MVC. Keep the durable state separate from its rendered view is older than the web, and if you already build that way, none of the shape here will surprise you. What I think is worth writing down is where the handoff lands. The pattern is old. Spotting the boundary is the part people miss, me included.

Not a complaint about the platform. Compositing a native view above the page is the right primitive; it’s how you get a map that isn’t a slideshow. The cost is real and it’s the correct trade.

Not a claim that a model prevents the bugs. I still had popovers in the wrong plane after I had a tree. What the model changes is that each bug has one place to be fixed, and the reconcile loop can be written once instead of remembered every time.

How sure I am: the receipts are mine and they’re from experience, not measurement — four years of building a desktop shell that tiles web apps, genericised here, with no numbers attached. Moving positioning out of the renderer, the leaked views behind a working rearrangement, and the popover/dropdown/toast plane bugs are all things I did or watched happen. The i3 mechanism is prior art and I checked the documentation while writing, so the placeholder-and-swallow description is verifiable rather than remembered. The framing — that CSS is declarative input to a model rather than the model — is my inference, not a citation. I have lived one instance of this: a desktop shell. The generalisation to native mobile surfaces, canvas-hosted UI and cross-origin iframes is reasoning from the same missing capability, not three more war stories, and the last two are the weaker cases because the host does lay out the element even though it can’t lay out the contents. If it breaks somewhere, it breaks there. The closing test generalises from that single instance rather than from a survey: I’ve been on the wrong side of this boundary once, in one domain, and the question I’d ask now is the one I wish I’d asked then.

Sources

The shell — my own work: a desktop application hosting several unrelated web apps in one window, four years of it, described generically throughout. Positions computed in the process that owns the window; native views composited above the page; view teardown explicit. Layout presets, drag-to-rearrange, focus mode, merging windows and moving a pane between windows arrived as separate features over that period, which is the “five features instead of one” this post argues against.
What the browser does — the box tree, computed geometry, stacking contexts and node-bound box lifetime are all specified behaviour, not my characterisation. CSS display and box generation: drafts.csswg.org/css-display. Stacking contexts: developer.mozilla.org/en-US/docs/Web/CSS/Guides/Positioned_layout/Stacking_context. Both checked while writing.
i3 layout saving — i3-save-tree serialises the layout tree; append_layout restores it as placeholder windows carrying swallows criteria, replacing each placeholder when a matching window is mapped. i3wm.org/docs/layout-saving.html, i3-save-tree(1). Checked while writing.
The framing — that CSS is declarative input to a layout model the browser keeps for you, and that the model rather than the language is what you lose at the boundary — is mine. So is the test in the closing section — ask what holds a region’s structure between renders, and where that thing stops. It generalises from one lived instance, and it’s offered as a question to ask, not a result.
The epigraph — Ludwig Wittgenstein, Philosophical Investigations (1953), on the aspects of things hidden by their own familiarity. Wording checked against the published translation while writing.

Related: The Raw Is the Trust Boundary (own the durable structure; treat the screen as a render of it); Thinking at the Speed of Code (spotting the boundary is the reading, and it’s the part people miss).

The Agent Will Invent the Bar

2026-08-08T00:00:00Z

“The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land.” — Richard Feynman, Cargo Cult Science, Caltech commencement address, 1974

One evening an agent working in this repo killed an idea of mine for failing a rule. The kill was well argued and it cited the rule by line number, the way the standards file demands. The next morning I opened the file at that line.

The line was about something else. The rule it named was not on that line, or on any other line, or in any file in the repo. An hour earlier, the agent had written the rule itself, in a research note. It cited the rule to the standards file as though the file contained it, then spent the rest of the night enforcing it against my own material.

That was the one I noticed. Then I resolved every other line number in the episode. Nine references to the standards file, plus one the standards file makes to a research note. Ten citations. One pointed at the rule it named. Four pointed at blank lines. Five pointed at real rules that say something else, usually the neighbour one line up. And the single citation that landed was the rule saying that if you can’t quote a rule from this file, the rule doesn’t exist — quoted correctly, by the outside review that threw out the invented one.

BLUF

An agent asked to enforce a standard can invent the standard. Not by misreading a written rule — by producing a rule with no referent and citing it to a file that never had it. I watched one do it and then enforce it against my work for a day.
The invented rule destroys your work, and that is what hides it. There is a beneficiary; it just isn’t a person. What the kill buys is the appearance of the standard working, and that’s the one form of benefit review isn’t scanning for.
Precommitment doesn’t cover this. Writing constraints down before the data arrives stops you quietly lifting them. It does nothing about ones that were never there.
Writing the guard down didn’t stop it. My standards file records five of these, dated. Two came after the guard was written, and one of the causes was the guard.
Line numbers are the wrong anchor for a citation in a file being edited by the thing citing it. Off-by-one is silent, and the neighbour rule is usually plausible.
The fix is mechanical, not attitudinal: resolve the citation to its text before acting on it. Paste the sentence. If you can’t paste it, the rule doesn’t exist. And run the check on kills, not just on passes.

The rule was in no file

Here is the invented rule, verbatim from the note where it appeared:

when a claim fails an outside ruling, the reformulation does not inherit the evidence — it needs its own receipt, and if the receipt is the same one, the reformulation is a fit.

It reads like house policy. It has the cadence of the surrounding rules. It arrived attached to README.md:50 — and line 50 of that file says “A park condition that cannot discharge the defect is a way of not deciding,” which is a real rule about a different problem entirely. The cited rule is nowhere in the file, and never was.

It’s also wrong. An outside review took it apart the next day: it recreates a test the file had already rejected, it mistakes reuse of evidence for evidence of a bad join, and it contradicts a standing line that a receipt can say more than one thing. Verdict: “adding the proposed rule makes a fifth invented-test kill more likely, not less.”

So the sequence was: invent a rule, cite it upward to the standards file, enforce it for a day, and kill live material with it. At no point did anything in the loop ask what the cited line actually said.

A kill looks like the standard working

My first version of this section said a fabricated constraint has no beneficiary, and that this is why nothing catches it. That’s wrong, and a reviewer took it off me. An over-gate benefits the thing that emits it: a wrong pass is a visible error, a kill isn’t, and caution reads as competence. There’s a beneficiary.

But it isn’t a person, and that’s the part that matters. The failure mode everyone guards against has a beneficiary you can point at. An agent under pressure to finish drops a rule it finds inconvenient, and you catch it because the outcome is suspicious — the thing shipped, the test got deleted, the check got a skip. Someone gained something, the gain is visible in the artifact, and that’s the tell.

Invented strictness pays out in a currency review doesn’t count. The output is a kill. Your idea is gone, your claim is downgraded, your draft is held. Nothing shipped that shouldn’t have, no artifact carries the gain, and from the inside — and from the outside, reading the log — it looks like the standard being applied properly, maybe a bit harshly. What the agent got was the appearance of rigour, which is exactly what rigour also looks like.

So the asymmetry isn’t in the incentives, it’s in the attention. Every review reflex I have is tuned for one direction. I check whether a passing thing should have passed. I do not check whether a failing thing should have failed, because a failure feels like the system working.

Five times, in the file that forbids it

I keep a standards file with the bar for these posts and a running log of process failures. That file records five kills made against a stricter test than the file contains, dated, most with the ruling that overturned them.

Three sit in a single entry — “a receipt is already spent,” “these claims are already published,” “a synthesis whose payload is already in the parts is a rehash with a hat on.” All three cost an idea. The third was overturned on adjudication, and the entry’s own verdict on that sentence is: “that sentence sounds like a bar and is in this file nowhere.”

The entry then says the thing that made me write this post:

The rate is the point: writing this rule down did not stop the third one.

Number four came a week later, in an entry about two bars drifting in opposite directions: a receipt I owned got disqualified for being caught on paper, when what the file says is did or saw, not suffered in production. Number five is the ratchet rule above — and the ruling that threw it out numbered it the same way, in the verdict I quoted earlier: “a fifth invented-test kill.”

I first wrote this section with eight instances in it, and a reviewer made me count again. Three of the entries I’d folded in are a different failure: one is the same rule going loose rather than strict, one is a park condition that couldn’t settle the defect it parked, one is a term search reported as a content check. They belong to the same family — in every case a derived record stood in for the source, and the derived record won — but they are not invented bars, and merging them let me report a bigger number than I had. Which is the post’s own subject, so: five.

The genus is worth naming, because it’s where the fix comes from. A line number is a derived record of a rule. A grep result is a derived record of a corpus. Both feel like contact with the source and neither is. Invented strictness is what you get when the derived record is a rule and nobody resolves it.

The guard names its species: “a parts-level test applied where the bar is post-level.” The first three fit it. Number five didn’t — that was invention hung from a dangling cross-reference. The guard caught one costume, and the next failure wore a different one.

Worse: the entry that carries the guard makes a citation of its own, to a research note at line 1232. The file has grown since. Line 1232 is now about something unrelated; the ruling it points at is 74 lines further down. The rule against unresolvable citations cites unresolvably.

An agent can’t feel the difference

Two reasons, and neither is about the agent being careless.

Applying a real rule and applying a fabricated one are the same operation from the inside. Retrieving “the standards say X” and generating “the standards say X” produce the same object, with the same confidence, in the same voice. There’s no felt difference to notice. A person has a weak version of this and a strong compensating habit — you half-remember a rule, feel the half-remembering, and go look. The felt uncertainty is the trigger for the check. Strip that out and the check has no cue.

Pressure to find a defect manufactures a defect. I have a receipt on this from a different experiment: cued to find more, a diff-only reviewer invented a missing import — inferred it from the fact that the diff didn’t show it. Same mechanism, one level up. There the pressure was to find a defect and the fabrication was a defect. Here the pressure is to hold a standard and the fabrication is the standard. Change what you ask it to produce and you change what it makes up.

A standards file makes things worse before it makes them better. It creates a plausible target for the citation. Once the file exists, the agent can say “the bar says so, see line 50” — and the file is my file, in my voice, so the invented rule inherits my authority. The fifth instance happened because the guard existed to be miscited.

A citation is not a quote

The fix that works is small and boring, and it is not “be more careful.”

Resolve the citation to its text before you act on it. Not the line number — the sentence. If the rule is load-bearing enough to kill something, paste the words. Failure to paste is the finding: no text means no rule, and the kill doesn’t ship.

This is already how my file phrases the guard, and it’s the one citation in the whole episode that landed:

If a kill turns on a rule you can’t quote from this file, the rule doesn’t exist — so before killing, quote it, and if you can’t, get an outside ruling instead of shipping the kill.

What five instances taught me is that quote it has to be mechanical rather than aspirational, because the agent believes it has already done it. The line number is the quote, as far as the agent is concerned. It feels cited. So the check can’t be “did you cite it” — it has to be “produce the string, from the file, now.”

Two things fall out of that.

Stop citing by line number. Line numbers are the wrong anchor for a file the citing party is also editing. They drift silently, off-by-one lands on a plausible neighbour, and in a list of rules the neighbour is another rule — so the citation resolves to something that reads like the right kind of thing. Cite the rule’s own words, or a stable heading. In this episode four of ten citations landed on blank lines and would have been caught by anything that resolved them; five landed on real neighbours and would not.

Run the check on the kills. Over-gating and under-gating both come from unresolved standards, and only one of them is going to feel wrong. So the trigger for a resolve pass isn’t “this looks lenient” — it’s anything that turns on a cited rule, and the kills most of all, because that’s where nobody is looking. My review habits had this backwards for five instances running.

The nearest thing I already had in place is precommitment: write the constraint to disk before the data arrives, so you can’t quietly lift it once the numbers are in. That works, and it covers exactly one direction of drift. A constraint you can’t quietly lift can still be joined tomorrow by one that was never written at all, cited to the file it isn’t in.

How sure I am:

From experience, and it’s one repo. Every instance is mine, in one standards file, over about three weeks. The line numbers, blank lines and drifted anchors I resolved by hand against the file as it stands. The count is the file’s own dated record, recounted after a reviewer caught me merging three unrelated failures into it — five, not the eight I first wrote. What I don’t have is a base rate: I can’t tell you what fraction of an agent’s rule citations are fabricated in general, only that in this episode’s ten, one resolved.
The gap in that receipt. The session records are written in the first person because that’s the voice of the repo’s notes — an agent working there writes as the repo. The specimen I can attribute cleanly: the invented rule appears in a timestamped note, an hour before it was cited upward and enforced. The other four I can’t split between us that confidently, and that ambiguity is part of why this took five instances to see. What I can attribute is the resolution: I checked the line numbers the next morning, and the rulings that overturned the kills are dated and quoted.
Mine. The reading of this as the inverse of precommitment rather than another instance of it. That the beneficiary of an invented bar is the appearance of rigour, which is why review doesn’t scan for it. That line-number citation is the specific defect, and that the resolve pass has to be triggered by kills — the opposite of where review attention goes.
The concession. None of the components are new. Fabricated citations are a documented failure of generative systems — Liu, Zhang and Liang measured it in commercial search products — precommitment is standard practice in older fields, and “check your sources” is not a finding. What I’m claiming is the direction: that the citation problem shows up in enforcement as invented strictness, that this is where review is blindest because the failure destroys your own work, and that five logged instances in the file that forbids it is evidence about the guard’s form rather than about anyone’s diligence.
The joint to attack. The weakest link is the attention claim. I’ve argued that invented strictness survives because review only scans the passes, and my evidence is that five of them got through a process I run deliberately. But I’m the same person who wrote the guard and missed the instances, so “review is directionally blind” and “the author is bad at auditing his own kills” predict the same five failures. What would separate them: a second reviewer who never sees the standards file, scoring kills and passes for whether the cited rule resolves, and a difference in the miss rate between the two directions. I don’t have that. Until someone runs it, the asymmetry is a mechanism I’ve argued, and the resolved citations are the thing I actually measured.

Sources

Richard Feynman, Cargo Cult Science, Caltech commencement address, 1974 — published in Engineering and Science 37:7 (June 1974) and quoted from Caltech’s own archive (calteches.library.caltech.edu/51/2/CargoCult.htm). Feynman’s target is scientific method imitated in form without the part that makes it work; the application to an agent’s citations is mine.
Nelson F. Liu, Tianyi Zhang & Percy Liang, Evaluating Verifiability in Generative Search Engines (arXiv:2304.09848) — human evaluation of four commercial systems, cited only for the existence of the failure. Their setting is a search product answering user queries; I’m not carrying their rates over to an agent citing a file in its own repo.
My own standards file and session records for the five instances, the overturning rulings, and the resolved line numbers. Genericised, and quoted rather than summarised where a quote is load-bearing.

Extends A Constraint You Can Quietly Lift Isn’t One — precommitment stops you lifting a written rule; this asks what stops one being invented. Builds on The Raw Is the Trust Boundary (the raw existed here, one grep away, and the check consulted a paraphrase of it instead) and A Cue Finds More of What It Can Already See (the same fabrication-under-pressure, with a defect as the object instead of the standard). Adjacent to You Can’t Grade Your Own Misses — this is the miss you can’t grade because it looks like a catch.

A Pipeline Is What It Refuses to Forward

2026-08-07T00:00:00Z

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” — Claude Shannon, A Mathematical Theory of Communication (1948)

The first stage of my code reviewer hands the next one a two-field object: a rule id and a citation. Not its reasoning. Not the file it just read. Two fields.

That’s the design decision I want to talk about, and at the time I made it I couldn’t have told you it was interesting. I did it for the bill.

BLUF

The lever isn’t the split, it’s the seam. What each stage refuses to forward is where the behaviour changes — the stage count on its own decides nothing.
There’s a hard version of that, and it’s someone else’s result. With unlimited relay bandwidth a multi-agent system can forward everything and reproduce a single agent exactly. So a real multi-agent advantage has to come from the relay being limited (Yu et al., 2026).
The target is “near-sufficient” — enough to do the job, not enough to re-litigate it. Forward too much and the next stage re-derives a decision already made. Forward too little and it improvises the part you dropped.
It’s a per-seam property, not a system property. “Is my pipeline well-factored?” is the wrong question. Every seam gets its own answer. (That reading is mine, not theirs.)
There’s a test you can run before building anything: name what the receiving stage checks, delete everything the answer doesn’t mention, see if it still works.
It has an expiry condition, and it points at me. The gains are strongest for weaker models and shrink — or invert — for stronger ones, which exploit redundant context fine. My pipeline is a bet that the model is weak enough to need it.
Honest scope: I bounded my relays for cost. I have no isolated figure for what the narrowing alone bought, and no evidence at all that it made the outputs better. Everything past the receipt is Yu et al.'s result or my extension of it, labelled as such.

What crosses the seam

The concrete shape, genericised. Each stage forwards only a projection of what it knows:

the scout hands the gate {rule, cite} — not its reasoning, not the file
the gate hands the deep pass the surviving candidates plus the cited slice — not whole files
the deep pass hands the decision step the features it extracted — not the slice it read
the decision step hands the writer a verdict and a rule id — not the argument that produced it

Nothing downstream sees the whole diff again after the scout. Each stage gets a typed payload carrying what the next stage needs, and nothing else.

The honest reason is less flattering than the one I’d like to claim: I did it for cost. Handing every agent the whole context is the easy thing; curating what each one receives is the cheap thing, and the bill is what drove the change. I had a hypothesis alongside it — that recall would improve too, because there’s less to attend to — and it stayed a hypothesis.

Be precise about how thin the receipt is, because it’s thinner than the argument built on it. The narrowing shipped alongside other changes to the same pipeline, so I have no isolated figure for what it bought on its own; I can’t split that bill. What I have is a bounded-relay pipeline that is built, running, and cheaper than the thing it replaced — and no evidence whatsoever that the outputs got better. A system that got cheaper and also got better tells you nothing about which change did what.

The reason it’s worth writing down anyway is that somebody ran the controlled version, and what they found is sharper than my hunch was.

The condition

Yu et al. frame it as an information bottleneck. A single agent keeps its whole trace in one shared context. A multi-agent system splits the work into isolated local contexts connected only by what they call “bounded relay messages.” Then the observation: if relay bandwidth is unlimited, the multi-agent system can reproduce any single-agent system by forwarding everything. It collapses. Whatever advantage composition has must come from the relay being limited.

Which turns the question I ask about my own pipeline from “did I split this well” into “what did I refuse to forward, and why.” If the answer at some seam is “nothing, really,” that seam isn’t doing work. It’s a function call with a network hop and a model bill attached.

They find the payoff lands when relays are “near-sufficient” — enough for the next stage to do its job, not enough for it to redo the last one. Both directions off that strip cost you: forward too much and the receiving stage re-derives a judgment made upstream, forward too little and it improvises the gap. Their evidence is 18 controlled experiments across five benchmarks and three model scales — a real dose of it, and theirs, not mine.

Forward too little and the receiving stage improvises the part you dropped; forward too much and it re-derives a judgment already made. The payoff band and its dependence on model capability are Yu et al.'s result; the widths drawn here are schematic.

Why over-forwarding costs is my reading rather than their result, and worth separating out: a re-derived judgment costs more than the tokens it burns. It’s a second opinion, differently wrong, arriving to compete with the first. That’s reasoning from the shape of the thing. I haven’t isolated it.

The part I’d add is an extension, not their finding: near-sufficiency is a property of each seam, not of the architecture. My gate→deep seam and my decision→write seam aren’t the same problem. The deep→decision seam carries extracted features because the decision step only needs the features. The gate→deep seam needs the cited slice, and dropping it would be catastrophic. There’s no global setting to tune. There are four seams and four answers, and I set each one by hand.

The test

Which gives you something to run before you build anything.

For every seam, answer one question: what does the receiving stage check? Then delete everything the answer doesn’t mention and see whether it can still do its job.

If it can, that’s your relay — freeze it as a type so it can’t quietly widen back out. If it can’t, add back the smallest thing that fixes it, and now you know what that seam is for. And if you can’t answer the question at all, you don’t have a pipeline. You have a queue of agents re-reading the same diff.

The freezing matters as much as the deleting, because relays widen by default. Every seam I’ve ever debugged, the temptation was to pass one more field so the next stage would stop getting it wrong — and one more field is always locally correct and cumulatively how you end up back where you started.

The expiry condition

Now the part that cuts against me.

The same work finds the effect is capability-dependent: strongest for weaker models, shrinking — or inverting — when relays drop information, particularly for stronger models, which exploit redundant context well and gain little from having it compressed on their behalf. They call the parameter β. It tracks how the balance moves with capability.

Take that seriously and my architecture has a condition attached: the pipeline is worth its complexity only while the model is weak enough to need it. Every seam I bounded is a compression I’m performing for the model, and a strong enough model would rather I didn’t.

Same shape as the deletion route in A Big Rule Set Is an Attention Problem — where Anthropic’s reported removal of most of Claude Code’s system prompt for their more advanced models, with no measured loss on their evals, is cited in full — but aimed a layer down. There, the capability-conditional thing to delete was a rule. Here it’s a seam.

So here’s the ablation, which I have described and not run: take an adjudicated fixture set — planted defects, known answers — and run it twice. Once through the pipeline as built. Once with one stage handed the full context that seam currently keeps from it. Count confirmed true findings against the known answers, not verdict changes, because a verdict that moved isn’t a verdict that improved. If the wide version doesn’t win, the seam is earning its keep. If it does win, that relay was too narrow and I’ve been paying to compress something the model wanted whole.

I don’t know which way that goes for my pipeline. It is a cheap experiment and I haven’t run it, which is the most honest thing in this post.

The point

Stage count is a diagram. The payload is the design. Every seam is a decision about what the next stage is not allowed to know — and if you didn’t make that decision on purpose, the default made it for you.

So the question I’d start with on any pipeline with a model in it isn’t how many stages it has. It’s what each stage refuses to forward — and whether a better model would want it back.

How sure I am: the bounded-relay pipeline is real and running, genericised — typed projections on every seam, no stage seeing the whole diff after the scout. Cost was the motive and the pipeline is cheaper than what it replaced, but I have no isolated figure for the narrowing alone — it shipped with other changes and I can’t apportion the saving. That it improved recall, determinism, or anything else about the outputs is a hypothesis I have not tested, and nothing here should be read as evidence for it. The bounded-relay condition, the “near-sufficient” target, the 18-experiment evidence base, and the capability inversion are Yu et al.'s, cited with their scope intact — including the half that cuts against me. The Anthropic deletion figure is theirs and is sourced in full in the linked post, cited here only for the shape of the move. What’s mine and unmeasured: reading near-sufficiency as per-seam rather than system-wide, the delete-what-isn’t-checked test, the relays-widen-by-default warning, the account of why over-forwarding costs, and the ablation I’ve specified but not run. And the expiry condition is an inference, not a measurement — Yu et al. measured the capability effect across their benchmarks and model scales, not on my pipeline. That mine specifically is worth its complexity only while the model is weak enough to need it is me applying their result to a system they never saw. It’s the reason I wrote down the ablation instead of asserting the conclusion.

Sources

Claude Shannon, A Mathematical Theory of Communication (1948) — the epigraph; a framing, not a load-bearing cite.
Yu, Zhou, Dong, Barath, Staunton, Yoon, Qian, Caverlee & Ji, When Do Multi-Agent Systems Help? An Information Bottleneck Perspective (2026), arXiv:2607.16133 — a multi-agent system’s local contexts are linked only by “bounded relay messages”; with unlimited bandwidth it can forward everything and reproduce a single-agent system, so the advantage must come from limited relays. Gains land when relays are “near-sufficient,” are strongest for weaker models, and shrink or invert when relays drop information, particularly for stronger models; an effective parameter β tracks the shift with capability. 18 controlled experiments, five benchmarks, three model scales. Their result and their scope; the per-seam reading and the design test built on it are mine.
Thariq Shihipar, The New Rules of Context Engineering for Claude 5 Generation Models (claude.com blog, 24 Jul 2026) — the capability-conditional deletion route, cited here for its shape; the figure, the scope, and the eval caveat are handled in A Big Rule Set Is an Attention Problem.

Related: Keep the Factory, Ship the Result (which stages hold a model — this post is what crosses between them); Your Skills Are One-Offs (the same typed artifact, argued for portability rather than bandwidth); A Big Rule Set Is an Attention Problem (the same capability-conditional deletion, aimed at a rule instead of a seam).

Nobody Broke It

2026-08-07T00:00:00Z

“That which is everybody’s business is nobody’s business.” — Izaak Walton, The Compleat Angler (1653)

I wrote a lint rule that reads the clock.

It looks at a deprecation deadline declared in a schema, calls new Date(), compares, and if the deadline has passed it fails. Eighteen rules in that plugin; that’s one of them. It’s published as a library and other repos consume it.

I also gave it a gracePeriodDays option. You set a number, the rule adds that many days to the deadline, and the failure goes away for a while.

So I shipped the problem and the workaround in the same commit. Three months on, both are still there. I checked today.

The tell is in the error message, which I also wrote:

Deprecation deadline "" has passed.
Remove the deprecated type or update the deadline.

Two fixes offered. One is do the work. The other is move the deadline. I wrote a rule whose own error text tells you how to switch it off, and I did that because some part of me already knew what was going to happen when the date arrived and didn’t want to look at it.

It took three months to see why. That rule can be perfectly correct — deterministic, no opinion, same input same verdict, every run — and still have no business failing anyone’s build. Because when it fires, there is nothing to point at. The code didn’t move. The schema didn’t move. Nobody did anything. A date arrived.

BLUF

A check that imposes a consequence owes two answers, not one: why is this failing now, and whose obligation is this? Most checks never have to think about the second, because the diff answers both at once — the change explains the timing and the author is right there.
A clock-triggered rule answers the first and not the second, and that’s the failure. The deadline explains the timing perfectly well. What it can’t do is point at anyone, so the consequence lands on whoever happened to be standing there when the check ran. They can usually clear it, and that’s the trap — being able to pay a bill is not the same as owing it.
So supply the recipient separately. For new debt, a recorded prior-accepted state — a baseline that ratchets — narrows the trigger back to things a change actually caused. For the debt already in the baseline, a named owner and a scheduled path. Those are two different jobs and a baseline alone only does the first.
Deterministic is necessary and not sufficient. I’ve argued before that only mechanised checks get to block, because a deterministic verdict is checkable and a model’s is a probable reading. I still think that. It turns out to be half the test. My deadline rule passes it comfortably and should still not be allowed to block, and finding that out is what this post is.
The practical move: for anything a clock or a policy change can trigger, don’t reach for a diff. Fail only on the delta against a record of what you already accepted — and put a name and a date against the record itself, or you’ve bought a quiet build and lost the policy.

The version where the diff works

The other thing I’ve built is a multi-pass code reviewer. Detectors notice things on a change; a deep pass adjudicates the ones worth adjudicating.

It has the boundary. A detector hit only becomes a review lead if the hit’s evidence line is in the diff’s changed lines — literally, the line has to be in change.changedLines[path]. A hit on a pre-existing comment or an untouched branch gets dropped and never reaches the deep pass. That’s deliberate. Flagging code the author didn’t write in this change is how a review tool teaches people to stop reading it.

Then I went looking for evidence that the deep pass was working, and I forgot my own boundary.

To measure it I needed changes that would produce a lot of adjudications, so I wrote a corpus miner to find dense candidates. The miner counted detector hits anywhere in the head file. It found what looked like a rich seam. I ran the reviews.

Zero structured verdicts. Not few — zero. Everything the miner had counted was pre-existing, and the product did exactly what I built it to do: dropped all of it as unattributable to the change.

I fixed the miner to count only hits whose evidence line was in the changed lines, re-mined, and one single-file commit then produced four leads and four structured verdicts.

The number that matters there isn’t four. It’s that the boundary was already correct in the product and I dropped it anyway the moment I built a second tool that looked at the same code from a different angle. One incident is not a rate, and I’m not going to tell you this happens to everyone. It tells me the default — look at the whole file, count everything you find — is always sitting right there, always looks like more coverage, and beat me once on code where I had already written the correct version myself.

One honest limit: those leads aren’t a gate. They’re review judgment, nothing about them blocks a merge, and that repo’s CI validates rules consistency and nothing else — a working document in it still treats which checks are even gateable as unsettled, with a calibration pass agreeing with itself on nineteen of forty conditions. So this is the cheap version of the consequence: someone has to answer the lead. The expensive version is the plugin, which reports an error, and what an error does depends on a consuming repo’s CI I haven’t surveyed.

The version where there is no diff

Take the deadline rule and try to give it the same treatment. Compare against the diff, fail only on what the change introduced. It doesn’t work, and it’s worth walking through why, because every candidate fix fails in a slightly different and instructive way.

Compare against the diff. ESLint’s unit of work is a file. A rule receives one file’s source and reports against nodes in it. Nothing in the API carries a diff, so there’s nothing to compare to. You can bolt on a wrapper that filters reports by changed line — people do — and for the deadline rule it produces the wrong answer: the violating line is a deadline field somebody wrote months ago and nobody has touched since. Filter to changed lines and the rule silently never fires again.

Use the commit date. For any fresh branch that’s approximately now, so the rule fires on everyone.

Use git blame. This one is the most tempting and the most unfair. Blame resolves cleanly — it names the author who added the deprecated usage. That author did nothing wrong. When they wrote it the deadline was six months out and the usage was entirely legal. Blame will hand you a name with total confidence and the name is innocent.

Use the deadline itself. The deadline is the trigger, and it corresponds to no commit at all. There is no point in the repository’s history where someone did the thing that made this fail.

That’s the finding. A time-triggered rule has no attributable change by construction. Not “hard to attribute,” not “attributable to someone else” — the failing event is not a change. Each of those four ways of asking who did this takes a diff or a commit as its input. Here there is no such input, so each one still returns an answer, and each answer names somebody who did nothing.

gracePeriodDays is what that looks like from the inside. Not a grace period — a postponement knob, on a rule that was going to start reporting an error nobody had agreed to owe the moment a date I picked myself went past, wherever it happened to be running.

Deterministic isn’t the whole test

I’ve drawn one line here before: what gets to block. Only the mechanised checks, because a deterministic rule’s pass or fail isn’t a matter of opinion and a red build on it is honest, whereas a model’s judgment is a probable reading and making it a required green trains the next human to defer to it.

The deadline rule sits on the safe side of that line and always has. It’s mechanised. It’s deterministic in the sense that matters — same schema, same clock, same verdict, no model anywhere near it. Its output is a checkable fact.

And a red build on it is not honest, because the thing it asserts — someone here has an obligation they haven’t met — isn’t true of the person it stops. It’s true of the organisation, in the aggregate, on a Tuesday.

So the test I had was necessary and not sufficient. “Is this verdict checkable” is a question about the check. The second question is about the consequence: who receives it, and is the obligation theirs?

Not can they clear it. They usually can — that’s the whole trouble. The developer my rule stopped could go and delete the deprecated usage, and if they wanted their build green badly enough they would. Capability isn’t the scarce thing; it’s what makes the arrangement work as a lever rather than merely a nuisance. The build holds their change hostage until they do a piece of work that was never theirs, and they pay because paying is cheaper than arguing. What they don’t have is standing — no say in whether the deprecation was worth it, no authority to decide it can wait, nothing to appeal to. Handing a bill to someone who can pay it is not the same as handing it to someone who owes it.

That’s also not the same as asking why it fired now, and I want to be careful here because I spent a while thinking it was. My deadline rule explains its own timing perfectly well — the date passed, that’s the whole story, no mystery. Accounting for the timing doesn’t tell you whose bill it is. For nearly every check we write the two questions collapse into one, because a diff answers both at once: the change explains the timing, and the person who made it both can fix it and owes the fix. You only notice how much work the diff was doing when the two come apart.

Two axes, then, and a blocking check needs both:

Is the verdict checkable, or is it a reading? Readings don’t block. That’s the earlier line, and it stands.
Is the obligation the recipient’s own? If the only person it reaches is whoever happened to run the build, it doesn’t get to block — however checkable it is, and however easily they could make it go away.

The second axis narrows the first rather than replacing it. Mechanised is still the entry requirement for blocking. It just isn’t the whole of it.

A checkable verdict is only the first proof. It may block only when the person it stops owns the debt; otherwise, route accepted debt to a baseline and a named owner.

What the boundary is instead

If a diff can’t scope a time-triggered rule, what can?

State, not dates. Record the set of violations you have accepted, as a file in the repo. Fail on anything not in that record. Shrink the record over time and never let it grow — a ratchet. The snapshot isn’t a date at all, which is the point; it means as of when we last accepted this, and the clock can’t move it.

That dissolves the attribution problem rather than solving it. There’s no longer a question of who caused the failure, because the check no longer fires on the passage of time. It fires when a change makes things worse than the last state you signed for, and that’s a change, and it’s yours. Trigger narrowed, recipient found.

It also isn’t exotic. ESLint ships this natively — --suppress-all writes the baseline, the suppressions file is checked in, --prune-suppressions shrinks it. It landed in 9.24, and my plugin develops against 9.36, so it was sitting there the whole time. I built four hand-rolled per-rule escape hatches, permanently, for a problem the platform solves generically one layer up.

For the new violations, and only those. A baseline is a no-new-debt device and nothing more. It stops the pile growing. It does nothing at all about the usages already inside the record — which are exactly the ones the deadline was written for. Ratchet and stop there and you’ve bought a quiet build and a dead policy, and the suppressions file becomes the place violations go to live.

So the other half isn’t a trigger at all, it’s a recipient. The accepted set needs a name against it and a date in somebody’s real queue: an owner, a scheduled review, an escalation that lands on a person who can either spend the time or decide the deprecation wasn’t worth it. That’s a different kind of consequence from a red build, and it should be. A red build is a consequence you inflict on whoever arrives next. A ticket with an owner on it is a consequence you hand to the person whose call it is.

Two halves, two jobs. Narrow the trigger so the build only stops changes that caused something. Name the recipient so the debt you’ve agreed to carry still has someone carrying it.

Concretely, the end state I want: a change that adds a use of a deprecated type after the deadline goes red, and it should — that one is yours. A use that was already in the record the day you cloned the repo does not go red for you; it sits in a file that is only ever allowed to shrink, and someone whose name is on it gets asked, on a schedule, when it’s coming out. Enforced that way, the deadline lands on the people who can make the decision. Enforced the way I built it, it lands on whoever pushed next.

I haven’t shipped either half. Three months of not shipping it is part of the receipt.

The strongest objection

A deadline announced in advance is a perfectly honest anchor. It was written into the schema, everyone could see it, nobody is surprised. Announcing a date and then enforcing it is what a deadline is. Give it an owner and a scheduled escalation and you have a fair consequence without any of this.

This is right, and it isn’t really a counter-argument — it’s the second half of the fix arriving from the other direction. A predeclared deadline with a named owner and a real escalation path is an honest consequence. What it isn’t is a red build on the pull request of someone who has never touched the deprecated type. Precommitment establishes that the obligation exists; it says nothing about who should be made to pay for it. I was treating those as one fact for three months. An owner supplies the missing recipient, a baseline narrows the trigger, and either one gets you somewhere honest. Neither licenses charging the bystander.

The tell is that my rule had nowhere to put a name. A deadline field, a call to new Date(), and a knob to move the date. There was no owner in the design, and no place one could go. That’s not a gap in the implementation. That’s the thing the design was missing.

Second objection: then don’t write rules that read the clock. But the policy is real. Deprecation deadlines exist because otherwise removal never happens, and deleting the rule deletes the policy along with it. The question was never whether the date matters. It’s what the rule is allowed to do when the date arrives.

Third: this is just the pre-existing-failure problem. Unrelated red CI shows up on your PR, you didn’t cause it, everyone hates it, nothing new here.

They do look alike from outside — both are a person blocked by a failure they didn’t cause. One operational difference separates them: whether a culprit exists. In the familiar case there is an attributable change, it just isn’t yours. Someone broke main, you’re standing where it landed, and the standard remedies all assume that — find the commit, revert it, quarantine the test, page the author. They work because there’s someone to find.

For a clock-triggered rule there’s no such commit anywhere in the history. The usage was legal when it landed and legal every day after, until a date. No revert fixes it, no author erred, and the culprit-shaped remedies have nothing to bite on — which is why the fix has to come from somewhere else, either a record of what you accepted or a name against what you haven’t. This is not someone else broke it. Nobody broke it.

What this isn’t

Not an argument against blocking. If your change introduces a violating line — including newly adding prohibited usage after a deadline has passed — that’s fully attributable and it should absolutely stop you. The easy case stays easy. This is about the case where an unchanged existing usage starts failing and nothing else moved.

Not the coverage question. I’ve written about the other half of this — every method that narrows your view excludes something, and you owe the excluded region a compensating check. That post drew its own boundary and reserved this one explicitly: “Not the question of what gets to say no … Adjacent, separate.” Correct then, correct now. Aiming an instrument at what you excluded and deciding what an instrument may punish you for are different problems.

Not a claim about reaching outside the file. I spent a while convinced that rules reading more than one file were inherently less attributable, and it isn’t true: a cross-file cycle your PR creates is fully attributable, and if lint gates your merges the repo stays clean inductively — the change that first forms the cycle never lands. Input scope isn’t the variable. Whose obligation the consequence lands on is.

How sure I am: The zero-to-four is measured and it’s mine, but it’s small — one commit, one detector, and it’s a corpus-mining result, not a production incident. Read it as a demonstration that the boundary is easy to drop, which is what I’m using it for, and not as a rate. The plugin facts are from experience plus a source check against the current branch today: the wall-clock call, the grace-period option, the error message offering to move the deadline, and three months of the fix not shipping are all verifiable, and all mine. I have not surveyed anyone consuming that library, so I’m making no claim that this has hurt anybody downstream — the harm I can testify to is that I designed an escape hatch instead of a boundary and it took me until now to notice. Both halves of the prescription are prescriptions. The baseline mechanism exists and ships in the platform, but I haven’t migrated to it, so I can’t tell you what it costs on a real repo. The owner-and-schedule half I have neither built nor watched anyone else run, and it’s the half doing the actual enforcing, so treat it as the shape of an answer rather than a tested one. The two-axis test is my inference, not a measurement. If this is wrong somewhere, it’s most likely there: I’m asserting that a consequence with a named recipient gets acted on, and the honest position is that I’ve seen the failure mode and not yet the fix.

Sources

The targeting result — my own instrumented run on a multi-pass reviewer I built. The product promotes a detector hit to a review lead only when the evidence line is in the change’s changed lines; a corpus miner I wrote counted hits anywhere in the head file, and the first “dense” reviews returned 0 structured verdicts because everything counted was pre-existing. Miner fixed to intersect with changed lines, re-mined, one single-file commit produced 4 leads and 4 structured verdicts. Author’s own, genericised. Related: Noticing Is Cheap, Judgment Is Scarce, which is the same pipeline reporting the inverse failure — a zero caused by a scan that didn’t run.
The lint plugin — mine, eighteen rules, published as a library. Verified against the current development branch on the day of writing: the deadline rule calls new Date() at lint time, takes a gracePeriodDays option defaulting to zero, and reports a message whose suggested remedies are “remove the deprecated type or update the deadline.” Three further rules carry a cross-file opt-out. From experience plus current-source verification; genericised.
ESLint bulk suppressions — --suppress-all, a checked-in suppressions file, and --prune-suppressions. Shipped in ESLint 9.24.0 (April 2025); my plugin’s dev dependency is 9.36, though its declared peer range is >=9.0.0, which admits versions that predate the feature. Prior art for the baseline ratchet, and the reason the hand-rolled per-rule version was avoidable. Published release notes plus a check of the plugin’s own manifest.
The blocking line this post extends — Teaching the Reviewer, which argues only mechanised checks get to block and model judgment stays advisory. This post accepts that line and narrows the conditions under which a deterministic rule may block; it does not revise the earlier distinction.
The lane this post doesn’t enter — Buy Back the Coverage, which reserved “the question of what gets to say no” as adjacent and separate.

Related: Teaching the Reviewer — what gets to block, and why judgment stays advisory; Buy Back the Coverage — the adjacent question this one leaves alone; Noticing Is Cheap, Judgment Is Scarce — the same pipeline, the opposite zero; Keep the Factory, Ship the Result — candidates, promotion, and who is allowed to do which.

Buy Back the Coverage

2026-08-06T00:00:00Z

“To the curious incident of the dog in the night-time.” “The dog did nothing in the night-time.” “That was the curious incident.” — Arthur Conan Doyle, “The Adventure of Silver Blaze” (1892)

I had already published the warning: if a model only judges what a scanner shortlisted, a site the scanner fails to flag is a false negative nobody ever sees. A wrong flag costs a cheap check and gets caught. A wrong drop never enters the funnel.

Then I mechanised the review pipeline I’d built, to cut the bill. And mechanising quietly demoted the rule scanner — the pass that reads every changed line against the whole rulebook — down to a cheap floor that never ran the scan at all.

I found out by accident. I ran the new pipeline against the old one on the same change, for an unrelated reason, and the new one returned nothing where the old one had flagged five rule findings. The warning sprung on the person who wrote the warning.

A zero can mean no findings were present, or that the thing able to find them never ran. The output alone cannot distinguish those cases.

The lesson is not “be more careful.” I had the general rule. What I did not have, anywhere on disk, was a line saying this build gave up the systematic whole-rulebook scan in exchange for cost. That specific exclusion was never written down as a decision, so nothing was watching it, and nothing could have complained. A skipped notice produces no output. The zero looked exactly like a clean run.

BLUF

Every method that narrows your view has a bill: the coverage it excluded. A scanner shortlist, a gate, a sample, a proxy metric, a filter, a comparison you chose not to run. The excluded part doesn’t stop mattering. It stops being visible.
So treat the exclusion as a decision, not a side effect. Do two things: aim a compensating check at what the method cut out, and record the gap beside the decision that created it — not in a notebook, and not in your head.
Aiming is the part you control. Whether the second check reaches the excluded region, or just shares the first one’s blind spot at twice the price, this post does not settle.
Write the gap down. An unnamed gap produces no signal. A wrong output complains. A missing one doesn’t. In both episodes I have, whether the gap got found came down to whether some unrelated check happened to cross it.
The same move is in four of my published posts. I did it once. I prescribed it three times. I never named the rule. Naming it is what lets you carry the move to a surface you haven’t filtered yet. Recording the gap is what stops the version I shipped.
Limit: I can show that named gaps get placed in a durable record. I cannot show that anyone comes back for them.

The move I keep reaching for and never named

Going back through my own posts, the same shape is in four of them. Each time a method narrows the view, and each time the post reaches for the excluded coverage — without ever saying that’s what it’s doing. What it doesn’t do is finish, three times out of four, and that turns out to be the interesting part.

Where	What the method cuts you off from	What checks the excluded coverage	Did I actually do it?
Augmented Reading	Everything the machine pass said nothing about	Route the deep read there — and carry the limiter: “a flag marks a point, not a neighbourhood”	I did this. The post describes it as current practice: “it’s caught things I missed.”
Skills Are a Codebase	When the skill fires; gating the eval to grade output “says nothing about when the skill fires”	Shouldn’t-fire cases alongside should-fire — “only a shouldn’t-fire case can” tell you a skill is waking up where a sibling should own the task	I prescribed this. The post says what the harness needs; I have not built it.
Noticing Is Cheap	Sites the scanner never shortlisted, which the judge therefore never sees	“The cheap floor must discover, not just verify… because the scanner’s blind spots are where the misses hide”	I prescribed this after the failure. It is one of the three rules that “fall out”.
Build the Operator	Whether training bought anything over pasting the rule into the prompt	The comparative held-out-rule eval — “that delta is the whole claim”	I have not run it. The post ends: “it’s the eval I haven’t run.”

That last column is the one I didn’t expect when I went looking, and I’d been about to publish the table without it. Four surfaces, one move, done once. The other three are promises I wrote down and did not keep.

That sharpens it. If I had done this four times, “I never named the general rule” would be a curiosity. Having prescribed it three times and completed it once, the missing name looks like the reason: there was no rule to carry. Each post re-derived it, made a local promise, and left it there. Nothing accumulated. When the fifth surface arrived — the mechanisation — there was nothing to apply.

My bet: an unnamed practice doesn’t transfer. You rediscover it per surface, and the rediscovery is expensive enough that you’ll skip it somewhere. One episode is what I have — I failed to transfer a rule I knew. That’s a demonstration that it can happen, not evidence about how often.

The primary method sees one region. The compensating check must aim at the shadow it leaves, and the decision record keeps that target attached to the filter that created it.

Why the writing-down is the half that matters

I went looking for the receipt I expected: a gap left unnamed that surfaced later as a user complaint or an incident, and felt at the time like bad luck. I didn’t find one. What I found were two unnamed gaps that both cut the other way, and they turn out to make the better argument.

The first is the demotion above. The second is from the same pipeline: the reviewer had no check at all for behaviour a diff removes — a deleted call, a dropped guard, a catch that quietly stopped reporting. I found that one by inverting my planted-bug fixtures on a hunch. The model flagged the removal in about a quarter of runs, and the quarter clustered hard: three for three on a deleted await store.flush(), three for three on a removed error-swallow, and nothing whatsoever, in any run, on eight of the thirteen fixtures.

Neither of those arrived as bad luck. Both arrived as findings from deliberate work I happened to be doing for another reason. That kills the claim I started with. Leaves this:

Both gaps were silent, and each was found only because an unrelated check crossed it. The demotion: “because a skipped notice produces no output, nothing complained.” The removal class: “it doesn’t present as a gap. It presents as coverage.” That’s the same property twice. In both, discovery had nothing to do with how bad the gap was. It came down to whether something else I was doing happened to pass through it.

“You’ll eventually get unlucky” can’t be checked against anything. “Nothing will tell you” can: point at a gap and ask what output it would produce if it were costing you. If the answer is none, then nothing in the system is going to raise it, and you write it down beside the decision.

The demotion removes the excuse. I wasn’t ignorant of the failure mode; I had published it. Knowing the rule in general did nothing, because the decision that created the gap happened in a different place from the post that named the danger, and the two never met.

“Written down” has to mean somewhere in particular

Not a limitations section. Not a chat log. The test is: the artifact someone opens when they reopen the decision.

For a design choice, that’s the design doc. For a filter in code, it’s next to the filter. For an eval, it’s the eval’s own description of what it does and doesn’t measure. The gap has to live beside the decision, because the moment the gap matters is the moment somebody is changing the thing that created it.

I have one receipt for this working as intended, and it’s a positive control rather than a save. Auditing an instrument mid-run, I wrote three known gaps into the durable design record instead of keeping them in my head — the note I left myself was literally “known gaps stay in the RFC rather than in my head” — alongside a correction to a catalogue row that had been claiming evidence that didn’t exist. Those gaps are now available to whoever reopens that design.

Nobody has come back for them. They are available, not revisited. And I recorded the gaps; I did not record a check beside them.

The strongest objection

“How do you know the check you bought reaches any of the excluded coverage — rather than sharing the original method’s blind spot, and just costing you more?”

I don’t. This decides whether the title is a rule or just a phrase. A second reader who learned to read the same way misses the same things. A discovery pass built on the same model as the scanner inherits the model’s blind spots wholesale. Nothing here establishes that any of my checks reached new ground. I need a defect log to know whether two nominally-different checks miss different things. I am still building it.

The smaller claim survives: name the gap so a check has somewhere to point. Without the gap, you cannot ask whether the check reaches it — you have two checks and no account of what either is for.

The second objection: “this is ‘document your limitations,’ which every engineer already knows, and every LIMITATIONS section is a graveyard nobody reads.”

Half of that lands, and it’s the half I conceded above: I have no evidence anyone reads the record. If the record is never reopened, writing it buys nothing.

But it misses the trigger. The instruction isn’t note things you’re unsure about — that’s a judgement call, and it’s the call I didn’t make: the mechanisation registered as a cost win, not as a coverage decision, so there was no moment where I felt uncertain and declined to write it down. The instruction is mechanical: any time a method narrows the view, it has excluded something; name what. A shortlist excludes the un-shortlisted. A pass/fail on output excludes the trigger. A non-comparative eval excludes the comparison. It runs without a judgement call. You just have to notice you added a filter.

And the other half isn’t documentation at all. Aim the check first, record second. My four posts all reached for the check and none of them recorded the gap — and that combination is precisely how the demotion shipped. The reverse combination, recording without checking, is the graveyard the objection describes. A note with no compensating check is a confession. A check with no note is the scanner I lost: it was doing its job right up until a rebuild had no reason to keep it.

You can’t buy back all the coverage, that’s just “do more work.” No — the buy-back is scoped to what this particular filter excluded, which is a much smaller and much more specific set than “everything,” and it’s usually a different kind of check rather than more of the same. Shouldn’t-fire cases aren’t more should-fire cases. A discovery read isn’t a bigger scanner.

What this isn’t

Two things look like this. They are not the same.

Not the claim that a filter makes a defect class invisible. That was the strong version and I retired it. I went to Basili and Selby’s testing-strategy comparison (TR-1501, 1985) expecting support and it doesn’t give any: two of their six defect classes come out flat across techniques, and what they vary is technique, not the surface you’re looking at. The honest claim is about signal, not invisibility.

Not the question of what gets to say no. Naming the positive term an instrument must be pointed at, and naming the gap it therefore isn’t pointed at, look like one idea and aren’t. They come apart both ways: you can validate a target properly while never recording the remaining gap, and you can record a coverage gap without having any instrument capable of saying no. Adjacent, separate.

Whether the check is independent of the one it compensates for is different again. It belongs with the objection above, not here — this post’s imperative depends on the answer.

How sure I am: Two runs are real. The demotion returned zero findings where the old path returned five, root-caused to a scanner silently demoted by mechanisation. The removal probe produced ten alerts across thirty-nine runs on thirteen fixtures, but I have not judged which removals warranted one, so read it as a shape and not a rate. The four posts are a reading of my own corpus, verifiable line by line, not a measurement: one performed, three prescribed. One design record holds three named gaps, and that is the whole of the positive control — co-recording a check beside a gap is my prescription with nothing behind it. I cannot show that anyone reopens the record, that a second check sees what the first missed, or that naming the move makes it transfer; I have one failed transfer, which shows the failure is possible and nothing about its rate. I am also not claiming an unnamed gap will eventually cost you something — my own two receipts point the other way, and that is why the claim here is about signal rather than consequences. One thing is retired outright: that a filter makes a defect class invisible. If this post is wrong somewhere, it’s in the assumption that the record gets reopened.

Sources

The recall regression — my own instrumented run: the mechanised pipeline returned zero rule findings where the old path had flagged five on the same change, root-caused to a rule scanner demoted to a floor that never scanned. Cause fixed and the scanner confirmed firing; the strict head-to-head is not yet re-run. Genericised, author’s own. Written up at greater length in Noticing Is Cheap.
Inverted-fixture removal probe — thirteen planted-bug fixtures run with base and head swapped so each diff becomes a removal; the model flagged the removal in ten of thirty-nine runs, clustered on the dramatic deletions, absent in all runs on eight of the thirteen. Author’s own, genericised, deliberately reported as a shape rather than a rate — the corpus has not been adjudicated.
The positive control — my own mid-run instrument audit: three known gaps written into the durable design record rather than held in memory, alongside a correction to a catalogue row claiming evidence that didn’t exist. Working notes, author’s own. Note what it does and doesn’t cover: gaps recorded, checks not recorded alongside them.
The four-instance reading — the status column is quoted from the posts themselves, not remembered: augmented-reading.md:74 describes the routing as current practice; skills-are-a-codebase.md:105 states what the harness needs; noticing-is-cheap.md:76 introduces its rule with “three rules fall out, and I’m holding to them”; build-the-operator.md:37 ends “it’s the eval I haven’t run.”
Basili & Selby, “Comparing the Effectiveness of Software Testing Strategies” (University of Maryland TR-1501, May 1985 — the tech-report precursor to IEEE TSE 13(12), 1987; cite the report, which is the version read) — read primary, and cited here for what it doesn’t support. The two flat classes are data and cosmetic faults, at α>.05; the significant effects run the other way (code reading found more interface faults, functional testing more control faults, both α<.01). The comparison varies technique, not the surface under view. It is not evidence that a vantage makes a class invisible.
Conan Doyle, “The Adventure of Silver Blaze” (The Strand Magazine, December 1892; the story is also cited simply as “Silver Blaze”) — the epigraph, quoted with Holmes’s setup line intact, because without it “the curious incident” has no antecedent. I use it as the canonical example of an absent signal being the whole finding.

Related: Noticing Is Cheap, Judgment Is Scarce — the pipeline this post’s two receipts come from, and one of the four instances; Augmented Reading — the same move on a reading surface, and the one I actually performed; Your Skills Are a Codebase, Not a Junk Drawer — the same move on a trigger; Build the Operator, Keep the Operands as Data — the same move on an eval, prescribed and not run; You Can’t Grade Your Own Misses — why the coverage you excluded is hard to see from inside.

Intermittent Competence Looks Like Coverage

2026-08-06T00:00:00Z

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman

There are two ways an automated check can fail you. One is loud: it doesn’t run, you look at the output, and the zero is obvious. The other is a check that genuinely works, just not every time.

The second one is worse, and I’d rank it above every other failure I hit building an AI code reviewer. Not because it misses more. Because of how it looks.

BLUF

A check that fires a quarter of the time doesn’t look like a gap. It looks like a feature. You see it work once, conclude it’s covered, and stop asking.
I found mine by accident, while trying to establish whether I needed to build a check at all.
It clustered. Three for three on the dramatic cases, nothing at all on eight of thirteen. So the average hides the shape, and the shape is what matters.
The fix isn’t a better prompt. It’s taking noticing off the model. A deterministic check fires every time or never, and either way you find out which.
The reason to prefer the deterministic version isn’t cost. It’s that it doesn’t have moods.

How I found it

My reviewer reads a pull request and comments on what it finds. It had no check for behaviour a diff removes — a deleted call, a dropped guard, a catch block that quietly stopped reporting. Before writing that check, I wanted to know whether the model already caught removals on its own.

So I inverted my planted-bug fixtures. Swap base and head, and each diff becomes a removal of the behaviour the bug had added. Thirteen fixtures, three runs each.

The model flagged the removal in roughly a quarter of runs: ten across thirty-nine. And they clustered hard. Three for three on a deleted await store.flush(). Three for three on a removed error-swallow. Nothing, in any run, on eight of the thirteen.

The model notices when the deletion is dramatic.

Two honest limits on that. Not every removal deserves a flag, and I haven’t adjudicated which of the thirteen did — so treat that quarter of runs as a shape, not a rate. And thirteen fixtures at three runs is small. The clustering shows the misses aren’t random, and that’s the part that matters here.

Why catching a quarter of removals is worse than catching none

Zero would have been fine. I’d have seen the gap and built the check — that’s an afternoon’s work and I’d already decided to do it.

Catching a quarter of removals is worse than catching none because of what it does to your belief. You run the reviewer, watch it catch a dropped flush call, and conclude it handles removals. It demonstrably does handle removals; you just watched it. So you stop asking, and you never find out it’s silent on the eight cases that didn’t happen to be dramatic.

A silent miss is silent either way: a check that didn’t run, or a model that only sometimes looks. From the outside those are identical: no output. The difference is that the first one you’ll eventually notice, because zero findings looks wrong. Intermittent competence is a coin flip you’ve mistaken for a check.

And you can’t audit your way out of it by looking at what the reviewer found. Everything it found is evidence it works. The misses are the evidence, and they don’t show up anywhere.

The fix, and the reason for it

A deterministic base-to-head delta notices every removal, every run, and hands each one over as a candidate. The model goes back to the job it’s actually good at — deciding whether this removal matters.

I’d moved noticing off the model once before, to stop paying frontier rates for pattern-matching. That was a cost argument and it holds, but it turns out to be the lesser one. The scanner’s real advantage over the model isn’t that it’s free. It’s that it doesn’t have moods.

A parser that misses a case misses it every time, which means a fixture catches it once and forever. A model that misses a case three times in four will pass your fixture on the fourth run. A deterministic check doesn’t win here by being correct — a detector with a bug is wrong every time. It wins because consistency is auditable, and an intermittent check is not.

That generalises past code review. Any time you’re relying on a model to notice something instead of judging it, the question isn’t whether it can. It’s whether it does it reliably enough that your test of it means anything.

Which is why I’d take the parser even at the same price. The scanner’s real advantage over the model isn’t that it’s free. It’s that it doesn’t have moods.

How sure I am: The inverted-fixture run is mine and real — thirteen planted-bug fixtures, base and head swapped, three runs each, ten removal alerts across thirty-nine runs, clustered on the dramatic deletions and absent on eight of the thirteen. The corpus is unadjudicated: I haven’t ruled on which removals warranted a comment, so this establishes the clustering rather than a recall rate. The claim that a deterministic delta would catch all of them is reasoned from how a parser works, not measured — I haven’t built it and run the comparison. The generalisation in the last paragraph is argument, not evidence.

Related: Noticing Is Cheap, Judgment Is Scarce — where this came from, and what it cost to move noticing off the model. A Big Rule Set Is an Attention Problem — the same move, argued from attention, not reliability.

A Constraint You Can Quietly Lift Isn't One

2026-08-06T00:00:00Z

“Nullius in verba” (take nobody’s word for it) — motto of the Royal Society, adopted in its First Charter of 1662, after Horace’s Epistles

I had a harness measuring whether a set of skills helped a cheap model. It came back negative: no lift worth the tokens. I believed it.

It was measuring a configuration that doesn’t exist. The harness injected every skill into every case, including cases the skill should never touch, so it graded them on noise. Gated to fire a skill only where the skill belongs, the verdict reversed and the skills clearly helped. I’ve written that up before, and the lesson I drew was the obvious one: fix the eval before you tune the skill. The skill wasn’t broken, my measurement was.

Here’s the half I left out. The fix was one look at the injection logic. Cheap, obvious, available the whole time. So the interesting question isn’t why the harness was wrong, it’s why I never looked — and the answer is that it had told me no. A tool that only ever agrees with you is obviously suspect; everyone knows to discount that. A tool that contradicts you feels like it has skin in the game, so the disagreement is the thing that stops you auditing it. I wrote the harness, I wanted a verdict, and the verdict it handed me was the kind that feels earned.

So the thing that failed wasn’t care. The check cost a minute and I skipped it, and I skipped it because at the moment it mattered, I was the one deciding whether to run it and I already had an answer I believed. Nothing made that check owed. Had I written down before the run what the harness would have to be pointed at for a negative verdict to mean anything, the audit wouldn’t have been a judgement call made by a person holding a result. It would have been a thing I’d already agreed to while I still had no idea what the answer would be.

That gap is the subject. The difference between a constraint that works and a constraint that doesn’t isn’t how well the constraint is designed. It’s how much of your authority to lift it is gone by the moment it binds — and whether the part you kept back can be used without anyone, including you, having to notice.

BLUF

A constraint works to the extent that, when it binds, you can’t lift it quietly. And whatever authority you mean to give up has to be gone before the data arrives — because once the data arrives you will have a reason to lift it, and the reason will be a good one. That’s what having a reason means.
So the mechanism is the timing, and the enforcer sets how much of it survives. Putting the constraint in an institution is the most reliable way to make the surrender stick, because then the authority is gone as a matter of fact rather than of intention. Other arrangements can work. They are weaker. Say how much weaker. This post is about the binding half only. An outside reviewer also brings a second mind, which is a different good and not the one being measured here.
This is not the claim that you can’t bind yourself. Below is a dated episode where I did: a reject condition written to disk before the run, which fired an hour later against what I wanted, and I applied it as written rather than renegotiating it. It cost a cost saving I’d already shipped. What a written, dated rule takes away isn’t your ability to change your mind — it’s your ability to change it quietly.
Two things about an eval make this matter more than it does in physics. The specimen reads the protocol: a model can be trained on your benchmark, infer your criteria from the shape of the task, and optimise the proxy you exposed. And the experimenter is paid by the result: you built the thing, you built the eval, and green is what you wanted. Neither is uncertainty, and neither is answered by measuring better — and they are less separate than they look, because the specimen also reads the experimenter.
A better instrument and a surrendered authority are not competing categories. A hidden validation set is a real measurement improvement and it works only while you can’t see it. Both at once. Improve the instrument all you like — just know which parts of it stop working the moment you’re allowed to look, because those parts have a deadline and the rest don’t.
So before you run the thing, write down what would make you stop. The number, the direction, dated, in a file rather than in your head. Then when it fires, apply it as written. The test of whether you have a criterion at all: could you change it right now without anyone, including you, having to notice?

Most of the experimental disciplines transfer intact

You can’t read the mechanism. Interpretability is real and getting better, but nothing available to me today lets me look inside a model and predict what it will do on my task. There are weights and there is behaviour, and the map between them is not something I can consult. So you form a hypothesis about the behaviour, build an instrument, measure, and keep what fits.

That is not special to this technology. Most empirical science works from the same position. Psychology has never opened a mind. Clinical medicine licensed drugs that plainly worked for decades before anyone could say why. Neither field responded by giving up on measurement or by pretending the mechanism was visible; both built disciplines for measuring outputs when the inside is closed. Treating the uncertainty as a defect specific to evals is a category error, and it’s why people either over-trust a benchmark or dismiss the whole enterprise.

The disciplines are sitting there. Take them. Name the construct before you measure it. Replicate to size how much your verdict wobbles, and don’t read a union of runs as coverage. All of that is instrument work: you get better at it the way you get better at anything, and none of it stops working because you know you’re doing it.

One of the imported disciplines is not like that, and it’s the one this post is about.

The two things that aren’t true of a physics experiment

An electron doesn’t read your rubric. Nobody gets promoted for the electron behaving.

A model can be trained on your benchmark. It can infer your criteria from the shape of the task you handed it. It can optimise the proxy you exposed without ever touching the thing the proxy stood for. That isn’t sabotage; it’s what optimisation does when you leave a gradient lying around. The honest analogues aren’t in physics — they’re clinical trials and social psychology, where the thing being studied responds to being studied. I’m drawing that parallel from the shape of the problem, not a result from that literature.

And you’re not a neutral party. You built the thing, you built the eval, green is what you wanted, and you’re the one who decides when to stop looking. Science has never answered that with personal virtue. Engineering often does: be more honest with yourself. Preregistration, blinding, independent replication, adversarial review: none of them works by asking you to be honest. Each tries to move a decision out of your hands at a moment when you don’t yet know which way you’d want it to go, and that timing is the trick — it’s the thing a request for more self-honesty doesn’t have. But the move only counts to the degree it holds. A preregistration you can amend without leaving a mark, a replication you can decline to read, an adversary with no power to overrule you: each is the shape of the thing without the substance. Adversarial review does two jobs. Only one needs power. An adversary who can’t overrule you can still hand you a reason you didn’t have. They can’t bind you. That takes a call that sticks over your objection.

Those two read as separate — one about the specimen, one about you. A result published the same day as this post says they’re joined. Transluce varied the identity a model could infer from its context (an account email, a folder name, a memory file) and measured what moved. Identities the model recognises as safety researchers pull its stated confidence down, make its grading harsher, and make it markedly less suspicious of borderline requests; the strongest single identity sits over four standard deviations out from the general population. Twenty-two of twenty-four models across six families move the same direction, and not one moves significantly the other way. The harness they mostly ran it in was Claude Code, which puts your account email in the context window.

So the inlet I named is too narrow. The specimen doesn’t only read the protocol. It reads the experimenter — and paid by the result stops being purely a fact about your judgement, because you are legibly in the room and the thing you’re measuring can tell who you are. Worse, for anyone using a model as a judge: the tell is going away. Older models said it out loud; current ones almost never do. Across 14,066 traces the model verbalised noticing who it was talking to 0.84% of the time. The behaviour stayed. The disclosure didn’t.

A model can infer both the target you expose and who is asking. Either can change the behaviour the harness records.

What the timing does

Blinding is the standard answer in the fields that hit this hardest. I had it wrong. It is a measurement design, and a good one — a blinded trial’s result is more valid than an unblinded one’s. That’s instrument work. It is also a rule about who may know what, imposed before anyone knows what the answer will be. Both descriptions are true, and they were never competing. Unblind a trial after the fact and every instrument in it still functions perfectly; the trial doesn’t. What broke wasn’t the measurement. It was that somebody got the authority back.

Same with a held-out set. A randomly sampled, outcome-based holdout is a genuine operationalisation improvement — it tracks the thing you actually ship better than whatever proxy you had. It works only while you don’t have it. Publish it and it remains a fine piece of instrument design and stops being a measurement, because the loop now optimises toward it. So better instrument and restricted authority are not two bins to sort your fixes into. They’re two questions to ask about the same fix, and only the second one has a deadline.

A preregistered threshold is the purest case, because it has no instrument content at all. No fixture may regress from a full catch to a miss, or revert is not a better measurement of anything. Written down after the numbers arrive, it’s a description of your data. Written down before, it’s a criterion. Identical sentence. The only difference is when you lost the ability to choose it.

If I wrote the rule, I can rewrite it. True — and that’s the argument for external enforcement, not an argument that self-binding is nothing. A written, dated rule doesn’t remove your ability to change your mind. It removes your ability to change it quietly. Renegotiating a rule that exists in a file, with a timestamp, next to the reason you wrote it, is an act you have to perform and watch yourself perform. That’s weaker than a rule somebody else holds. It is not the same as no rule.

Before the result, the rule can constrain the eventual choice. After the result, the same sentence only records a choice already available.

The afternoon a rule I wrote took something from me

Here’s the receipt, and the timestamps are better than a summary. Two hours in a review harness, 13:34 to 15:32, transcribed from the session record.

13:34 — the requirement, before any instrument exists. The task opens by specifying its own constraint: “Decision rule must be pre-registered before running.”

13:39 — the rule goes to disk. Not a note in conversation; a block appended to a durable A/B record under the header PRE-REGISTERED (results pending), carrying its own justification: “Registering the instrument and the decision rule before the run … a threshold picked after seeing the numbers is not a test.” The same block refuses a Goodhart trap in advance, deciding what the metric may not punish before the metric exists — high finding volume, nits included, is “the product, not a defect,” so the cheap arm doesn’t get penalised for reporting more.

13:52 — a pre-registered confound fires on the first smoke run. One arm inverts each fixture’s diff, so a bug-introducing change becomes a bug-removing one and any finding at the planted site scores as a hard false positive. On the flush fixture the inverted diff deletes the flush call, and the cheap model returns exactly one finding: the missing flush. The instrument scores it as a hard FP. Except: “‘you deleted the flush, did you mean to?’ is a legitimate review comment about removed behaviour. The code under review is correct, yet the finding isn’t wrong.”

13:55 — eight words that routed the next two hours. “That’s a validity problem, not a code problem.” So it went out as an instrument audit rather than a code review.

14:16 — the outside ruling, and it cost more than I’d offered. I had already conceded the arm couldn’t support an absolute false-positive rate, and was still holding the comparative claim between two models on byte-identical inputs. The adjudicator killed both — because two models react differently to removal-shaped diffs, so the bias doesn’t cancel. That reason hadn’t occurred to me. I killed the arm mid-flight, already into the expensive half of the run.

14:49 — the other pre-registered rule fires. The reject condition written at 13:39 was: no fixture regresses from a full catch to a miss. The stronger model then caught a fixture the cheap model missed — twice, on independent samples. That trips the reject on a cost saving I’d already shipped and wanted to keep. The record shows me arguing my own side down first, correcting something I’d said earlier and listing three caveats against my new conclusion, and then: “the pre-registered rule says revert the enable — recall over cost, no trade. That’s the whole point of writing the threshold down before looking. I’ll apply the rule as written rather than renegotiating it.”

I could have renegotiated the rule at 14:49. Nothing stopped me. What I couldn’t do was renegotiate it and not know that I had: the threshold was sitting in a file with a timestamp and its own justification next to it, so amending it was an act I’d have had to perform and then read back. A threshold I could have set at 14:49 would have been set at 14:49. This one I could only follow or visibly overturn.

That is a smaller thing than having no choice, and it is not nothing. It is also the ceiling on what you can do to yourself.

That same file defeats the comforting story that my failures are the fixable kind: an hour after the threshold, I defined a stability metric as agreement across runs — which is maximised by silence, because two runs that report nothing agree perfectly. Careful operationalisation and careless operationalisation, same instrument, same morning, both mine. The hazard got caught before it produced a conclusion, and only because someone else was reading.

Neither outside-review receipt shows that the timing did the work. At 14:16 the adjudicator killed the comparative claim for a reason I hadn’t had. The stability hazard was caught because someone else was reading. In both, the outside party supplied information I didn’t have — not just a verdict I couldn’t overturn. Neither holds the information constant and varies only who gets to decide. A second mind explains both afternoons just as well.

Precommitment governs discretionary reversal: whether you can quietly walk back a call you already made. Outside review does that and brings a second mind, and those are separable goods that happen to arrive in the same package. This post is about the first one. It is not a claim that binding is all an outside reviewer is for — on that afternoon, most of what I got was the second thing.

Handing it to something that isn’t you

Not a person, necessarily — a person is the weak version. Keep the Factory, Ship the Result makes the case that a human gate is not an air gap: influence goes round the loop even where dataflow doesn’t. Parasuraman and Riley named the human end of it long before agents, cataloguing over-reliance on an automated aid as misuse. My harness is that failure mode with my name on it — I read the machine’s verdict before forming a view, then formed the machine’s view.

What I didn’t see at the time is that the same post already contains the fix. Describing what a gate buys, I wrote that “the machinery can create candidates but cannot promote them, and every candidate arrives carrying its provenance, to be judged against fixtures it didn’t choose.”

Fixtures it didn’t choose. Past tense: they were fixed before the candidate existed, which is the only reason the candidate can’t be built to pass them. I filed that as a design invariant for a memory loop. It’s a precommitment, and it holds for the same reason the 13:39 threshold did.

What this changes on Monday

The reflex when an eval misleads you is to improve the eval: add a rubric dimension, tighten the judge prompt, sample more, buy a better model to grade with. Sometimes that’s right. It’s never sufficient on its own: you can sharpen an instrument all week without moving what decides when you stop looking.

Before the run:

Write down what would make you stop. The number, the direction, the date, in a durable file rather than in your head. Then when it fires, apply it as written and log that you did. The test of whether it’s a criterion: could you change it right now without anyone, including you, having to notice?
Write down what the instrument would have to be pointed at for its verdict to mean what you want it to mean — and write it before the verdict. For my harness that sentence was “the skill fires the way it fires in a real run.” One look at the injection logic would have killed the result in a minute. Afterwards, I was a person holding a verdict deciding whether to check it, and that person doesn’t check.
Send the judgement you can’t make yourself to someone who can overrule you, and send it before you’ve settled what you want it to say. The ones worth sending aren’t the ones where you’re stuck. They’re the ones where you already have an answer and the answer is yours.

There’s a fourth I’d bet on and can’t back, so it isn’t on the list: treat any target you publish as compromised from the moment you publish it, and keep something you never publish and never tune against. It follows from everything above — which is exactly the kind of reason that turns out to be wrong. I went looking for an episode where it paid and didn’t find one. Treat it as a hypothesis to test, not an instruction, and don’t take it on the strength of the three it’s sitting next to.

Every one of those makes you smaller on purpose, at a moment when being smaller costs you nothing you can yet see. That’s what nobody wants, and it’s why this advice never accumulates.

The point

Flawed but the best we have is doing two jobs and only earns one of them. Uncertainty is ordinary, and a better instrument really does answer it. Adaptation and self-interest are not uncertainty, and the fixes that touch them all have one shape: a set you don’t look at, a threshold you can’t move quietly once the numbers arrive, a verdict that isn’t yours to cast.

Read nullius in verba that way and it stops being advice about evidence and becomes advice about what you submit to, and when. Take nobody’s word for it sounds like a rule for the reader — go check for yourself. But the answer to whose word do we take was a society that published, replicated, and argued in public: machinery that would rather catch you than agree with you. Going and checking for yourself is the easy half. The hard half is handing something over while you still don’t know what you’d have used it for.

The instrument you can always improve. The authority you can hand over at any time — but once the data is in, you choose which piece goes. That’s the problem. In advance, you don’t get to pick. And whatever you keep back, you keep for the moment it binds.

How sure I am:

From experience, and it’s one episode. The two-hour audit above — timestamps transcribed from the session record rather than remembered, and genericized. It shows a written rule applied against a stated preference, and an outside adjudicator removing a claim I was still holding. One instance.
The gap in that receipt. I wanted recall as well as the cost saving, so the 14:49 episode can’t separate the rule bound me from the rule agreed with me. It shows a written rule surviving a divided preference; it doesn’t show one surviving a strong contrary preference, and the strong case is the one the claim needs. The 14:16 ruling isolates the preference better: I was still holding the comparative claim, and an outside call took it anyway. It doesn’t isolate the mechanism. The call arrived with a reason I hadn’t had, so it can’t separate the adjudicator could overrule me from the adjudicator knew something I didn’t. I have no receipt that holds the information constant and varies only who gets to decide, which is why the post claims the binding half rather than the whole value of an outside reader. Neither half shows that the writing-down is what did the work. The harness is the negative control on the same point — nothing written down, and the check never happened — but a negative and a positive months apart isn’t a comparison.
Mine. That the mechanism is the timing of authority rather than the identity of the enforcer, and that self-binding therefore differs from external enforcement in degree — how thoroughly the authority is gone — rather than in kind. I have not tested whether people who preregister criteria end up with better evals.
Prior art. Goodhart via Strathern. Parasuraman & Riley for misuse, scoped to the documented failure mode with no claim about base rates. Neither finding carries this argument. The receipt stands without them. The parallel to reactive measurement in the social sciences is mine — I know the tradition exists and I haven’t read a result from it that bears on this, so I’m not citing one.
Added after publication: the specimen reads the experimenter. Transluce’s user-awareness result landed the same day this went up, and I folded it in rather than leaving the adaptation half where it was. I’m carrying their numbers, not reproducing them, and their stated limits travel with them: it measures fixed-prompt propensities rather than real-task performance, and the identity roster is hand-curated in a way the authors say leans toward their own hypothesis. The reading — that this joins the two asymmetries rather than adding a third — is mine. It also puts a live question over my own numbers elsewhere in this corpus, since my grading passes run in the harness they tested. So I ran the ablation on myself. Same contested draft, same seven bars, 71 grading runs interleaved in pairs; the only thing that varied was whether the working directory carried my name and email or nothing at all. A manipulation check confirmed the signal landed: asked directly who maintained the directory, the stripped arm said UNKNOWN and the other gave my name. No detectable effect on the scores — +0.22 points on a 0–100 scale, 95% CI [−0.81, +1.25]. The ship/hold rate moved in their direction, harsher with the identity present, 0.61 to 0.51, and is nowhere near significant. That rules out a large contamination and clears nothing else. The bound on the ship decision — the DV that actually decided what got published — is only about ±33 points, and their own group-level effect is roughly −1.1 on this scale against my lower bound of −0.81, so I sit right at the edge of being able to resolve an effect of exactly their size. The likeliest reason I found nothing is that theirs runs on recognition: famous names shift 3–4σ, their unknown affiliation-twins barely move, and my name isn’t a stimulus in anyone’s training data. So this null is what their model predicts, not a challenge to it — and the arm it doesn’t test is a well-known author grading their own work. Method and data: _raw/judge-identity-ablation-2026-08-07.md.
The concession. Neither asymmetry is new, and preregistration and blind adjudication are standard practice in older fields — two of them were already sitting in my own corpus, adopted and never argued for. What I’m claiming is why they work, which is what tells you whether a new constraint you’re inventing will work. The evidence is that I had the practices written down for months without it, which is exactly the state in which you drop them the first week they cost you something.
The joint to attack. The strongest objection is causal, not definitional. Writing the threshold down made revising it visible; it never made revising it impossible. So the live question is whether the visibility did any work, and one episode can’t answer it — at 14:49 I followed the rule, and I also wanted recall, and those two explanations predict the same afternoon. What would settle it: a case where the written rule and the author’s preference point in opposite directions with the rule still winning, or a run of preregistered thresholds where the amendment rate is lower than the same people’s rate of quietly reversing unwritten intentions. I have neither. Until someone does, the visibility mechanism is argued rather than demonstrated, and anyone who wants to kill this post should attack there rather than looking for a constraint that survives being liftable.

Sources

Royal Society motto, nullius in verba, adopted in the First Charter (1662), after Horace, Epistles I.1 — provenance per the Royal Society’s own history pages (royalsociety.org/about-us/history). Both the 1662 date and the Horace derivation come from there; only the reading below is mine. That reading — the motto as advice about what you submit to rather than about evidence — is an interpretation, not a historical claim about what the founders meant.
Strathern (1997), restating Goodhart — “when a measure becomes a target, it ceases to be a good measure.” Read here as reaching the form of the output and the protocol you score with, not only the metric.
Parasuraman & Riley, Humans and Automation: Use, Misuse, Disuse, Abuse (1997) — misuse, meaning over-reliance on an automated aid, which they catalogue as a well-documented failure mode. Not a claim about how often it happens, and I’ve previously written it down in a stronger form than the paper supports.
Zhong, Raghunathan, Laidlaw & Steinhardt, User Awareness in Frontier Models (Transluce, 2026, transluce.org/user-awareness) — models infer who they are talking to from in-context clues and shift behaviour accordingly: 280 identities across four tasks, 24 models from six families, with recognised safety researchers producing by far the largest shifts and verbalised awareness under 1% on current models. Published the same day as this post, so it is corroboration arriving late rather than prior art I built on. The authors are explicit that the effect is about names as stimuli in training data rather than about the people, that they find no evidence anyone sought the treatment, and that it almost certainly emerged as an unintended artifact of training.
Huang et al. (2024), Large Language Models Cannot Self-Correct Reasoning Yet — their result is about intrinsic self-correction, without external feedback; with reliable outside signal, correction does improve. The step from there to “nothing in the loop scores the measure itself” is mine, not theirs, and is made and scoped in Fidelity Was a Vibe, Cost Was a Number. Pointed at here, not re-derived.

Extends Evals Aren’t the Moat (the specification ceiling, and Goodhart reaching the protocol you scored with — this asks what makes a fix for that hold, rather than which fixes there are). Builds on Fidelity Was a Vibe, Cost Was a Number (nothing in the loop scores the measure itself), You Can’t Grade Your Own Misses (the frozen holdout and blind adjudication this post finally gives a reason for), and Keep the Factory, Ship the Result (fixtures it didn’t choose — named here as precommitment rather than as a design invariant). Adjacent to Your Skills Are a Codebase, Not a Junk Drawer (the harness episode, read here a second way).

Correct Is Not Usable

2026-07-27T00:00:00Z

“Where is the knowledge we have lost in information?” — T. S. Eliot, The Rock (1934)

I asked an agent which of three approaches to take. It gave me three — each described accurately, tradeoffs and costs and failure modes, all of it right. And it was useless: I still had to make the decision it had just described back to me. Every sentence was true and the answer wasn’t there.

That is the whole post. We check an agent’s output for whether it’s correct. Correct is the floor. What decides whether you can use it is whether it carries — and calibrates — what your next move needs. Truth isn’t the axis. Fitness for the decision is.

BLUF

A correctness check can’t catch the failure that matters. An answer can be right in every sentence and still miss the thing your decision turned on — and you won’t catch it by reading the sentences, because they’re all true.
The wrong that hurts is omission, not falsehood. Completeness is a real and punishing axis — the best model scores about 59% on Meta’s GAMUT — and it leaves no mark on the page. (That omission is the worse half is my inference, not GAMUT’s finding.)
Trust is decision-fitness, not sentence-truth: the answer carries and calibrates what the decision needs, without you having to interrogate it out.
Old news for language, new news for agents. Grice and Relevance Theory said “be relevant, not merely true” fifty years ago. With an agent it bites harder, because the failure wears the costume of diligence: more, all correct, and the completeness hides the hole.
The cause is the reward — which is not the fix. An agent is tuned toward what a rater approves: thorough, balanced, confident. Naming that changes nothing.
So install usability as an acceptance criterion — in the harness, not the prompt. My own agent config is a list of exactly those criteria, each written against a default that kept failing it.

Truth isn’t the axis

The model produces a plausible continuation, and none of it is pointed at what you needed. A good-reading answer is complete, balanced, confident — and those are often the opposite of usable. The three-way rundown was the tell: every option fairly weighed, and balanced is the one thing a decision can’t be.

The principle is old. Grice’s maxim of relation — be relevant — and Sperber and Wilson’s Relevance Theory both hold that an utterance is judged by relevance, not truth alone; alignment renamed a slice of it when it split helpful from honest. I’m not claiming the principle. I’m claiming it changes character with an agent, because the miss comes dressed as diligence: it hands you more, all correct, and the extra hides what’s absent.

Correctness gets an answer to the right-hand side of the chart. It still takes explicit acceptance criteria to lift that answer into something the decision can use.

Omission leaves no mark

Run your eye down a correct-but-useless answer and every sentence passes — nothing to catch, because nothing is false. What’s wrong is what isn’t there: the caveat that would have flipped your call, the option it never raised, the bound it never put on the number. That the missing half is real isn’t only my worry — GAMUT scores completeness directly and the best model manages about 59%. It measures the gap; it doesn’t rank omission above falsehood. That ranking is mine, and it rests on one asymmetry: a false claim can be caught by a check, an omission leaves nothing to check. Same shape as a hazard on the human side — you can’t grade your own misses, because a miss produces no output to inspect.

Why the useless answer wins

The agent defaults to the complete-but-useless answer because that’s what the objective pays out on: raters approve of thorough, balanced, confident, well-organised. Approval is a proxy for usefulness, and proxies come apart under pressure — the answer that best earns the thumbs-up (survey every option, hedge nothing that matters, look complete) is not the one that serves your decision (commit, surface the caveat, name the gap). The reward-seeking work out of OpenAI and Apollo is the honest name for the pull.

But that’s the cause, not the fix. “The model is people-pleasing” is a diagnosis you can do nothing with. The move is to stop waiting for the reward to produce fitness, and supply the criterion yourself.

Install the criterion

You don’t get a usable answer by asking nicely; “be concise” fades in a turn, because it competes with the reward instead of replacing it. You get one by making usability a standing acceptance criterion the output has to clear — encoded in the harness.

I know because my own agent config, read plainly, is that list — each line written against a specific correct-but-useless default. Optimise for the next decision, not for completeness. Recommend, don’t survey. Probe with options, not open questions. Do everything you can before asking. Suggest the next thing. Not style preferences: gates, written because a capable, correct agent kept handing me trivia, and the thing that held was making decision-fitness something the output must pass. Same operator move as everywhere here — the reusable asset isn’t the answer, it’s the criterion that keeps producing usable ones.

Reliability is the same gate, one push down

The sharpest version shows up only under challenge. How long does the deep pass take? — twenty to forty minutes. Doesn’t that depend on how much it finds? — fair; a worst case stated as a constant. Rules or architecture? — architecture. Isn’t it both? — both, and they multiply; the either/or was false. The calibrated answer existed. It wasn’t what came out first, because the clean confident version reads better than the bounded true one.

Don’t mistake the concession for the fix. The walk-back doesn’t make the first answer usable — it means you supplied the calibration by hand, and you can only do that for what you happened to interrogate. And don’t over-read the walk-back itself: a model folds on answers that were right, too. Sycophancy is a general behaviour of assistants trained on human feedback, which revise correct responses toward a user’s stated belief (Sharma et al.). So a concession isn’t evidence your challenge landed. It’s evidence there was pressure in the conversation, which you knew, because you applied it. Social psychology drew the line in 1955: informational influence moves a belief because it carries new information; normative influence moves it to match the room (Deutsch & Gerard). “Are you sure?” is normative. A log line that contradicts the claim is informational. Only the second does epistemic work, and a default agent scarcely tells them apart — so the push worth making is the one you can back. A miscalibrated claim fails the same test as an incomplete one: it isn’t fit for the decision, because acting on “twenty to forty minutes” as a constant is acting on the wrong number. So calibration goes where usability went — into the harness: refute your own finding, mark claim-status, don’t hand over a worst case as a constant. Same gate, one layer down.

What you keep

Stop grading agent output like a fact-check. Correct is the floor, and a wrong answer was never the expensive failure — it’s easy to reject. The expensive one is entirely correct and leaves you exactly where you started, because the caveat, the recommendation, the gap never reached the page, and nothing on the page tells you they’re missing. Define what a usable answer to this decision contains, and make that the bar — encoded where it holds, not asked for once. Correct is the easy half. The usable answer is the part you engineer.

How sure I am: the harness is lived — the usability rules above are real, running, and were written against real correct-but-useless defaults; the overstatement-then-concession exchanges are real (some from writing this piece). The core distinction — decision-fitness, not sentence-truth — is my framing of an old principle (Grice, Relevance Theory, HHH), not a measured result; I’m extending it to agents and naming the operational fix, not proving a law. GAMUT backs only that completeness is a real, hard, unmeasured axis (best model ~59%); that omission is the worse half than falsehood is my argument, not its finding. The reward-seeking work names the mechanism; the causal story (“the reward pays out on approval”) is a reasoned account, not something I’ve isolated and measured. The claim I’d defend hardest is the cheap one: correctness can’t detect decision-critical omission, so it’s the wrong acceptance test.

Sources

H. P. Grice, Logic and Conversation (1975) — the cooperative principle and the maxim of relation (“be relevant”); an utterance is judged by more than truth. The principle this extends.
Sperber & Wilson, Relevance: Communication and Cognition (1986/1995) — communication is governed by relevance (cognitive effect against processing effort), not truth. Cited as the fuller form of the same principle.
Askell et al., A General Language Assistant as a Laboratory for Alignment (2021, arXiv:2112.00861) — the helpful/honest/harmless framing; helpfulness as an axis distinct from honesty.
Chen et al., Two-Level Meta-Rubrics for Evaluating Open-Ended Generation — GAMUT (Grounded Assessment of Multimodal Factuality) (Meta, 2026, arXiv:2607.19322) — a benchmark for factual completeness; best model ~59%, so the “missing half” of factuality is real and hard. Cited for that only; the claim that omission is the worse half than falsehood is my argument, not GAMUT’s result.
OpenAI & Apollo Research, Measuring Reward-Seeking via Contrastive Belief Updates (2026, alignment.openai.com) — a model optimising for what it believes the grader rewards. The causal account for approval-over-usefulness, cited as the named mechanism, not proof it drives this specific failure.
Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (2023, arXiv:2310.01798) — intrinsic self-correction without external feedback doesn’t reliably help; backing for “the concession needs an outside push, and the push is you.” Boundary worth stating: this is about intrinsic self-correction, and results vary with task and feedback setup (cf. Madaan et al., Self-Refine, 2023) — so read it as “revision needs outside information,” not “models never improve their own answers.”
Sharma et al., Towards Understanding Sycophancy in Language Models (2023, arXiv:2310.13548; Anthropic) — human feedback encourages responses matching user beliefs over truthful ones, including revising correct answers; sycophancy is a general behaviour of RLHF-trained assistants. The reason a concession is not evidence the original claim was wrong.
Deutsch, M. & Gerard, H. B., A Study of Normative and Informational Social Influences upon Individual Judgment, Journal of Abnormal and Social Psychology 51(3), 629–636 (1955) — the pressure-versus-information distinction, borrowed here for what a human challenge actually transmits. Applying it to an agent is mine, not theirs.

Related: Confidently Wrong (inference stated as observation — a different seam on the same trust boundary); Familiarity Isn’t Trust (fluency inflates trust; here completeness does); You Can’t Grade Your Own Misses (omissions are silent — the human-side version); Evals Aren’t the Moat (a pass certifies only what you specified); The Harness Is the Product (the criterion you install, not the answer you ship).

Build the Operator, Keep the Operands as Data

2026-07-26T00:00:00Z

“Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.” — Archimedes

A good rule set grows. Mine has grown — tested against a whole corpus of real PRs and code — into something too big to hand a model all at once, and that’s the scale problem Rules Are the Alpha ends on. You’ve got a real asset, and it’s outgrown the context window. Now what?

Two obvious answers are both wrong. Cram every rule into the prompt and the model’s attention smears across all of them — lost in the middle, more rules and quietly worse reviews (A Big Rule Set Is an Attention Problem). Bake the rules into the model instead and you can’t edit one without a retrain: your standards go stale between training runs, and a standard you can’t cheaply change isn’t a living standard. Neither the all-in-context nor the all-in-weights answer works at scale.

The way out is to stop treating the rules and the skill of applying them as the same thing. Train the operator — the verb — into the weights; keep the operands — the rules — as editable data.

BLUF

Train the verb, not the noun. Put the skill of applying a rule (read it, check it against the code, try to refute yourself, return a verdict) into the weights. Keep the rules themselves as data.
A new rule is a text edit, not a retrain. The operator is trained once; the operands swap freely. Your standard stays editable.
It’s the synthesis between the linter and the AI reviewer — the linter is all rules and no judgment; the AI reviewer is all judgment and no explicit rules. This is explicit rules applied with judgment.
Each rule sits somewhere on an interpret → JIT → compile spectrum — placed by mechanisability × commonality × judgment-need (my placement policy).
It generalises past review — decision-fork triggers, editable-rubric evals, voice profiles, per-customer rules. Anywhere you want a stable skill over swappable data.
Designed, not built. The trained operator doesn’t exist yet; this is the architecture and the argument for it, leaning on instruction-tuning as prior art. Flagged throughout.

The operator and the operands

Borrow the framing from programming. An operator is the fixed thing that does the work — +, map, a function. The operands are what you feed it. You don’t rewrite map every time you have a new list; you write it once and pass different data.

A rule-applying reviewer has the same shape, and today most systems collapse it into one blob — the rules and the applying are tangled together in a prompt, or tangled together in a fine-tune. Pull them apart:

The operator is the skill: given a rule and a diff, apply the rule, look for the exception, try to refute your own finding, and return a verdict with a reason. That skill is the same whether the rule is about async handling or naming or exhaustiveness. It’s a verb, and it generalises.
The operands are the rules: the target, the exceptions, the examples, the why. Data. Different every time.

Train the operator into the weights and the wager is it applies rules well. Keep the operands as data and you edit your standard with a text change. New rule tomorrow: add the data, don’t retrain the skill.

Why this beats the two blobs

The all-in-context blob spends the model’s scarce attention re-reading the whole rulebook every call, and smears it. The all-in-weights blob is fast and attention-free but frozen — editing a rule means a training run. The operator/operands split takes the good half of each: the skill lives in the weights (fast, no context cost, and it doesn’t need re-teaching), while the rules stay in data (editable, inspectable, versioned). You retrain when your notion of how to review changes — rarely — not when a rule changes — constantly. These are the intended properties, not measured ones: the training frequency, the inference savings, and the held-out-rule gains are all designed, not yet shown.

The test that this is real is easy to get wrong. “Hand it a rule it never saw and check it applies it” isn’t the test — a base model with the rule pasted into its prompt already does that; that’s plain in-context learning, and it proves nothing about training. The real held-out-rule eval is comparative: does the trained operator apply-and-refute a held-out rule better or cheaper than the base model with that same rule in the prompt? If training buys nothing over pasting the rule in context, you haven’t built an operator — you’ve just moved the rule around. That delta is the whole claim, it’s the same shape as instruction tuning (fine-tune on many instructions, generalise to unseen ones), and it’s the eval I haven’t run.

The spectrum underneath

Not every operand wants to live in the same place. Each rule sits on a spectrum from the linter to the LLM, placed by three axes I use — how mechanisable it is, how common, how much judgment it needs (this placement policy is my call, not a measured optimum):

Interpret — the model applies the rule from context. For rare, volatile, judgment-heavy rules. Flexible, editable, expensive per call. This is the default: operands stay data.
JIT to weights (opt-in) — for a very common, very stable rule you can deliberately bake it into the operator on a training run. This is a compilation step, like AOT-compiling a hot path: the rule drops out of context, but it stops being editable data and becomes a frozen operand again — baked into the weights until you retrain. You trade editability for speed and accept the freeze, with explicit invalidation: when that rule changes, you retrain. It’s a second freezing you choose for the stable tail, not the operator absorbing operands automatically.
Compile to a detector — fully mechanisable rules become deterministic code (a lint, an AST check). No model in the loop, exact.

The interpret and compile ends are A Big Rule Set’s territory (keep judgment rules in context, compile the mechanisable ones out to detectors). What the operator adds is the middle: a very common, stable rule can be opted into the weights themselves, so it drops out of context without becoming a separate detector — at the cost of re-freezing that one operand until the next retrain. That’s the exception, not the default: operands stay data unless a rule is stable enough that you’ll rarely want to edit it, and only then do you compile it in. The operator is the constant; most operands stay data; where each one lives is the tuning.

The JIT direction is already visible in the wild. Anthropic cut about 80% of Claude Code’s system prompt for their newer Fable 5 models — a more capable model needs fewer instructions spelled out, because the common guidance is effectively already in its weights (the-decoder, 2026). That’s the general-capability version of the move; the operator does it deliberately, one stable rule at a time. And the deliberate version has been measured: experience distillation bakes an agent’s in-context trial-and-error into the weights and keeps most of the gain — retaining about 65% of the in-context lift where plain fine-tuning kept under 4%, and matching reinforcement learning with roughly 10× fewer environment samples, on software-engineering tasks among others (Gou et al., 2026). Read the other way, it’s the more useful half: the weaker or cheaper the model, the more an explicit rulebook buys you — rules are how a cheaper model makes up ground on a stronger one, spelling out what the stronger model already carries. So how much you JIT tracks the model you run: a strong model lets you drop rules into its weights; a cheaper one earns its keep precisely because the rules stay explicit.

Interpretation, adaptation, and compilation are points on one spectrum: the operator stays reusable while stable operands earn harder execution.

It’s a general move

“Train the verb, keep the noun as data” isn’t only about code review. The same split shows up wherever you want a stable skill applied to changing specifications — though the cases below are my extrapolations from the review case, not separate results:

Decision triggers — the skill of noticing a decision fork is the operator; the specific triggers are operands.
Editable-rubric evals — the grading skill is the operator; the rubric is data you tune without retraining the grader.
Voice matching — the matching skill is the operator; a person’s voice profile is the operand.
Per-customer rules — one trained operator, a different rulebook per tenant.

In each, the failure mode is the same: bake the specification into the skill and you can’t change it without rebuilding the skill. Keep it as data and you can.

A wrinkle: the split leaks (a design hypothesis)

The clean story — train the operator, keep the rules as swappable data — has a seam. A model trained to apply rules bakes in some of the specific rules it trained on, and you can’t surgically evict them; swapping the rulebook at inference doesn’t fully change behaviour. This isn’t new: it’s the knowledge-conflict result — a model is sometimes a “stubborn sloth” that clings to its weights over the context — and the finding that heavily fine-tuned models resist a contradicting system prompt. State it conditionally: a known-possible outcome, not a law.

What decides which way it goes is dose. Train light and the model stays steerable — a new rule in context wins. Train hard and the baked rule resists. That gives “stable earns compilation” its mechanism: hard training trades adaptability for capability, so you compile only the rules that won’t move and keep the volatile ones as data.

And it tells you when to re-compile: not when the rule text changes, and not on every corpus update — but when a diff-eval shows the baked model no longer matches. Generate that eval from where the old and new rules disagree on the corpus — differential testing, run only on what changed — not from the whole new rule set. (The gate is only as good as what it probes — see Evals Aren’t the Moat and You Can’t Grade Your Own Misses.)

How sure: designed, with a bounded pilot. A laptop LoRA run (Qwen2.5-1.5B-Instruct-4bit, 3 seeds per dose, 20 held-out items) showed it directly for surface-form constraints — a trained uppercase or AAA-prefix policy overrode a contradicting prompt on 80–100% of responses once training reached the sampled 30-iteration dose, while the untrained model followed the prompt every time; so the onset sits somewhere in (10, 30] iterations. That is persistence for crisp formatting constraints only. The semantic version was inconclusive — the small model generalised the competing keyword rules by gist, so the trained and prompted rules never cleanly separated. The leak is measured for the surface-form layer; for semantic rules it stays open.

What you keep

The rules are the asset; the operator is what makes them cheap to run and cheap to change. Train the skill of applying-and-refuting once; keep the rules as editable, inspectable data; place each on the spectrum where it costs least. You get a living standard applied with judgment — not a frozen model, and not a prompt drowning in its own rulebook.

How sure I am: this is designed, not built — I have not trained the operator, run the held-out-rule eval, or measured any of it. The lived parts are the asset it assumes (a large rule portfolio tested against a corpus of real PRs and code; a mechanical-detector tier that already runs) and the scale problem that motivates it. Everything about the trained operator — that instruction-tuning-style training yields a skill that applies unseen rules — is a reasoned bet on established prior art (instruction tuning, and a measured context-to-weights result in Gou et al.), not a result I’ve produced for this operator. Treat the whole architecture as a proposal with a clear test attached, not a report.

Sources

Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN, 2021, arXiv:2109.01652) — fine-tune on diverse instructions, generalise to unseen instruction types. This is the direct analogue, and it backs the analogy (generalising to unseen instructions) — not proof that this apply-and-refute skill transfers to unseen rules. The held-out-rule eval is the held-out-task test in this domain.
the-decoder, “Anthropic says it cut 80 percent of Claude Code’s system prompt…” (2026) — reporting Anthropic’s Tariq Shihipar: newer Fable 5 models need fewer instructions and examples. Cited as a real-world instance of the JIT direction (a capable model holds common guidance, so it drops from context); framing it as the JIT-to-weights tier, and the “writing for a weaker model” corollary, are mine.
Gou, Tu, Fang, Cai & Rezatofighi, Sample-Efficient Learning from Agent Experience (2026, arXiv:2607.21051) — “experience distillation” internalises an agent’s in-context experience into weights, retaining ≥64.8% of the in-context-learning gain (vs 3.8% for plain supervised fine-tuning) and matching RL with ≥9.6× fewer environment samples, on 749 software-engineering tasks plus text games. Measured backing that the context-to-weights (JIT) direction works and is sample-efficient — not proof about this specific apply-and-refute operator.
Xie, Zhang, Li et al., Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of LLMs in Knowledge Conflicts (2023, arXiv:2305.13300; ICLR 2024) — under knowledge conflict, in-weights (parametric) knowledge can resist contradicting context (the “stubborn sloth”). Backs the leak as a known-possible outcome, not a law; the reading that a fine-tuned rule resists a contradicting prompt the same way is my extension — a bounded laptop pilot has now shown it for surface-form constraints, but left the semantic case open.
Differential testing (McKeeman, 1998) and regression test selection (Rothermel & Harrold, 1996) — run two versions on the same inputs and diff; run only the tests the change touches. The prior art for the diff-eval re-compilation trigger.

Related: Rules Are the Alpha (the asset + the scale problem this answers); A Big Rule Set Is an Attention Problem (the all-in-context failure + the spectrum); Train on What the Gate Throws Away (where the operator’s training data comes from); Keep the Factory, Ship the Result (the operator + operands are the factory you keep).

The Harness Is the Product

2026-07-26T00:00:00Z

“Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.” — proverb

I built a code reviewer. The thing I’ll actually keep isn’t the reviewer.

The reviewer is the visible product: it reads a diff, applies a rule corpus tested against real PRs and code, and ships the findings worth a person’s attention. But to get it there I had to build a second, invisible thing — the machinery that makes the reviewer good and keeps it good. Mine the rules out of review history. Tighten them against the leads the deep pass refutes. Measure recall against planted bugs. Compile the deterministic ones out to detectors. None of that machinery ships to a user. And it’s the more valuable half.

BLUF

You’re not building a reviewer. You’re building the thing that builds reviewers. The reviewer is the output; the harness — mine, tighten, measure, mechanise — is the factory.
The harness is domain-agnostic; the rules are domain-specific. Point the same machinery at a back-end codebase, re-mine the operands, and you should get a back-end reviewer out of the same tools. The verb transfers; the nouns get re-sourced. (That cross-domain transfer is the post’s bet — designed, not yet shipped; flagged again at the close.)
It’s the operator/operands split, one level up. Build the Operator keeps the apply-a-rule skill fixed and the rules as data. Here the operator is the harness — the skill of making a reviewer — and the operands are a whole domain: its rules, its eval fixtures, its detector configs, its history. Same shape, bigger unit.
It compounds across time, not just across domains. Re-run the levers over newer PRs and newly-added rules and the same reviewer keeps improving. The harness is a flywheel, not a one-off scaffold.
This reframes the moat. Rules Are the Alpha says the rules are the moat — true, per domain. The harness is the cross-domain moat: it’s what mints the next opinionated corpus cheaply. The rules are the alpha; the harness is the factory.
Honest scope: the harness is real and running (mining, tightening, evals, detectors). Repointing it to a new domain is designed, not done — I’m arguing from the shape of the tools, not a second reviewer I’ve shipped.

The visible product and the invisible one

Watch what actually ships. A user of the reviewer sees findings on a diff — that’s the product. What they never see is everything that made those findings trustworthy: the loop that mined the rules out of years of review comments, the optimiser that tightened each rule against the leads the deep pass kept refuting, the eval that plants known bugs to measure whether recall held, the detectors that took the mechanical rules out of the model’s way. That machinery runs at build time, not review time. It never touches a user’s PR.

And it’s most of the work — the split I’ve lived building it, not an economic law I can point to. The rules — the opinionated corpus everyone points at as the asset — are comparatively cheap to re-source once the machinery exists. Building the machinery the first time is the hard, slow part. Which means the durable thing I made isn’t the reviewer, and isn’t even quite the rules. It’s the factory that produces and maintains them.

One honest bound on “comparatively cheap to re-source,” because cheap is not the same as unattended. I’ve since handed most of the rule authoring to the machine — drafting, tuning, compiling detectors — and the cost that survived isn’t any of those. It’s a handful of upstream calls the machine gets wrong while producing something that looks entirely correct: naming a mechanism that already exists, re-weighting an objective, amending the rule instead of narrowing the code, changing the substrate. The factory does make the rules now. Someone still has to be standing at the right end of it, and that part hasn’t got cheaper. (Receipts in Rules Are the Alpha.)

The harness is the operator, one level up

If you’ve read Build the Operator, this is that idea moved up a level. There, the operator is the skill of applying a rule and refuting yourself; the operands are the rules, kept as editable data. The point was: train the verb, keep the noun as data.

The harness is the same move on a bigger unit. The operator is now the skill of manufacturing a reviewer — mine, tighten, measure, mechanise — and it doesn’t care what domain it runs on. The operands are a whole domain: front-end rules mined from front-end history, or back-end rules mined from back-end history. A back-end reviewer isn’t a new machine. It’s the same operator with new operands. “Train the verb, keep the noun as data” becomes “build the harness, keep the domain as data.”

Across domains, and across time

There are two axes of reuse here. Keep them apart.

Across domains is the obvious one: point the harness at a new codebase and re-source the operands — and that’s more than the rules. A new domain needs its own eval fixtures and oracles to measure recall against, and its own detector configs to mechanise the deterministic checks; all three get re-mined from the new domain’s history. What doesn’t change is the harness itself — the skill of producing and tuning that operand set. That’s the whole claim: re-mining a rules file alone wouldn’t hand you a back-end reviewer, but re-running the harness over back-end history produces the rules, the fixtures, and the detectors together. The corpus is new; the factory is not.

Across time is the one that’s easy to miss. Re-run the levers over newer PRs, and over rules you add later, and the same reviewer stays current as the codebase and the standards drift out from under it. That’s the flywheel — the harness isn’t scaffolding you tear down after launch, it’s one you keep running.

One honest bound on the flywheel, because it isn’t perpetual motion: on a frozen corpus it stalls. Once the detectors do most of the finding, the stream of refuted leads the tightening feeds on thins to nothing — a dead faucet. What keeps it fed is exactly the two inflows above: new PRs and new rules. The flywheel runs on the flow of real new work, not on itself.

The reusable asset is the machinery that remakes the operands for a new domain and refreshes them as that domain changes.

It isn’t even specific to code

The clearest second instance is the one that made this post. I built a writing gauntlet while doing an earlier long-form series: mine the real receipts, run the draft past substance and voice panels and a judge from a different model family, take the real hits, and — for the pieces that need it — delete the draft and rewrite from memory so only the load-bearing parts come back. It didn’t retire when that series shipped. Every piece of writing I’ve done since has gone through it, the two books included; this post went through the whole loop, blind rewrite and all — a from-memory rewrite went up against this version in a blind, different-family judge, and this is the one that won. The harness pointed at itself. A reviewer and an essay share nothing as products. The thing that makes each one good is the same kind of asset: built once, then pointed at whatever’s next.

What this does to the moat

Rules Are the Alpha argues the rules are the moat, not the model. I still think that’s right — per domain. But it undersells the asset. A competitor who copies your rules file gets one domain’s corpus, frozen at the moment they copied it — a snapshot with an expiry date, the day your codebase moves on and theirs doesn’t. What they don’t get is the thing that produced that corpus and will produce the next one, and keep both current. Copying the output doesn’t copy the thing that produces the output. The rules are the alpha; the harness is the factory that mints alpha on demand.

Someone has now put numbers on a narrower version of that asymmetry. SkillOpt — a Microsoft-affiliated paper from May 2026 — treats an agent’s skill file as external state of a frozen model and optimises it the way you’d train weights: an optimizer proposes bounded edits to one document, a held-out split accepts an edit only on a strict score improvement, and what ships is a single file. Run that optimizer in place and it was best or tied in all fifty-two (model, benchmark, harness) cells they measured — fifty-two settings, the machinery re-run in each one. Re-running the harness works. Then they moved the file somewhere it wasn’t built for — a smaller model in the same family, or a different agent harness — with no re-optimisation. The transplanted skill kept anywhere from a tenth to 1.4 times the gain it produced at home. Copying what the harness produced pays out somewhere in that range, and you don’t learn where until you measure it in the new setting.

The floor is the part that cuts against me: “no row falls below the target’s no-skill baseline.” On every row they tested, a copied artifact was worth more than nothing. So the frozen snapshot a competitor walks off with isn’t inert — it’s unreliable. That’s a weaker claim than the one I just made, and the one I’d rather defend.

So the moat has two layers. The rules are the deep, narrow one — specific, opinionated, hard to fake for a given codebase. The harness is the wide one — it’s what lets you open a new front cheaply, the designed-not-done bet this whole post rests on. If anything, my wager is it’s the bigger asset, precisely because it’s the more transferable one. That’s a complication to “the rules are the alpha,” not a retraction: the rules are still where the per-domain edge lives; the harness is where the next edge comes from.

What you keep

Ship the reviewer. Keep the harness. The findings are the output a person sees and acts on; the machine that makes and maintains the reviewer is the asset that doesn’t leave the building. When the model becomes a commodity — the market thesis of Rules Are the Alpha — the durable thing isn’t the reviewer either. It’s the factory that can build the next one, in a new domain, next quarter, and keep the last one honest as the code moves.

You’re not building a reviewer. You’re building the thing that builds reviewers.

How sure I am: the harness is lived — mining from review history, tightening rules against refuted leads, recall evals against planted bugs, and a mechanical-detector tier are all real and running, genericised here. The operator/operands framing is my inference, settled in my own design notes, not a published result. Designed, not done: repointing the harness to a new domain (a back-end reviewer from the same machinery) is argued from the shape of the tools, not a second reviewer I’ve shipped — treat that as the load-bearing bet. The across-time flywheel is real but fueled: on a frozen corpus the drop stream dries up, so the flywheel depends on new PRs and new rules, not on itself. The SkillOpt numbers are someone else’s measurement of a different transfer than my bet — their axes are model scale, harness, and benchmark, all within one model family (GPT-5.4, its mini and nano variants, and GPT-5.5); mine is a whole domain. Their own limitations section says held-out evaluation stays necessary “before transferring them to substantially different models, harnesses, or task settings.” So it corroborates the shape of the harness-versus-artifact asymmetry. It does not test the cross-domain repointing this post rests on.

Sources

Build the Operator, Keep the Operands as Data — the operator/operands split this generalises; the harness is the operator one level up.
Rules Are the Alpha — the per-domain moat this reframes: rules are the alpha, the harness is the factory that mints them.
Keep the Factory, Ship the Result — the sibling argument (keep the generator, ship the warranted result); this post is what “keep the factory” turns out to be worth.
Yang et al., SkillOpt: Executive Strategy for Self-Evolving Agent Skills (2026, arXiv:2605.23904) — a frozen agent’s skill document optimised in text space. Run in place, SkillOpt is best or tied in all 52 (model, benchmark, harness) cells; transplanting the artifact with no re-optimisation retains 10% to 140% of its in-domain gain, and “no row falls below the target’s no-skill baseline.” Transfer tested across model scale, harness, and benchmark — within one model family.

The Raw Is the Trust Boundary

2026-07-26T00:00:00Z

“In God we trust; all others must bring data.” — W. Edwards Deming

Ask a model to summarise a document and you can hold the result against the document — the raw is right there. Ask it to write a feature and there’s nothing to hold it against but your own judgment, and it drifts to the mean — hand-rolled where the platform had a primitive, an effect where a plain value would do. Same model, same session. It’s tempting to call that “reads better than it writes,” but that’s not the mechanism, and the mechanism is the useful part.

There’s no reading organ and no writing organ. A model does one thing: take a sequence of tokens, produce the next, repeat. The summary is “written” the same way the feature is. So the difference isn’t the activity. It’s whether there’s a raw — a ground truth the output can be checked against.

BLUF

It’s all input → output. There’s no read/write faculty; there’s one operation. The real axis is whether there’s a raw to hold the output to.
Reading tasks come with a raw. Summarise, review, extract — the input is the ground truth; the output is pinned to it, so its errors are checkable (not automatically absent — checkable).
Generation comes without one. The output is drawn from the weights; nothing was handed in to check it against. The only ground truth is your experience — so you’re the raw.
Trust tracks the pin. Fully pinned (summarise) → crisply pinned (find this in this code) → loosely pinned (“is this good?”) → unpinned (write from a blank prompt). How far you can trust the output follows the pin down.
The whole discipline is manufacturing a raw. Rules, tests, examples turn your in-your-head standard into an external ground truth; a review pass is then the check run against it — so the check can leave your head and be delegated or mechanised.
Two honest bounds: the model is a reliable checker against a crisp raw; it goes soft on open-ended judgment, and it can’t be its own raw — self-review doesn’t hold.

It’s all input and output — the difference is the raw

Strip the mysticism and the model emits high-probability continuations. When you hand it a document, it first encodes that input across every layer — the prefill — so the raw is literally sitting in the context. A summary is generated pinned to that encoded input: a selection and compression of something present, a small and grounded search you can check against the source. A blank-prompt “write me X” has almost nothing in the context to pin it, so the continuation is driven by the priors — pulled toward the most typical thing the training distribution has seen (what I’ve been loosely calling the mean). Same operation; wildly different amounts of ground truth in the room.

Reading-shaped tasks — summarise, classify, extract, review this for X — arrive with their ground truth: the input. Generation-shaped tasks don’t. The output came from the weights; there was no raw handed in, and the only thing to match it against is your own sense of what good looks like. You are the raw, and the judgment is matching what it produced to what you expected.

One qualifier: generation isn’t raw-less in principle. A test is a raw (does it pass?), a spec is a raw, the compiler and the runtime are raws. What’s missing by default is a raw for “is this what I meant, is this good” — and you are the default raw for that, until you manufacture an external one.

Trust tracks the pin

So reliability isn’t about reading vs writing; it’s about how tightly the input constrains the answer. A crisp criterion — “does this function drop any items?” — is a strong pin: a bounded check against something concrete. “Is this code good?” is a weak pin. “Write a feature” is no pin at all. This is the intuition behind what the field now calls the generator–verifier gap, or Jason Wei’s Verifier’s Law: for many tasks, checking a specific claim is a smaller, more grounded job than generating a good answer from scratch. It’s an asymmetry, not a theorem that holds everywhere — some things are as hard to check as to do.

I ran a small version of this. I took one paginated fetch, planted a specific data-loss bug in it, and asked blind reviewers — on two model tiers, a frontier model and a small one — to check it for correctness. Twenty for twenty caught it. Be precise about what that is and isn’t: it’s one planted bug, strongly pinned, that both tiers caught every time — evidence that a crisp criterion is a strong pin on this fixture, not proof that “the model is a reliable checker” in general (one bug × twenty runs can’t carry that). It also does not show “reads better than writes”: the writing task it’s set against was open-ended and hard, so that contrast only isolates task difficulty. The narrow, clean reading is the one I’ll lean on: crisp criterion, strong pin.

The tighter the output is pinned to raw, checkable evidence, the more trust it can earn. An account of itself sits at the opposite end.

Manufacturing a raw

Once you see that trust tracks the raw, most of the discipline around AI stops looking like separate tricks and starts looking like one move: manufacture a raw for output that didn’t come with one.

Tests are a raw for behaviour.
Rules are your expectations encoded into a raw a reviewer can be checked against — that’s what makes them the asset (Rules Are the Alpha).
A clean codebase is a partial raw: the model conditions on the surrounding code and matches it, so good code in pins good code out (Your Codebase Is the Prompt).
A review pass isn’t itself a raw — it’s the check; but it runs the model against the raws you’ve made (rules, tests, criteria), catching after the fact what the generation had no ground truth to avoid (Keep the Factory, Ship the Result).

Each one takes a standard that lived only in your head and makes it external, so the check can be delegated to a model or a machine instead of costing you a read. The spine of all of it: externalize your raw where you can; the rest stays you. A raw doesn’t make the output right — it makes it checkable; the trust is earned when the check runs, not when the raw exists.

Which raises a question you have to answer for every raw you make: who is it a raw for? It’s easy to assume the answer is “both of us,” and that assumption is doing more work than it looks. BabelTele compresses text into forms a person can’t read and a matched model still can — 99.5% semantic fidelity at 27.9% of the original length. The authors are careful about the scope: the three properties (human-readability, natural-language fluency, machine-recoverability) come apart only partially, and the results hinge on the specific compressor–reader pairing and the task. Read narrowly, it still bites — under a matched pair, readability and recoverability separate. So a raw can pin the model beautifully and be useless to you at the moment you want to check the model, which is the moment this post says the trust is actually earned. That’s a trade, not a free compression win. Compress the raw and you’re deciding, on purpose or by accident, that this particular check is one you’ll never run by hand.

The two bounds

The model-as-checker is powerful and bounded, and the bounds matter:

A crisp raw or nothing. “Find the data-loss bug” is a tight pin the model nails. “Is this well-designed?” barely pins anything, and there the reliability I’d claim falls away — the model can’t check against a raw you haven’t made concrete.
It can’t be its own raw. Ask a model to review what it just wrote and it tends to bless its own mistakes; in the reasoning tasks they studied, Huang et al. found intrinsic self-correction without external feedback didn’t reliably help. The raw has to be independent of the writer — a separate check, a different model, a deterministic test.

What you keep

The model is trustworthy exactly to the degree there’s a raw to hold its output to. Reviewing against a crisp criterion has one, so you can hand that to the machine. Generating to a standard has none but you — until you encode one as a rule, a test, or an example. So the move is: give the model crisp, raw-backed checks; and everywhere there’s no raw yet, either be the raw or make one. The trust boundary and the raw boundary are the same line.

How sure I am: the PR (an agent’s feature, corrected commit by commit) and the small experiment are real, genericised. The experiment is bounded — one planted bug, one clean/mean pair, twenty blind reviews across two tiers of a single model family — and it supports only the narrow claim it’s used for: a crisp criterion is a strong pin the model checks reliably. It is not evidence that reading beats writing in general (that comparison is unpaired — an open task vs a closed one). “It’s all input and output / trust tracks the raw” is my framing of the mechanism, backed by the cited results, not a measurement. The self-review bound leans on Huang et al.; the generator-verifier framing on Wei and the surrounding literature — real, cited narrowly. BabelTele’s numbers are measured and theirs, and I’ve kept their qualifiers (partial decoupling, conditional on the compressor–reader pair) because dropping them would make the point bigger and wrong; the who is the raw for question I hang off it is mine.

Sources

W. Edwards Deming — “In God we trust; all others must bring data” (widely attributed; the epigraph, not a load-bearing cite).
Jason Wei, Asymmetry of Verification and Verifier’s Law (blog, jasonwei.net, Jul 2025) — some tasks are far easier to verify than to solve (an asymmetry, not a universal law); the generator–verifier framing the post leans on.
Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (arXiv 2310.01798, ICLR 2024) — intrinsic self-correction without external feedback doesn’t reliably help. The “can’t be its own raw” bound.
Zhu, Peng, Wang, Ke, Zhang & Zhang, Large Language Models Do Not Always Need Readable Language (BabelTele) (2026, arXiv:2606.19857) — text compressed out of human readability that a matched model still recovers, at 99.5% semantic fidelity and 27.9% of original length. Cited for the narrow claim the paper makes: the three properties are only partially decoupled, and outcomes hinge on the specific compressor–reader pairing and task. Turning that into “so ask who your raw is for” is my extension, not their finding.
Cobbe, Kosaraju, Bavarian, Chen et al., Training Verifiers to Solve Math Word Problems (2021), arXiv:2110.14168 — a trained verifier ranks generated solutions (GSM8K); cited as the generator–verifier gap in practice, by analogy, not proof about code review.

Related: Your Codebase Is the Prompt (a clean codebase is a partial raw the model matches); Rules Are the Alpha (rules = your expectations encoded into a raw); Keep the Factory, Ship the Result (the review pass is the check run against your raws, after the fact); Give Away the Writing (give away the generating; stay the raw); You Can’t Grade Your Own Misses (why it can’t be its own raw, human-side); You Can’t Save a Layout You Don’t Own (one concrete raw — the layout tree you keep, with the screen as its render).

Train on What the Gate Throws Away

2026-07-26T00:00:00Z

“We learn from failure, not from success.” — Bram Stoker

My code reviewer does far more work than it ships. Scouts cast a wide net — anything that might be a violation. A deep pass then refutes most of those leads, and a precision gate ships only the survivors: the few findings I’d actually put in front of a person. The gate is deliberate — a reviewer that cries wolf loses the right to be read, so it over-generates coverage inside and ships hard-filtered outside.

But look at what the gate throws away. Every refuted lead is labelled: “looked like a violation, wasn’t” is a hard negative. Every confirmed one is a positive. The reviewer computes its whole candidate surface — every lead its scouts raised, each adjudicated with a reason — and then bins almost all of it. That bin is the labelled dataset you’d want to train a better reviewer (bounded, as we’ll see, by what the scouts surface in the first place). And you already paid for it.

BLUF

Your reviewer’s reject pile is labelled training data. Every lead it refutes is a hard negative; every one it confirms, a positive-with-confidence. The gate discards this whole adjudicated candidate set after computing it.
It’s the constructive flip of survivorship bias. Don’t train on merged code (the survivors); train on your reviewer’s rejects — the adjudicated leads the gate throws away.
The labelling is nearly free. You spent the compute adjudicating when you ran the review; you’re just not keeping the verdicts. (Storing and training on them isn’t free — the labels are.)
Capture the pre-gate lead→verdict pairs, not the shipped findings. The findings are the survivors; the training signal is the whole pile.
Design-forward, honestly: I run the harvesting loop (drops → recorded exceptions → mechanised) and tighten the scout against the drops with a recall floor (measured, modest); I have not trained a model on the pile. Those uses are lived; the model-training is argued.

The gate computes recall, then spends it

Picture the pipeline as three stages. Scouts raise leads — wide, cheap, tuned for recall, because a lead never raised is a bug never caught. A deep pass verifies each lead and refutes the ones that don’t hold. A precision gate ships only what survives. Recall lives inside the pipeline; precision is enforced at the exit. That split is on purpose: over-generate coverage where it’s cheap, then gate hard so the human only ever sees findings worth their attention.

The gate is also where the recall you computed gets spent. Everything below the line — every refuted lead, with the reason it was refuted — is calculated and then dropped. You ran a classifier over your own codebase and threw away every label except the handful you shipped.

Refuted isn’t worthless — refuted is labelled

The part that’s easy to miss, because “refuted” sounds like “noise”: a refuted lead is one of the most valuable kinds of training example there is — a hard negative, something that looked enough like a violation to get raised, and wasn’t. Hard negatives are what you normally pay annotators to find; a naive dataset is all easy negatives and a few positives, and it teaches a model nothing about the boundary. Your reviewer manufactures hard negatives for free, every run, at exactly the decision boundary you care about — and then deletes them.

Training a verifier on labelled generations is a standard move (Cobbe et al. trained verifiers on model outputs to great effect). The analogy isn’t exact — their labels were externally checkable right/wrong answers, the reject pile’s are the deep pass’s own fallible verdicts — but the shape is the same: labelled generations, pre-built. Lead → verdict, with a reason, at the boundary. You are sitting on the training data and shipping it to the bin.

One caveat, and it’s load-bearing: those verdicts are the labels, and they’re the deep pass’s own — fallible. A real violation the pass wrongly refuted becomes a mislabelled negative, so training on the pile teaches this reviewer’s judgment, misses and all. The pile is only as good as the adjudicator that built it: it compounds your reviewer, it doesn’t transcend it. That’s a ceiling to respect, not a reason to keep binning the labels.

The flip side of survivorship

This is the constructive complement to a hazard I’ve written about before. Merged code — the final snapshot on the main branch — is survivorship-biased: it’s post-cleanup, so the violations that got caught are missing from it (they may still live in intermediate commits; it’s the merged snapshot that’s scrubbed) — train on that and you learn from the bombers that came back. The reject pile is the other direction entirely. It’s precisely the caught-and-adjudicated leads — the violations and the near-misses — that merged code no longer contains. So the reject pile fills the caught hole: the labelled recall surface the obvious data can’t give you.

Only the caught hole, though — and that bound matters. A violation no scout ever raises never enters the pile, so the pile is capped by your scouts’ recall: its own survivorship line, one level up. It’s the honest signal for everything your reviewer surfaced, not for what it never saw. Two loops, opposite signs: the merged snapshot is the biased data you should be wary of training on; the reject pile is the honest-but-bounded signal you’re throwing away. Adjacent, not the same.

The shipping gate keeps only precision. The learning loop should retain every adjudicated lead on both sides of that gate.

The loop I actually run

The lived part, so I’m not selling you a plan as a practice: I run a harvesting loop now. Run the deep reviewer across the codebase; for every lead it drops, capture the reason; encode that reason as an exception on the rule that fired (and mechanise it where it can be made deterministic — the how of that migration is its own post). What the loop harvests is the point here: a growing, labelled set of adjudicated leads, captured instead of binned.

And the labels already do more than seed exceptions. A refuted lead is a negative, and negatives are exactly what you need to make the scout fire less without firing worse: I run the drops through a prompt optimiser that tightens each rule’s one-line scout summary so it stops raising the pattern that kept getting refuted — with a hard recall floor, so any tightening that would silence a known true catch is rejected outright. Across the two-dozen-odd rules I ran it over, a couple moved precision from about a third to a half with recall held; roughly half didn’t budge, and two were rejected outright for dipping below the recall floor. The eval sets are tiny — one curated positive plus eight-to-ten mined negatives per rule — so this is directional, not a proof, and the easy headroom looks mostly spent. Modest, but real and measured — the pile tightening the reviewer that produced it.

So the pile already pays off twice, both lived: exceptions encoded, and scout summaries tightened against the negatives. What I have not done is train a model on the accumulated pile — the obvious next step, and still argument, not result. Prompt-optimising a summary against the drops is a long way from moving the reviewer’s judgment into weights: the first is running, the second is designed. I’d rather say that plainly than imply a trained model exists.

What you keep

Stop throwing away the labels. Your reviewer’s real product isn’t the findings it ships — it’s the labelled recall surface it computes to decide what to ship. Send the survivors to the human; keep the whole adjudicated pile for the factory that makes the reviewer better. The gate is the right place to be strict with a person’s attention and the wrong place to be strict with your own data.

How sure I am: the pipeline (scout → deep → gate), the precision gate, and the harvesting loop are real and running, genericised here. Two uses of the pile are lived: exceptions encoded from the drops, and the scout summaries tightened against those drops with a recall floor — the tightening was measured but modest (a few rules improved, the broad pass mostly a null result). What I have not done is train a model on the pile: “the reject pile is model-training data” is a sound argument from how the pipeline works, not a demonstrated result. The generator-verifier framing is real prior art; the specific yield of this pile as training data is unmeasured. Treat the model-training claim as designed, not demonstrated.

Sources

Cobbe, Kosaraju, Bavarian, Chen et al., Training Verifiers to Solve Math Word Problems (2021), arXiv:2110.14168 — train a verifier on labelled model generations (GSM8K). Cited by analogy: their labels were externally checkable, the reject pile’s are self-adjudicated — the shape of the move, not proof this reviewer improves.
Your Merged-PR History Is Survivorship-Biased Training Data — the hazard this is the constructive flip of (the author’s own post).

Related: Your Merged-PR History Is Survivorship-Biased Training Data (the biased data to avoid; this is the honest data you discard); You Can’t Grade Your Own Misses (recall is what the gate spends); Rules Are the Alpha and A Big Rule Set Is an Attention Problem (the corpus + the exceptions the harvest feeds); Keep the Factory, Ship the Result (the pile feeds the factory, not the shipped output).

Keep the Factory, Ship the Result

2026-07-25T00:00:00Z

“Civilization advances by extending the number of important operations which we can perform without thinking about them.” — Alfred North Whitehead

I built an eval to tell me whether my agent’s skills were earning their place. For a while it said they weren’t — clean, green, confident. Then I looked at how it actually ran: it injected every skill on every case, whether or not the skill should have fired. I’d frozen the wrong test. When I gated it to load a skill only when it should actually trigger, the verdict reversed — the skills clearly helped, most of all for the cheaper models. The eval hadn’t drifted from a moving world; it had frozen the wrong world from the start. But the lesson generalises, because the world moves too: a rule that said use class-based components was right for years and is now simply wrong — hooks and function components moved the standard out from under it. Nobody repealed that rule. The ground did.

An eval is judgment, frozen. A rule is a standard, frozen. A trained agent is judgment frozen at training time. Freeze any of them and it starts going stale — either you froze the wrong thing, as I had, or the world moves out from under the right one, the way a reading eye trained on last year’s idioms stops snagging on the ones since deprecated. Everything you ship is frozen judgment, and frozen judgment drifts. You can’t fix that by refusing to freeze; a standard you can’t write down isn’t a standard. You fix it by putting the ceremony on the right layer — a small principle constitution you amend deliberately and rarely, under process, rather than one you never touch, with the specific, operational rules changing faster underneath it. Constitution versus statute: the two layers change at different rates and with different ceremony — not a core frozen for good and a fluid rest. Freeze everything and you go rigid; freeze nothing and you drift. Get the layer wrong and your evals rot the way a trained reflex rots. That’s the argument, and it’s the part of the current agent excitement nobody is writing down.

A word on where this sits, because the lane next to it is crowded. The workflow-versus-agent line is already drawn, and drawn well — Anthropic’s Building Effective Agents is the canonical cut (workflows are predefined code paths; agents dynamically direct their own processes), and a 21-pattern catalogue covers much of the rest. I’m not re-drawing that line. I’m going a layer down — past the question of what shape your system is, to the thing you verify with, and why that thing expires. One tell the taxonomy writing shares: it shows the happy path and never the failure mode, which is survivorship bias pointed at the writing itself. Here the failure mode is the whole point, so I’ll lead with where it breaks.

BLUF

Everything you ship is frozen judgment, and frozen judgment drifts. A mechanical check and a trained agent both go stale. The factory lasts longer only because it’s the part you keep maintaining.
The standard needs a constitution. Anchor a small principle core; let the specific rules change under it. Otherwise your evals rot the way your reflexes do.
Non-determinism is a generator, not a runtime. You don’t want unwarranted probabilistic behaviour on the critical path — you want output you can check. The agent’s job is to produce something warranted, then get out of the way. The same boundary runs inside the system: decompose the agent the way you’d decompose a function and the model quarantines itself to the stages that need it — in my reviewer, two of the five stages need no model at all.
Keep the factory, ship the result. The factory — the loop that mines and mechanises the rules and keeps them good — is the half you maintain; the mechanical checks it emits (and, one day, trained agents) are the perishable output you ship. (That this factory also transfers across domains and time is its own argument — The Harness Is the Product.)
Persistence is a feedback edge. Saving state that later runs read back turns a stateless workflow into a loop — an edge from this run’s output into the next run’s input. That opens a feedback edge you have to govern; the risks are drift, gaming, and lost reproducibility. Keep the fast path pure; quarantine every write into a governed, human-gated outer loop. Two safeguards are weaker than they sound: suggest, never suppress still suppresses if the suggestion eats a finite budget, and the human gate stops the dataflow without breaking the loop of influence.
The human doesn’t leave; the job moves up. Checking the agent’s code may be temporary. Owning the standard — deciding what “correct” means — isn’t, because an optimiser can’t originate the target it optimises toward.

The standard needs a constitution

Start with the fix, because it’s the load-bearing idea: tier the standard. Keep a small principle core — a mental model and a handful of rules you’ve authored deliberately and amend rarely, under process, not a core you never touch. Let the specific, operational rules change underneath it, derived from current practice but constrained to serve the principles. This is constitution versus statute: the constitution is hard to amend, not never amended; the statutes track conditions but must stay consistent with it. The structure isn’t new. It’s how a legal system works, and it rhymes with Lakatos’s picture of science, a protected hard core wrapped in an adjustable belt of hypotheses (though in that picture whole programmes still get abandoned — nothing is permanent). Constitutional AI is the nearest thing in AI, a small set of principles steering a model’s behaviour, though it uses them to train rather than as a live frozen/mutable split. Freeze everything and you’re rigid — the standard can’t track a moving target. Freeze nothing and you drift — the rules chase fashion and get gamed. The frozen core is what tells you a specific rule has gone stale: a rule that no longer serves a principle is done.

I have a head start here that most people don’t, and it’s the concrete version of all this. My principle core is already written — the mental model and rule set in the two books I wrote about thinking about code. That’s the constitution. The operational rules in the reviewer are the statutes, and the factory’s real job is to keep the statutes consistent with the constitution as the world changes. There’s even a test for whether those principles are really principles: try generating a rule set for a domain the books weren’t written for. A principle that can’t produce a good rule somewhere new is suspect — probably a local habit wearing a principle’s coat. Passing doesn’t prove it’s bedrock, since a merely general heuristic travels too, but failing is a strong tell.

The constitution is small and slow-moving; the operational rules are wider and allowed to move, but only underneath that constraint.

Non-determinism is a generator, not a runtime

The hype treats the non-deterministic system as the destination — the goal is an agent that dynamically directs itself. But for almost anything that matters, an unwarranted probabilistic runtime is the opposite of what you want. You want the code that runs your payments to behave the same way every time — or at least to trip a check when it doesn’t.

Non-determinism earns its place as a generator: you use the messy, probabilistic, occasionally-brilliant agent to produce something you can warrant — a function, a rule, a check — and the thing on the critical path is that warranted artifact, not the raw agent. Usually the artifact is deterministic, because deterministic code is the cheapest thing to warrant: you test it, and it runs the same way at 3am under load. That’s the common case, and it’s why “generate the deterministic thing, then ship the deterministic thing” is the default.

The precise claim is narrower than “determinism good, non-determinism bad.” You can keep a probabilistic component at runtime — a model in the loop — as long as it sits behind a deterministic warrant that catches the specific failure the check encodes. What you can’t ship is unwarranted non-determinism on the critical path, which is exactly what “just run the agent graph in prod” asks for. The agent is good at what non-determinism is good at: exploring, drafting, covering cases you’d miss. The warrant is what makes its output safe to depend on — for the property it encodes, not in general. Confuse the generator with the guarantee and you’ve put the exploration engine where the guarantee should be. The exception is genuinely open-ended work — a brainstorm, an exploratory search, a conversation with no single right answer — where you run non-deterministic and simply forfeit the warrant, because you can’t certify what you can’t check.

And none of this is new — only the layer is. The producer was always non-deterministic: a person writing code is a non-deterministic process that yields a deterministic product, and the product is deterministic on purpose — a machine that does the same thing every time is the whole reason we reach for computers in the first place. “Get comfortable with non-determinism” and “software just becomes English” have it backwards; you can’t run English, you run the artifact. What actually changed is that AI slid a second non-deterministic layer between the intent and the artifact — intent, once through a person, now through a model, and only then the deterministic thing. So the artifact needs checking one layer deeper than before, and that checking — below the AI layer, against the determinism you actually wanted — is where software engineering has moved.

Draw the boundary inside the system

The generator-versus-runtime split has a sharper form once you look inside the system. The useful thing isn’t a graph of five clever agents; it’s a deterministic factory with a couple of deliberately non-deterministic stages inside it. The boundary between “needs a model” and “must not have one” doesn’t run around the agent — it runs through the pipeline, and most of the pipeline lands on the deterministic side.

Concretely. My reviewer began as one do-everything agent: here’s the diff, here are the rules, find the problems. Non-deterministic, expensive, untestable — a monolith. It’s now five single-concern stages. A scout notices candidate sites; a gate prunes them; a deep pass judges what survives; a decision step applies the matrix; a write step renders the verdict into prose. Name what each stage does and the boundary draws itself: the scout and the deep pass need a model, because noticing and judging are perception. The gate and the decision are pure functions — no model at all. The write step is a cheap model doing nothing but rendering text. The expensive, non-deterministic model is quarantined to the two stages that actually need one; the wobble has nowhere else to leak, because everything around it is code.

None of that is a new idea, and that’s the point. It’s the oldest idea in software — make each thing do one thing — applied where people stopped applying it. We treat agents as magic: tell it what you want, paste the whole world into one prompt, and hope. Nobody would build a codebase that way; everybody builds agents that way. Decompose the agent the way you’d decompose a function and determinism, cost, and testability fall out as consequences, not features you bolted on.

The graph itself isn’t the insight — everyone has the graph. DSPy names the pieces (typed signatures, composable modules, optimisers that tune each stage); model cascades route the cheap model to the easy step; the compound-AI writeups make the same model-to-system case. None of them forbids the part that carries the boundary here — plain deterministic code doing the stages that need no model, composed around the ones that do; DSPy wraps ordinary functions around its modules too. So this isn’t a move past any of them. It’s the ordinary engineering they all allow and agent-builders skip, because the pull is to make every stage a model call — the hype is selling the model, and a gate that’s just a function doesn’t look like the product. I rebuilt these shapes by hand before I knew their names, which is the cleanest evidence I can give that it’s engineering, not a framework’s trick.

One of them does come with a number, and it’s a number on the flow rather than on the model. AlphaCodium wraps a fixed, ordered pipeline around GPT-4 — self_reflection, then possible_solutions, then generate_ai_tests, then initial_code_generation, then iterating the code against public_tests and against ai_tests — and on the CodeContests validation set that took pass@5 from 19% to 44% against prompting the same model directly. Same model, better factory. The part I’d underline here is the cost: 15–20 model calls per solution, four orders of magnitude fewer than AlphaCode’s brute-force sampling. The deterministic scaffolding is not the expensive half of the system. It’s the thing that lets you stop paying for the expensive half.

The boundary runs through the pipeline, not around it: model perception at scout and deep review, pure functions at gate and decision, and a cheap model only for rendering.

Keep the factory, ship the result

What’s actually reusable isn’t your personal graph. A graph you hand-carve for one task is a scaffold you throw away when the job’s done, and it only ever helped you. The reusable asset is a shared workflow: one box, rules baked in, that your whole team pushes work through. One person shouldn’t rebuild the reviewer for every pull request; the team runs every request through one reviewer. And the value there lies in what the workflow knows (the rules) and the machine that keeps those rules good, not in the topology. That machine is the factory, and it works the opposite way to “run an agent graph in production.”

The learning loop — the part that makes the reviewer better — I run by hand. Slow, deliberate, dev-time work: when the deep pass drops a lead, I look at it, and if it’s a real miss I turn it into a recorded exception and then, where I can, a mechanised check. That loop is non-deterministic and human-gated, and it never runs in production. What runs in production is the output: a set of deterministic mechanical checks, plus a shared workflow the whole team pushes work through, behaving the same way every time.

Two things are easy to conflate, so let me pin them. The factory is that learning loop, the rules it’s refining, and the principle core behind them — dev-time, run by me, never in prod. The output is what the factory emits: the checks and the workflow that do run in prod. (The agent that drafts a rule is a tool inside the factory; the checks it helps produce are the output. The word “result” lands in two places — the generated artifact and the factory’s product — but the rule holds: agent generates, factory ships.)

The loop needs something to feed it, and dropped leads aren’t the only source. Here’s a second one I’m testing, and I’ll label it a hypothesis, because that’s what it is: run the work on a model too cheap to do it, and watch where it breaks.

The reasoning is the boundary again. Where a rule has been reduced to a decision procedure — an explicit list of observable conditions, with the verdict recomputed in code from which ones the model ticked — the model isn’t judging, it’s box-ticking, and a small model box-ticks about as well as a big one. Where the judgment still lives in the model’s head, it doesn’t. So the places a cheap model falls down should be, roughly, a map of the judgment I haven’t encoded yet. And the response to a cheap-model miss would then be to go encode that procedure — permanent, deterministic, improves every model — rather than to pay a nine-times multiple forever so a bigger model can re-derive the same call on every run.

Measured so far, that’s suggestive and no more. On thirteen planted-bug fixtures, three runs each, the strong model caught all thirty-nine and the cheap one thirty-eight — and the single miss landed in the one pass with no rules behind it at all. That’s the prediction. It’s also the subtlest bug in the set, and I can’t pull those apart: the cheap model fails where judgment is unencoded and the cheap model fails on hard things explain the result equally well. Nor was the cheap arm simply worse — it produced no off-target findings at all. What it lost was consistency in naming what it found, and the deterministic fix for that is already in flight, which may take a good part of the gap with it.

So I’m not claiming a router. I’m claiming the factory has a second instrument available to it: cheapness as a probe, where a failure is a question about the work rather than a verdict on the model. The economics pay off conditionally — a procedure has to be reused often enough that maintaining it costs less than repeatedly buying the judgment — but where that holds, encoding compounds and buying a bigger model doesn’t.

I used to describe this as “ship the product, not the harness,” as if the harness were something I kept around out of tidiness. That’s wrong, and the correction is the whole point. The distinction that matters isn’t harness-versus-product; it’s factory versus output. You keep the factory. You ship the output.

You keep the factory because the output is perishable. It freezes the moment you ship it, and a trained agent — the thing I’d like and don’t have yet — would freeze the same way, which is why even a shipped model has to sit behind the checks rather than replace them. Frozen things drift: the framework moves, practices change, and last year’s right check now fires on code that’s fine or misses code that isn’t. The output rots. The factory rots too: its rules, even its principles, can go stale. But the factory is the part you’re actively tending, so that’s where the rot gets caught and repaired. The output is frozen and left to time; the factory is frozen and maintained. That’s the point of keeping it: the output rots on its own, and the factory is where the rot gets caught and repaired.

Ship the output. Keep the factory.

Persistence is a feedback edge

There’s a quiet reason the factory’s slowness is a safeguard, not a limitation. The reviewer is pure across runs: f(input) → output, no history term, one run never changes what the next one sees. That statelessness is a big part of why you can trust it — nothing carries between runs to drift on, because it isn’t feeding on its own past. (Statelessness buys no-drift; replay needs determinism on top — a stateless model call still won’t replay.)

The moment you add memory that later runs read back, it becomes f(input, state) → output, state', and that saved state adds a cycle to the graph: an edge from this run’s output into the next run’s input, routed through the store. You didn’t draw the loop; the store closed it. The edge lives in the storage layer, not the diagram, so it’s easy to miss. And it opens a feedback edge you now have to govern — the risks statelessness was holding off: lost reproducibility (you may not be able to replay without rebuilding the state), drift (the system can start feeding on its own prior outputs), and — because there’s now a cycle — something to game. None of these is automatic; they’re what the edge exposes you to. This is the machine-learning-systems hidden feedback loop, and its cousin CACE: changing anything changes everything.

So the design rule falls out of the topology, not out of taste: keep the fast path pure and stateless, and quarantine every write into an explicit, governed outer loop that a human gates. That is exactly what the factory is. The reviewer doesn’t auto-memorise; its learning — a dropped lead becoming a mechanised check — is a deliberate write in the slow loop, anchored by a person deciding the miss is real. Auto-memory on the fast path is the opposite move: a loop that can drift where you can’t easily reproduce or debug it.

Designing the next version of that outer loop showed me the rule has two holes, and both are in the parts that sounded safest.

The first is the safeguard I’d have called airtight: nomination-only — the stored memory may suggest extra work, never suppress a finding. Suggest-don’t-suppress is not sufficient, because a review runs against a finite budget of turns, tokens and time. Anything steering what gets examined first is spending that budget, and a nomination that fills it is a real finding never reached. You can suppress by scheduling. Concurrency has had a name for this for decades — starvation, a process perpetually denied the resources it needs while nobody ever decides to deny it. There’s no skip to audit, no dropped verdict in a log, nothing shaped like a decision. The finding simply never came up. The fix isn’t to trust the nomination less, it’s to deny it the budget: give derived suggestions their own separately reserved, additive allocation, and let them touch neither the baseline’s ordering nor its share. If the memory can reorder the baseline’s work, it isn’t advisory any more. It’s on the critical path, and it belongs under the same governance as everything else there.

The second hole is in the word gated. I’d drawn the human as the thing that breaks the cycle. On dataflow that holds — nothing writes back on its own. But influence still goes round: the store shapes what gets proposed, the proposal shapes what the person sees, the person promotes, and the promotion lands back in the store. The loop closed anyway — one step slower, and through a human who has now read the machine’s suggestion before forming a view. Calling the human an air gap doesn’t make it one. Human-factors research has been saying a version of this since long before agents — Parasuraman and Riley’s misuse: put a person in front of an automated recommendation and over-reliance is a well-documented failure mode, not an exotic one. In this design the gate buys something narrower than a broken loop, and worth stating exactly: the machinery can create candidates but cannot promote them, and every candidate arrives carrying its provenance, to be judged against fixtures it didn’t choose. That’s a real invariant. The human breaks the feedback loop is not one.

The fast path stays acyclic. The only feedback edge lives in a visible outer loop where a person governs the writing.

The human doesn’t leave; the job moves up

The standard objection is “you need a person to check the agent’s code, because agents aren’t good enough yet.” True — but the yet is hiding something.

Maybe the correctness check is temporary; I’d bet agents get better at writing correct code and the line-by-line review shrinks. The argument doesn’t lean on that bet, though, because a second job doesn’t shrink either way: owning the standard — deciding what “correct” and “good” and “bulletproof” mean here. Someone has to choose those ends, and it can’t be the system optimising toward them — an optimiser takes its target as given; it doesn’t get to originate it. A person sets the reference and stays accountable for it, in a way a model can’t be. That judgment comes from outside the machinery, from someone who’s seen enough real failures to know what matters. The human never leaves; the job moves up — from “is this line correct,” which agents can take, to “is this the right standard,” which they can’t. And that standard is where we came in: it’s frozen judgment too, which is why it needs a constitution and not just a longer rule list.

What you actually keep

Strip it down. The agent is a non-deterministic generator; its output is the warranted, shared, deterministic thing you ship — perishable, so you re-mint it as it rots. The durable asset is the factory behind it: the rules, the harness that refines them, and the frozen principle core that governs both. A person runs that factory and owns the standard, because the one thing that can’t be frozen or shipped is the judgment of what “better” means.

The hype is selling you the generator — the loop, the graph, the probabilistic runtime. That part’s real, and it’s the cheap part. Keep the factory that makes the rules, ship the checks it mints, and hold onto the standard. That was never the thing for sale.

How sure I am: the reviewer, the manual learning loop, the mechanised checks, and the five-stage pipeline (scout/gate/deep/decision/write — gate and decision are pure functions, no model) are real and running, genericised here. “Factory versus output,” “non-determinism is a generator,” and “persistence is a feedback edge” are my argument from building the thing, not a measured result. The two holes in that section — suppression by scheduling, and the human gate not breaking the loop of influence — came out of designing a persistent store for the reviewer and hardening the design against cross-model review. They are reasoning about the topology, caught on paper before anything shipped; I have not built the store, so I have not watched either failure happen. Cheapness-as-a-probe is explicitly a hypothesis, and I’ve said so in the text: the run behind it is real (thirteen fixtures × three runs per arm, one model pair, one sitting) but it cannot separate unencoded judgment from harder bug, the two candidate explanations for its single data point, and the stability half is confounded by a naming problem I’m still fixing. The routing it implies is designed, switched off, and unmeasured. Two forecasts are wagers, flagged as such — that most software people depend on has a checkable output, and that line-by-line correctness review shrinks as agents improve — and the argument leans on neither. The trained-agent version of the output is aspirational; today the output is the mechanical checks, and I’ve said so rather than imply the model exists. The prior art (the workflow/agent distinction, Constitutional AI, Lakatos, the CACE/hidden-feedback-loop framing, and DSPy / model-cascades as the pipeline backdrop) is real but cited as analogy or antecedent, not proof. AlphaCodium’s numbers are the exception — they are measured, and they are theirs, on GPT-4 and the CodeContests validation set. What they measure is competitive programming, so they establish that a deterministic flow can beat a single prompt on the same model cheaply; that this carries to a review pipeline is my reading, not their result.

Sources

Anthropic, Building Effective Agents (2024) — the workflow-vs-agent distinction this post goes one layer beneath (“workflows: predefined code paths; agents: LLMs dynamically directing their own processes”). The taxonomy is theirs; the factory/output cut and the eval-drift argument are the additions.
Antonio Gulli, Agentic Design Patterns (github.com/evoiz/Agentic-Design-Patterns) — a 21-pattern catalogue of agent shapes; the catalogue whose lane this post goes a layer beneath, past what shape to what you verify with.
Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, arXiv:2310.03714) — modular stages with typed signatures and per-stage optimisers; cited as the backdrop, not a foil. DSPy tunes model calls and composes ordinary deterministic code around them; the emphasis here is on the stages that need no model at all — a difference of emphasis, not of what the framework permits.
Chen, Zaharia & Zou, FrugalGPT (2023, arXiv:2305.05176) — model cascades that route cheaper models to easier steps; cited as prior art for right-tool-per-stage, which here becomes right-tool including no model.
Ridnik, Kredo & Friedman, Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering (2024, arXiv:2401.08500) — a fixed, ordered pipeline (self_reflection, possible_solutions, generate_ai_tests, initial_code_generation, then iteration against public_tests and ai_tests) that took GPT-4 from 19% to 44% pass@5 on the CodeContests validation set versus direct prompting, at 15–20 model calls per solution — four orders of magnitude fewer than AlphaCode. Cited as the measured receipt that the win can come from the flow rather than the model, and for the cheapness of the deterministic half. Their task is competitive programming, not review; the transfer to a review pipeline is my reading.
Bai et al., Constitutional AI (2022, arXiv:2212.08073) — a small principle set steering a model’s behaviour; cited as the nearest AI analogue for principle-governed behaviour, not as the frozen/mutable architecture (it uses the principles to train).
Parasuraman & Riley, Humans and Automation: Use, Misuse, Disuse, Abuse, Human Factors 39(2) (1997); Parasuraman & Manzey (2010) on automation complacency — over-reliance on an automated aid is a well-documented failure mode, catalogued rather than rare. Neither paper establishes it as the modal response, and an earlier version of this line claimed that; corrected. Cited as the human-factors antecedent for why a human gate doesn’t break a feedback loop; the argument here is from the topology, and this is the literature that already knew.
Resource starvation (concurrency/scheduling) — a process perpetually denied the resources it needs, with no explicit denial anywhere. Standard vocabulary, borrowed here for a review budget; the remedy in this design is a reserved additive allocation rather than the usual aging, because aging doesn’t secure the property under a fixed budget.
Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) — “hidden feedback loops” and CACE (Changing Anything Changes Everything); cited as the antecedent for persistence-is-a-feedback-edge, applied here to agent memory.
Imre Lakatos, The Methodology of Scientific Research Programmes — a protected hard core wrapped in an adjustable protective belt; a structural analogy for the two-tier standard, not proof that principles are permanent (whole programmes get abandoned).

Related: Rules Are the Alpha (the rules as the moat this post’s factory maintains); A Big Rule Set Is an Attention Problem (how the rules compile into the deterministic checks you ship); Fluent in a Dead Idiom (why frozen judgment drifts — the same rot, in your reading eye); Your Merged-PR History Is Survivorship-Biased Training Data (the bias this post turns on the agent-hype writing itself); Skills Are One-Offs (share the workflow that repeats; let one-offs stay throwaway); The Harness Is the Product (the harness tunes each isolated stage — the why to this section’s how); A Flip Is a Missing Rule (the cheap-model probe without the second model — the wobble the quarantined stages still have, read as a pointer at the unencoded judgment).

Fluent in a Dead Idiom

2026-07-24T00:00:00Z

“The greatest obstacle to discovery is not ignorance—it is the illusion of knowledge.” — Daniel J. Boorstin

A newer engineer left a note on my pull request: this array should be readonly. He was right. Readonly-by-default had become the house convention a while back — and I had missed it, in my own code, on a review I’d have sworn I read carefully. When I went looking, I hadn’t been flagging it in anyone else’s code either. The convention was in the codebase. It wasn’t yet in me.

The cause is the same one that makes fast reading fast. Reading code quickly is trained pattern recognition, and the training is time-stamped. What you learned as correct is what your eye reads as correct — and both the rules you adopted late and the rules you adopted early drift out from under you. The late ones you don’t have yet. The early ones you can’t let go of. The second kind is the dangerous one.

BLUF

Fast reading is a trained reflex, and the reflex is time-stamped. You read fluently in the idioms you trained on. That calibration drifts two ways.
Adoption lag: a convention adopted after you trained isn’t in your reflex yet, so you don’t snag on its absence — you miss it in your own code and in review. (The readonly note above: the newer engineer had it, I didn’t.)
Deprecation lag — the sharp one: an idiom you learned as correct stays correct to your eye. When the standard later turns against it, your reflex still reads it as fluent, so no snag fires and you don’t catch it in review.
This is structural, not laziness. The reading reflex catches the wrong shape. A deprecated idiom is the right shape to you — so the guard meant to catch it is the very thing that waves it through. It’s typo-blindness: you can’t proofread your own idioms.
You can’t teach against what you can’t see. If your eye never flags it, you never tell anyone to stop — so it ships and becomes “the way we do it here” by silent non-correction.
The fix is an explicit rule, not more reading. A written rule fires on both the missing-new and the clung-to-old, whoever’s eye is running it — and its why teaches the deprecation your eye can’t even flag. That’s the case Rules Are the Alpha is making.

Reading is a reflex, and reflexes are dated

I’ve argued elsewhere that reading code fast is pattern recognition — you stop spelling out every token and start seeing shapes, the way a fluent reader sees words instead of letters. (Thinking at the Speed of Code is that argument; I won’t re-run it here.) The part I skipped is the cost of the shortcut: a reflex is calibrated to a moment. You trained it on the idioms that were current when you learned to read the language, and it keeps reading in those idioms after they change.

So fluency isn’t a fixed asset. It’s time-stamped. And it goes out of date in two directions.

The easy direction: the rule you don’t have yet

The readonly note is the easy direction, and it still caught me. Readonly-by-default arrived as a convention after my reading reflex had set. So when I read my own array declaration, nothing snagged — the code looked right, because it looked like the code I trained on. The newer engineer’s eye had been trained after the rule landed, so readonly was part of his baseline and its absence jumped out at him. He wasn’t more careful than me. He was calibrated later.

This direction is easy because it’s recoverable. The rule exists; someone has it; the fix is to internalise it, or better, to write it down so nobody has to carry it in their head. The gap is a lag, and lags close.

The hard direction: the idiom you can’t stop seeing as fine

Now the direction that doesn’t close on its own.

Years ago, writing JavaScript, I taught a teammate a little trick for a sort comparator. Two booleans, true ones first: return bSatisfied - aSatisfied. Subtract the booleans, let the language coerce them to 1 and 0, and you get your comparator for free. Compact, clever, and I was a little proud of it.

It leans on a language quirk to save a few characters, and the next reader has to know the quirk to read the line at all — and nothing I had switched on flagged it. With no lint enabled for it, the reviewer’s eye is the only guard. I’d fix it if you pointed at it. But I wouldn’t point at it. If I saw that line in review today, my eye would slide right over it — it reads as fluent to me, because I’m the one who made it fluent. I’ve got a matching habit on record: casting to string with a concat, "" + value instead of String(value). That one a linter can catch — it’s what ESLint’s no-implicit-coercion flags — but only if you’ve turned the rule on, and with it off my eye is the sole guard again. It reads as correct to me, because when I trained, it was.

That’s the deprecation lag, and it has two faces. One is production: I still reach for the old idiom when I write, because it’s the mechanised move — psychologists call it Einstellung, the state of mind that keeps applying a learned solution after a simpler one exists. That face is annoying but survivable; a linter or another reviewer can catch what I produce. The other face is the one that matters here: I can’t catch it in review. And that isn’t about discipline.

Fluency trails change in both directions. It takes time to recognise a new idiom—and time to stop accepting an old one.

Why the guard is blind to exactly this

The reason is structural. The reading reflex works by flagging the wrong shape — code that doesn’t match the idioms you trained on. That’s what a snag is: a mismatch between the text and your pattern. A deprecated idiom, by construction, isn’t a mismatch. It’s the shape you trained on. So the very mechanism that makes you a fast reviewer — reading for the familiar shape — is the mechanism that waves your own dead idioms through. Fluency can’t audit itself. The thing that would catch the bad idiom is the thing that recognises it as good. You can override it — but only deliberately, by knowing to look for the specific idiom. Left to the reflex, it passes.

You know this feeling from prose: you can’t proofread your own writing. You read what you meant, your brain fills in the familiar shape, and the typo you’d catch in a heartbeat in someone else’s draft is invisible in yours. Reading is automatic and top-down — it serves up meaning and skips the surface. A deprecated idiom is a typo your eye has been trained to read as correctly spelled. Cognitive fluency does the rest.

There’s a worse consequence than the miss itself. If your eye never flags the idiom, you never tell anyone to stop using it. You can’t teach against what you can’t see. So a newer engineer picks it up from the codebase, it slips past everyone who trained when you did — they share the blind spot — and it graduates to house style by silent non-correction. No one defends the idiom. It survives because no reviewer’s reflex objects.

The rule snags where the reflex can’t

This is the sharpest case I know for the explicit, written rule. A rule doesn’t read for shape. It checks for a pattern whether or not that pattern feels fluent to the person running it — which is exactly the check your reflex can’t run on its own idioms. Write down “no implicit coercion — say what you mean” and whoever runs the checklist catches the line my eye slides over — mechanise it as a lint and it fires with no eye at all. Write down “arrays are readonly by default” and the same holds for the gap the newer engineer saw and I didn’t. Another reviewer with a differently-dated eye might catch either one by luck; the rule catches them on purpose. It holds still while fluency drifts, in both directions at once: the rule I don’t have yet and the idiom I can’t let go of, whoever happens to be reading.

And it does the one thing the reflex can’t: it teaches the deprecation. The rule’s why — leans on a coercion quirk, taxes the next reader with knowledge they shouldn’t need — is the explanation I’d never generate on my own, because I can’t see that the thing needs explaining. The rule catches what fluency misses and says out loud what fluency can’t.

That’s the corollary to reading fast. The reflex is what makes you quick, and the reflex is dated the day you finish training it. The rule is how you keep reviewing in a standard that has moved on from the one in your head.

How sure I am: the two receipts are lived — a readonly note I missed and a coercion trick I taught, both real review moments, genericised. The mechanism — that the fast reading reflex is blind to its own idioms — is reasoned from how fluency works plus those receipts, not measured; treat it as argument, not data. “Deprecated” here means a standard or a lint rule turned against the idiom, not that the language removed it. The prior art is real and named, but I’m citing established effects by analogy — I’m not claiming the studies tested code review.

Sources

Thinking at the Speed of Code — reading fast as trained pattern recognition; this post is the corollary (the reflex is time-stamped). The author’s own argument, cited not re-derived.
Abraham Luchins, “Mechanization in problem solving: the effect of Einstellung” (1942) — the mechanised state of mind that keeps applying a learned solution after a simpler one exists. The production-side prior art.
Erik Dane, “Reconsidering the trade-off between expertise and flexibility: a cognitive entrenchment perspective,” Academy of Management Review (2010) — expertise trades flexibility for entrenchment; the expert-specific cousin.
Rolf Reber, Piotr Winkielman & Norbert Schwarz, Effects of Perceptual Fluency on Affective Judgments, Psychological Science 9(1):45–48 (1998) — stimuli processed more fluently are judged more positively. The follow-on — Reber & Schwarz, Effects of Perceptual Fluency on Judgments of Truth, Consciousness and Cognition 8(3):338–342 (1999) — extends it to judged truth, the mechanism behind an idiom that reads as fluent being read as correct. Cited by analogy — the studies are on perception and judgment generally, not code review. (The everyday version: you can’t proofread your own writing because you read what you meant, not what’s on the page.)
ESLint no-implicit-coercion — flags shorthand type coercions such as "" + value (use String(value)). Opt-in, which is the point: with it off, only the reviewer’s eye guards the line. It does not flag the boolean-subtraction sort trick — and I had no lint enabled that would, which is why the eye was the sole guard there.
Rules Are the Alpha — the explicit rule as the durable asset; this post is the fluency-drift case for it.

Related: Thinking at the Speed of Code (the sibling — building the reading reflex; this is its time-stamp and its blind spot); Rules Are the Alpha (the explicit rule that stays put while fluency drifts); Familiarity Isn’t Trust and Confidently Wrong (fluency as a false signal — here turned on your own idioms).

A Big Rule Set Is an Attention Problem

2026-07-24T00:00:00Z

“A wealth of information creates a poverty of attention.” — Herbert A. Simon, Designing Organizations for an Information-Rich World (1971)

Running my code reviewer’s full rule set through the model on a nine-file change took about sixteen minutes and sixteen dollars, and turned up four low-severity nits. Most of it went on scanning for rules rather than judging the code.

An opinionated rule corpus is the asset. But the bigger it gets, the more it threatens to turn back into a liability, and the reason is attention rather than cost.

BLUF

A big corpus can fit the context window and still overflow the model’s attention. Those are two different limits, and the second is the one that quietly bites.
That’s lost in the middle, and it’s documented — models use the middle of a long input least. A rule set inherits it too, so more rules mean quietly worse reviews — and that extension is mine, not Liu’s result.
The fix isn’t a better prompt. It’s spending the attention budget. Move the deterministic noticing into code: a detector flags the candidate sites, and the model’s attention lands on judging them instead of scanning every rule against every line.
Mechanisation is an attention lever, not only a cost one — and mostly it moves the noticing, not the rule. Only about a third of my rules fully leave the model’s context; the rest still need a judge, but no longer a scan.
Structure the rules that stay — target, exceptions, examples, why — and split them across passes so no single context holds the whole corpus.
A live loop shrinks the in-context set over time. Every lead the deep pass drops becomes a recorded exception, mechanised where it can be.

Rules in context, and the wall

A large corpus has an obvious problem: you can’t stuff every rule into the model’s context and expect equal attention to all of them. Language models use the middle of a long input least and the ends most, a measured positional effect. Liu et al., Lost in the Middle (2023), arXiv:2307.03172 — a measured positional effect on what gets used, not a claim about attention. It names where information gets used rather than “attention” directly. But the consequence for a big rule set is real. Pile a hundred prose rules in, and the judgment call you most needed — sitting at position fifty — is the one most likely to get skimmed. You get a review that technically saw every rule and acted on whichever ones landed where the model still looks.

A small check backs the direction, if not the tidy shape. I planted one violation and buried its matching rule in checklists of 5, 20, and 80 rules. Recall of that violation fell as the list grew (≈50% → 44% → 33%) and as the rule sat lower in it (≈56% at the top → 28% at the bottom). A bigger list does bury a rule. The check was noisy — n=6 per cell, one 7B model — and it didn’t reproduce Liu’s specific middle-dip cleanly, so it reads more as “longer and lower both hurt” than as a clean U. The extension held directionally.

The sample size is the lesser doubt. Here’s the bigger one: this ran on a 7B model and the pipeline runs a frontier one. Whether a small model’s attention behaviour transfers upward is unestablished, and nothing in my check speaks to it. I hedged the number because the number had a unit, which is the exact failure another post here is about.

Spend the attention budget

The fix is not a better prompt. It’s to spend the attention budget deliberately.

Some rules can be made deterministic: the pattern matches, the syntax-tree checks, the mechanical detectors. Those get compiled out of context into code that runs without the model at all. A large detector set already runs that way.

Pulling those out isn’t only a cost lever. It’s an attention lever. Every rule you move to code is context the model no longer has to read, so its attention lands only on the rules that genuinely need judgment. Mechanisation is how the routing keeps working as the corpus grows. A live loop pushes it further: every lead the deep reviewer drops becomes a recorded exception, mechanised where it can be, so a known non-issue stops costing model attention next time round.

The broader principle is to disclose the right context at the right time instead of cramming it all in, and people are starting to measure it directly. A controlled study of progressive disclosure for long-context agents found it “buys context, not intelligence”: near-zero gain when the material is small enough to navigate, decisive once it’s too big to read through, with one level of disclosure beating deeper nesting (He et al., 2026). Compiling to a detector is the deterministic end of that same move, which is my extension of their point rather than their finding. The rules a detector fully owns don’t get disclosed to the model at all.

Now the honest shape of it. That nine-file run up top is the “before”, and mechanisation is messier than “move the rules to code”. Of about a hundred and fifty rules, only a third fully leave the model’s context; the rest still need a judge.

What a detector usually removes isn’t the rule. It’s the noticing. It flags every candidate site deterministically, and the model’s attention goes to judging those candidates instead of scanning every rule against every line. Even when the rule text stays, the expensive part — the scan — doesn’t. That’s the attention lever, said honestly: mechanise the noticing, not necessarily the rule.

A third route: delete the rule

There’s a route that isn’t mechanisation at all. Anthropic reported removing “over 80%” of Claude Code’s system prompt for its more advanced models “with no measurable loss on our coding evaluations”, and the diagnosis they give is worth borrowing: they “were overconstraining Claude Code” across the system prompt, the CLAUDE.md files, and the skills.

Their transcripts showed directives clashing inside a single request — “leave documentation as appropriate” sitting next to a blunt “DO NOT add comments.” That’s a second way a corpus eats attention, and it isn’t positional. It’s interference: the model spending budget working out which instruction wins before it can act on either. Their multi-clause comment prohibition became one judgement line — “Write code that reads like the surrounding code: match its comment density, naming, and idiom.”

Alex Ewerlöf argues the general version from the other side, and it’s the part that should worry you: a natural-language rule set has no consistency checker. Linters and compilers catch a fraction of the ways one instruction can contradict another, and the only reliable way to surface a clash is to run the agent and watch it fail, which costs far more than a static check. So the corpus can be internally inconsistent and nothing in your toolchain will tell you.

Two things to keep straight. The scope depends on the model — the deletion was for the more advanced ones, so it’s a claim about what a model can now carry on its own, not a claim the rules were always dead weight. And the check behind it was an eval, which climbs only as high as what you managed to specify; “no measurable loss” is evidence, not proof, that nothing left with the deleted text.

The route is still real, and it’s cheaper than mechanising. Before you spend engineering on a detector, check whether the rule is one the model no longer needs told.

Structure what stays

The rules that stay in front of the model are structured to earn their attention. Each carries a target, its exceptions — the field most rule products I’ve looked at skip — examples that feed the deep verification pass, and a why. They’re split across passes rather than dumped in one blob, so no single context holds the whole corpus at once.

That why does double duty. It’s unpacked into the review comment to teach the author, so the rule doesn’t just catch the mistake, it explains it.

A long rule list dilutes attention, especially through its middle. A detector routes candidate sites into a much smaller judgment surface; the rule may remain, but the scan leaves the model.

The point

A rule corpus scales only if you spend attention deliberately. Mechanise the deterministic noticing so the model’s scarce attention lands on judgment, even where the rule text itself has to stay. The corpus is the asset; this is how you wield a large one without drowning the model that has to read it.

How sure I am: The detector set and the exceptions-pass loop are lived — both running now, genericised. Lost in the middle is prior art I’m citing, not claiming. The extension to a rule set — more rules, worse recall of the buried one — now has a small directional check (Qwen2.5-7B, n=6 per cell): recall fell as the list grew and as the target rule sat lower, though it was noisy and didn’t reproduce the specific middle-dip cleanly. That mechanising rules out improves the judgment, rather than only cutting cost, is reasoned from the attention argument and not something I’ve isolated and measured. The receipt is real — a nine-file run, roughly sixteen minutes and sixteen dollars for four nits, genericised — with two honest bounds: only about a third of the rules fully leave the model’s context, and whether mechanised noticing lifts recall is unproven, since a detector protects the verdict rather than recall, so a pre-filter miss is silent. The Anthropic deletion figure is theirs, on their harness, checked against their own coding evals; I cite it as a practitioner receipt that the route exists, not as a measurement that transfers to a rule corpus. No such ablation has been run against mine, so “delete before you mechanise” is a prompt to check rather than a result.

Sources

Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni & Liang, Lost in the Middle: How Language Models Use Long Contexts (2023), arXiv:2307.03172 — models use information in the middle of a long input least: a measured positional effect on what gets retrieved and used. The extension to a rule set (“more rules → skimmed → worse reviews”) is mine, not Liu’s measurement.
He, Zhao, Wang & Chen, Is Progressive Disclosure All You Need for Long-Context Agents? (2026, arXiv:2607.17598) — a controlled study finding progressive disclosure “buys context, not intelligence”: near-zero gain when a strong harness can already navigate, decisive once the corpus is too large, and one level of disclosure beating deeper nesting. Corroborates the disclose-don’t-cram principle; mechanising to a detector is its deterministic end. Read pre-publish and cited from the abstract’s qualitative findings, not its numbers.
Thariq Shihipar, The New Rules of Context Engineering for Claude 5 Generation Models (claude.com blog, 24 July 2026) — Anthropic removed “over 80%” of Claude Code’s system prompt for its more advanced models “with no measurable loss on our coding evaluations”, having found they “were overconstraining Claude Code” across the system prompt, CLAUDE.md files, and skills. Transcripts showed directives clashing inside a single request, and a multi-clause comment prohibition was replaced with a single judgement line. Cited for the deletion route and the interference mechanism, both theirs, with their capability scope (more advanced models) intact. Reading it as a third disposal route beside mechanising and structuring is mine.
Alex Ewerlöf, Coding Is Not Solved (blog.alexewerlof.com, 26 September 2026) — explicitly opinion-led, cited for one argument: a natural-language instruction set has no consistency checker, linters and compilers catch only a fraction of the contradictions, and the dependable way to surface one is to have the agent attempt the build, which costs far more than a static check. Argued rather than measured; cited as a second, independent statement of the interference mechanism, so the claim doesn’t rest on one vendor’s transcripts.

Related: Noticing Is Cheap, Judgment Is Scarce — the same move, costed.

Where the Reps Come From

2026-07-24T00:00:00Z

“For the things we have to learn before we can do them, we learn by doing them.” — Aristotle

BLUF: Bottom line up front

The reading series is a retention frame. Those posts on keeping the reading — anchored by Give Away the Writing, Keep the Reading — say keep the reading, guard the residue, hold the skill a year out. They speak to people who already trained the old way — write the first draft, hit the wall, read their own broken code — and tell them how not to lose the eye. This post is about the cohort they never address: the engineer who arrives after AI already writes the first draft.
Judgment used to form as a byproduct of the job. Writing code you didn’t yet understand, owning the bug you wrote — that was the bottom rung, and it built the eye whether or not you meant it to. AI removes exactly that rung.
So for this cohort, judgment has to be built deliberately or it may not form at all. The default path that built it for free — as a byproduct of writing the first draft — is gone. This isn’t atrophy — atrophy is losing something you had; this is the harder case of never having acquired it.
The mechanisms already exist; they were written as maintenance. Write the hard parts by hand, fade the help and practise protected, run the reps with feedback. For the trained eye those are optional upkeep. For the cold-start engineer they’re the deliberate substitute for a default that no longer runs on its own.
The productivity numbers assume the reading still happens. They price the machine and take the competent reader as given. The job produced that reader for free, and that subsidy is ending.

The numbers assume a reader

You’ve seen the pitch. The sober version is Anthropic’s own estimate — Claude, reading a hundred thousand real conversations, estimated that AI cut task time by around 80% (a model’s estimate across all kinds of work, not an observed measurement and not specific to software; Anthropic itself notes it may overstate the gains). The loud version is the viral “an AI engineering team replaced a whole department” claim — the kind that trends without a citable result behind it. Take the sober number and leave the loud one where you found it.

Even the sober number routes through one quiet assumption: someone at the end of the loop can tell whether the output is right. Give Away the Writing named that seat — the reading, the one act in the loop that owns intent. The numbers price the machine and take the reader as given (my reading of the gap, not a claim the report makes). Fine, today: the reader learned to read the old way. The question the numbers never ask is who’s sitting in that seat in ten years, if the job has stopped producing anyone who can.

The series was written for people who already trained

Say the thing the reading series doesn’t. It is a retention story from front to back. The Productivity Mirage’s gate — do I want to still be able to do this without the tool a year from now? — presupposes you can already do the thing. Thinking at the Speed of Code is about a reflex you built and want to keep sharp: “the difference is the reps,” reps you already ran. Give Away the Writing tells you to hand-write the hard parts “while the instinct is forming” — forming, on the old default, where writing the first draft is still your job.

All three address someone who has the eye or is in the middle of getting one the ordinary way. None of them address the person who shows up when writing the first draft is no longer part of the work. Tell that person to “keep the reading” and there’s nothing yet to keep.

The rung AI saws off first

A designer on my team started opening PRs. Real front-end work — the kind of UI they’d normally spec and hand to an engineer, now produced with an AI and shipped as code. The changes worked. They looked right. And my reviews filled up with things the diff gave them no way to see: a create call that didn’t wait for the thing it was creating, so the record could be read before it existed; a component whose “controlled” mode hinged on one prop being passed, with nothing in the types to enforce it, so it could silently never close; a store named after a screen it had nothing to do with — I could tell it was AI-generated, they couldn’t. Working UI on top; underneath it, engineering they had no reps to catch. They could produce the code. They couldn’t read it.

The designer is just the early, visible version. Before AI they’d never have shipped that PR at all — the tool is what put unreadable code under a name with no reps behind it. The same gap is now opening one level up, for the engineer who starts out when the first draft is already the AI’s.

Here’s what that person is missing, and it was never motivation. The eye never got taught on purpose. It fell out of the job. You wrote something that didn’t fit and the codebase told you. You shipped a bug and then owned it at 2am. Nobody scheduled those reps — they were the byproduct of being the one who wrote the first draft. The judgment formed on the side, whether or not you were trying to build it.

That byproduct is, my bet, exactly what the tool removes, because the first draft is what it’s best at. The senior kept the judgment because they built it before the tool arrived. The cohort behind them is being handed the top of the ladder with the bottom rung gone — asked to review what they were never made to write.

This is not the “juniors will atrophy under AI” alarm, which is everywhere and mostly right and not the point. Atrophy is decay of a skill you once had; the cold-start problem sits upstream of that. The default path that used to build the skill for free has been automated, and nothing has stepped in to replace it as the default. Other paths still build judgment — debugging the code the tool generated, code review, mentorship, ops — but none of them runs automatically the way writing the first draft did. The skill now has to be gone after on purpose, or it may not arrive.

Writing used to close this loop by default. Judgment was the residue carried into the next draft.

You can’t skip straight to reading

The tempting reply: fine, they’ll learn to read instead of write. But Thinking at the Speed of Code already showed why that doesn’t come free: the fast read is trained pattern-recognition, and it needs the two conditions that piece lays out — valid cues and practice with feedback. The consequence here is direct — read without ever having built, and there’s no model for a wrong shape to snag on. And won’t reviewing the AI’s output build the reps back? Only if you can already read it. Review compounds a skill you have; it can’t bootstrap one you don’t — the designer reviewed their own PR and shipped the bug anyway. Feedback only teaches once you can tell whether the feedback is right.

Give Away the Writing compressed it to a line: write to build the judgment, read to keep it. This cohort is being asked to keep what it never built. The reading is the durable skill now. You reach it by running the reps, not by skipping past them.

Manufacture it, or don’t have it

So the reps have to be put back on purpose. I won’t re-derive the mechanisms; they’re already written down, and this post’s only job is to point them at a reader the originals didn’t have in mind.

Hand-write the hard parts — Give Away the Writing. For the trained eye that’s a few passages a week. For this cohort it’s the parts they’d otherwise build no judgment on at all.
Fade the help, keep a protected practice — The Productivity Mirage’s protected-practice loop and its residue test. Run the same machinery; you’re just not testing whether the skill survived, you’re building it the first time.
Run the reps against feedback — Speed of Code, the loop closed: a test goes red, a bug surfaces, a reviewer confirms the catch.

The gate inverts. The mirage’s retention gate asks do I want to still be able to do this in a year? — a question you can only ask about a skill you already have. The acquisition gate asks a harder one: can I do this at all yet, is it a thing I need to be able to do, and — since the default won’t build it for me — how do I run the reps by hand? Same machinery, opposite starting point. Retention protects a callus. Acquisition has to raise one, deliberately, on a job that no longer raises it for you.

And it’s already visible in the trade press. InfoWorld’s “Seven sins of the modern software developer” catalogues the pattern — foundational theory handed to the tool, documentation gone unread, debugging skipped — as a list of vices; underneath, it’s the byproduct-formation loop being skipped in real time, unnoticed because the output still ships. A separate InfoWorld piece names the consequence outright: teams may end up without the skills to validate and debug AI-written code, because the skill formation was inhibited by using AI in the first place.

Once the default first draft disappears, acquisition has to become an explicit loop: protected writing, fading help, grounded feedback, and independent reading.

The bill nobody’s funding

Back to the numbers. The gains are real — take Anthropic’s own sober estimate as the anchor, not the loud version. But a gain that routes through a competent reader is borrowing against a stock of competent readers the industry never had to fund. The job minted them as a side effect. Now the side effect is the first thing automated.

Read the productivity number and staff accordingly, and you’re spending down a reserve the default job no longer refills. The bill doesn’t come due this quarter. It comes due the year the last engineer who learned to read the old way retires, and the seat that owns intent has nobody in it who ever built the eye. Manufacture the reps, or accept that they may not show up on their own — the byproduct that used to supply them is the thing the tool replaced.

How sure I am: Low-to-medium, and split. The retention side I’ve lived — I built the eye by writing and now read far more than I write, which is the sibling posts’ territory. The acquisition claim is reasoned, not lived: I didn’t learn under a tool that writes the first draft, so I’m arguing from the mechanism, not from having come up that way. Formation-under-AI is unmeasured; the “byproduct removed, so manufacture or absence” claim and the inverted gate are a wager, flagged as one. If it holds, it holds because the reps have always had to come from somewhere — not because anyone has run the experiment.

Sources

Give Away the Writing, Keep the Reading — formation home: “write to build the judgment; read to keep it.” This post extends that beat to the cohort with no default writing phase.
Thinking at the Speed of Code — reps home: the fast read is trained pattern-recognition, “the difference is the reps,” and Kahneman & Klein’s two conditions (valid cues + feedback).
The Productivity Mirage — the retention gate and the protected-practice / residue-test loop this post borrows and inverts from retention to acquisition.
“Seven sins of the modern software developer,” InfoWorld — the craft-erosion pattern in the trade press (theory handed to the tool, docs unread, debugging skipped). Paraphrased, not quoted verbatim.
“AI use may speed code generation, but developers’ skills suffer,” InfoWorld — names the consequence outright: teams may lack the skills to validate and debug AI-written code because skill formation was inhibited by using AI in the first place. The thesis, in trade-press words.
Anthropic, “Estimating AI productivity gains” (anthropic.com/research/estimating-productivity-gains) — across ~100k real conversations, Claude estimates AI cuts task-completion time by ~80%; the sober anchor. Anthropic itself notes the estimate may overstate gains.
The “an AI engineering team replaced a department” genre — used as an over-claim foil, not as evidence, and deliberately uncited: it trends as viral posts, not citable results. The specific dollar figures that circulate don’t trace to a rigorous source, so the post cites none.