A Flip Is a Missing Rule

When the reviewer answers twice, stop tuning the model and go finish the rule.

“In the face of ambiguity, refuse the temptation to guess.” — Tim Peters, PEP 20, The Zen of Python (2004)

A colleague sent me the cleanest bug report I’ve had all year. He’d taken an open change, squashed it, and opened it again — same content, byte for byte, as far as the reviewer could tell. The automated reviewer left a comment on the second one that it hadn’t left on the first. Two runs, same input, two answers. He offered it as a determinism problem, which is what it looks like.

It isn’t. That comment was the reviewer telling me, in the only language it has, that one of my rules doesn’t say what to do at the boundary — so it had been guessing, and this time it guessed the other way.

BLUF: Bottom line up front

• A flip is usually not model noise — it’s where your specification ran out and the noise got purchase. The title is the compression; this is the claim. You can’t turn sampling off, and that’s the point rather than the objection: a rule that determines its answer gives sampling nothing to flip between, so the places a reviewer does flip are disproportionately the places its rules underdetermine. The claim is conditional on one control I didn’t run: you have to pin the effective input, not just the diff. A flip doesn’t prove a rule is missing. It marks where one would bite, which is weaker and still worth more than anything else you get for free.
• Sort the flips before you spend, because they don’t all want the same fix. Two aren’t the reviewer’s doing and you clear those first: a flip in the line number is your own ruler, and a flip that came from a different effective input — shifted context, different retrieval, a model version nobody chose — is your pipeline, not your specification. What survives splits two ways: a flip in noticing the trigger wants a detector, and a flip in the verdict on a trigger found both times wants a decision procedure. The split isn’t clean — a detector whose scope is itself model-judged can flicker in both places at once — but sorting roughly still beats paying to suppress all of them.
• Sort every determinism lever by one question: does it remove the judgment call, or just average it? Union of N, ensembles, and context-splitting all average it. Caching freezes it. A closed decision procedure is the only one that removes the judgment — though what it mostly does is move the judgment somewhere observable, which is the honest version and the one I’ll argue.
• Caching is the worst of them, because it works. It converts a visible flicker into an invisible wrong answer and calls that stability.
• The measurement is the diagnosis. Two runs over the same diff agreed on four of nine findings — and my first count said three of ten, because part of what I was measuring was my own ruler. Splitting the disagreements by cause told me which rules were underspecified — that’s a work list, not a noise floor.
• I got this wrong first, in a way worth copying. I read the residue as an irreducible floor and moved on. It wasn’t. Two of my rules were asking a question they couldn’t justify asking, and the flicker was the model declining to answer it.
• You will not fix this at the sampler. Temperature and seed weren’t available to me, and where greedy decoding is available, endpoints still aren’t deterministic. Don’t wait on that lever; it isn’t the lever.

Two runs, three of ten

I ran the same rules pass over the same real diff twice and diffed the findings. They shared 3 of 10.

Hold that number loosely for two sections — I counted it wrong, and the correction is coming. But the shape survives the correction, and the shape is the point: a reviewer that agrees with itself on a minority of its own findings is not a check, it’s a draw from a distribution.

I had already measured the same shape from the other side and published it as good news: running decorrelated passes and taking the union caught about 2.5x the findings of a single pass, with round-to-round overlap of 25–35%. Two measurements, taken for opposite reasons, finding the same large variance. Not the same number — that one shares findings over each round’s own size, this one shares them over the union of the pair, and I don’t have the per-run counts to put them on one basis. So read them as two sightings of one phenomenon, not as a matched pair.

So the variance was never in doubt. What was in doubt was what it meant. In the coverage frame, low overlap is upside — each draw reaches findings the last one missed, so you draw again. In the determinism frame, the same wobble is a defect. One phenomenon, two readings, and the readings recommend opposite actions: one says run it more, the other says make it stop.

Both readings are right, about different things, and conflating them is how you end up buying the wrong fix. Union of N is a real, measured, published lever — for coverage. It does not make the reviewer decide. Ten distinct findings across the union of three runs, where the best single run gave four, is more sight — not more agreement.

Four things a flip can be, and two of them aren’t the reviewer

The 3-of-10 was a single number covering unrelated causes. Splitting it was the whole game.

Noticing-flicker. Rules with no detector behind them, where a scout has to spot the trigger by reading. Some runs it sees the borderline case, some runs it doesn’t. This is a sight problem, and I’ve already argued the answer: mechanise the noticing, and the scanner doesn’t have moods.

Verdict-flicker. Rules that do have a deterministic detector — the pattern is found, reliably, every run — and then the judgment on it flips. Detection is deterministic; the judgment isn’t. This is the interesting class, because there’s nothing left to mechanise. The trigger is already code. The wobble is downstream of it, in the part where something decides whether this particular instance actually matters.

Anchor-jitter. The same finding, reported one line apart between runs. This one wasn’t a defect at all. My comparison keyed findings on exact line, so a finding that moved by one line counted as one disappearance plus one appearance — two diffs for zero changes. Within any single run there was exactly one such finding. I had been measuring my own ruler and calling it instability.

Strictly, anchor-jitter isn’t a kind of flicker at all, sitting beside the two above. It’s a failure of the comparison’s identity function — a bug in how I decided two findings were the same finding. It belongs on this list because it arrives looking exactly like the others, and because it’s the cheapest one to find.

Effective-input drift. Here is the one I did not measure, and it is the load-bearing gap in this post. “Same input” and “same effective input” are different claims. I sent the same diff; I did not verify that the model got the same prompt. Between two runs, context assembly can shift, retrieval can return different neighbours, tool output can differ, the base repo can move, requests can route to a different model version or a different endpoint configuration, and a cache can be warm one time and cold the next. Any of those produces two answers from what you called one input, and none of them is a missing rule. So the honest form of this post’s claim is conditional: once you’ve pinned the effective input, a surviving flip is where your specification ran out. I pinned the diff and I did not pin the rest, which means my own 4-of-9 is an upper bound on specification flicker and not a measurement of it. If you run this instrument, log the assembled prompt and diff that first. It’s the cheapest control here and I skipped it.

Two of these you clear before spending anything, and they’re the two that aren’t the reviewer’s doing: my ruler and my pipeline. Two of my ten “disagreements” were an artifact of how I was counting, and any effort spent stabilising them would have been spent on nothing.

Which is the correction I promised: collapse that pair and there were nine distinct findings, not ten, and four were shared, not three. Four of nine — about 44%. I’ll use that number from here. Note which direction it moved: the reviewer was more self-consistent than my first count said, and the headline number got less alarming. It’s still a minority. Two runs, the same diff, and they agree on fewer than half their own findings — the case doesn’t need the inflated version, which is the only reason I trust the deflated one. All four stable findings were detector-backed — which points the same direction from the other end.

You cannot delete your way out of the judgment

The obvious escape from verdict-flicker is to remove the verdict. The detector already fires deterministically; post the detector hit directly and skip the model. No judgment, no flip.

I audited every detector-backed rule in that tier for exactly this. Not one was safe to promote. I can’t tell you how many rules that was — the count sizes a catalog that isn’t mine to publish — so read “not one” as exhaustive over the tier, not as a volume claim. A small tier would make it a weak result, and you have no way to check which it is. That’s a real limit on the claim, not a formality. Every single one had a genuine semantic exception the judgment step was catching: a guard that’s present but sitting in a comment, a parse that provably can’t throw on this input, a pattern that’s a problem in general and fine right here, a rewrite that would quietly change coercion behaviour. The tier wasn’t badly built — it was well-calibrated, and the judgment step was earning its place on every rule in it. Promoting any of them would have traded a flicker for a false positive with nothing behind it.

So the judgment stays. Which means the wobble is structural unless the rule changes, and that’s what forces the real question.

The levers sort by one question

Does the lever remove the judgment call, or just average it?

Lever	What it does to the judgment call	What you actually get
Union of N runs	Nothing. The model decides, N times.	Coverage — measured at ~2.5x findings. Not agreement.
Ensemble of models	Nothing. Narrows the shared blind spot.	A smaller blind spot. Not a decision.
Split the context for attention	Nothing. More attention on the same undefined call.	Better noticing.
Cache the verdict	Freezes it — whichever way it fell.	Consistency, and your first mistake made permanent.
Split noticing from judging	Nothing — but it tells you which flicker you have.	A diagnosis.
Closed decision procedure, with a default	Moves it somewhere observable, and removes it outright wherever a condition is detector-backed. The rule decides; the model only reports conditions.	An answer, and a shorter list of things that can still flip.

Five of those six leave the boundary undefined. That’s not a criticism of them — four are things I use and one I’ve published a measurement in favour of. It’s a statement about what they’re for. If your complaint is “the reviewer misses things,” the top rows are your levers. If your complaint is “the reviewer answers twice,” none of them touch it.

Caching deserves the specific warning, because it’s the one that looks like it worked. Cache the verdict and the flip disappears immediately and completely. What you have bought is consistency, not truth: the first answer becomes the permanent answer, and if the first answer was wrong you have made a wrong answer stable and taken away the only signal that would have told you. A flicker is a defect that’s still visible. A cached flicker is a defect with the alarm cut.

The human isn’t consistent, they’re n=1

Worth meeting head-on, because it’s the strongest version of the opposite reading. Alex Ewerlöf, arguing that coding is not a solved problem, puts the flip at the centre of the case: “a human can be wrong consistently, but a model can be wrong about something it was right about.” He reads it as intrinsic — a gap at least one breakthrough away. If that’s right, this post is mis-aimed: the flip is a property of the tool, and the thing to do about it is wait.

The comparison is mispaired, and the mispairing is the first row of the table above. Ewerlöf sets one model over N runs against one human over N sittings. A run is not a sitting. A sitting carries everything the last one left behind — the position you took, and the fact that you remember taking it. A run carries none of that. N runs of a model are closer to N reviewers than to one reviewer asked N times.

Not N strangers, though. Every run shares the same training, so they’re correlated draws — closer to N reviewers out of one tightly-trained cohort, who disagree about the same handful of cases rather than scattering. That’s the same correlated-errors problem that blocks a second pass from measuring the first one’s recall, arriving from the other direction, and it predicts the shape the flickers actually had: a small specific set, not noise spread evenly across the findings.

Read that way, consistency stops being a property of humans. It’s a property of having asked one of them. Two reviewers disagree about an unspecified boundary as readily as two runs do; you just never run that experiment, because you assign the review to one person and ship what they said. The specification is exactly as missing. You sampled it once.

Which is why the stickiness is a cache and not a virtue. A reviewer’s prior is sticky because they remember deciding and would be embarrassed to reverse it. That damps the flicker without closing the boundary — the cache row’s trade, made in a person. The model flips more because its priors are weaker, not because it’s less trustworthy about the parts that were actually specified.

It also explains a row I measured rather than reasoned. Union of N buys coverage and not agreement because union of N is a panel, and panels have always worked like that.

What survives untouched is the accountability half, and it’s the stronger half. A model can’t be on call, can’t carry a consequence, can’t be punished. That asymmetry is real and no determinism work touches it. But it’s a claim about who answers for a wrong verdict, not about whether the verdict is right, and borrowing it to carry the reliability claim is what makes the reliability claim look like a difference of kind. It’s a difference of degree. A difference of kind you wait out. A difference of degree you work on.

I haven’t measured human flip rates against model ones, and the gap may be real and large. What I reject is the inference — that a flip is a species marker, and the answer is to wait for a better model.

I called it a floor. It wasn’t a floor.

Here’s the part I’d want someone to tell me.

I mined three flickering findings as decision-boundary cases and wrote a worked example into each rule, then ran three fresh passes. One case — a refute example, on something the rule was flagging that plainly wasn’t a violation — went from appearing in 1 of 2 runs to 0 of 3. It stopped showing up — which at three runs is the most I can say, and “gone” is not available to me. A precision win and a determinism win in the same edit, on one finding. The other two were confirm examples on genuinely borderline cases, and they barely moved: 1 of 2 → 2 of 3, and 1 of 2 → 1 of 3.

I wrote that up as an irreducible floor. Semantic rules wobble; examples damp the clear cases; the borderline residue is the cost of having judgment in the loop at all. It’s a tidy conclusion and it has the shape of hard-won engineering wisdom, which should have been the tell.

A second review pass, from a different model family, told me the framing was wrong: a flip means the rule lacks a decision procedure. I pushed back before I checked, then checked. It was right, and worse than right — it predicted the specific failure. I went back to my two stuck cases:

• One was flagging a derived collection that gets rebuilt from its source every time. Nothing mutates it independently, so nothing can drift out of sync. It’s display scaffolding, not the defect the rule exists to catch.
• One was flagging a purely local view-state transition inside a state updater. That’s the updater doing its job, not domain logic trapped in the wrong layer.

Neither was a violation. I had written confirm examples asserting a verdict the rule could not justify. The examples weren’t failing to stabilise the model — they were failing to be true. The model wasn’t wobbling because judgment is inherently unstable. It was wobbling because I’d told it to answer yes to a question where the honest answer was no, and it could see the case didn’t fit.

That is the whole argument in one receipt, and I only have it because I was wrong out loud first. An irreducible floor is unfalsifiable and comfortable — it explains any residue you have, requires nothing of you, and would have stopped the work exactly where the work got useful.

A demo treats one instance; a procedure defines the policy

The fix wasn’t a better example. Once you know the rule was asking an unjustified question, the repair is to make the rule answer it — a decision procedure, which I’ve described elsewhere as “an explicit list of observable conditions, with the verdict recomputed in code from which ones the model ticked.” Closed, this time: the conditions under which this confirms, the conditions under which it refutes, and what to do when the evidence isn’t visible. That last clause is the one everybody skips, and it’s the one that ends the flicker, because “the evidence isn’t in front of me” is exactly the state that was being resolved by a guess. Give it a default — refute, for a reviewer, because an unsupported comment costs more than a miss you’ll catch another way — and the guess at the verdict is gone.

Which is where the strongest objection to this whole post lives, so let me put it in its harshest form: you didn’t remove the judgment, you relocated it. The verdict is recomputed in code, fine — but something still has to decide whether condition C holds, and for most of my conditions that something is the model reading the diff. So the flip doesn’t die, it moves one level down, from “does this matter?” to “is there an unlogged catch here?” A procedure with model-evaluated conditions is not a deterministic procedure.

That’s correct. Two things make the relocation worth doing anyway. The conditions are observably true or false against the code, which the verdict never was — “is there a guard on this path” has an answer you can point at, where “does this instance matter” was asking the model to hold the rule’s purpose in its head and rule on a case the rule never described. And some conditions are detector-backed, which takes them out of the model’s hands entirely; each one you can mechanise is a piece of the procedure that stops being able to flip at all. So the mechanism is a ratchet, not a switch: you move the guess to a place where it’s cheaper, checkable, and sometimes removable.

What I can’t tell you is how much flicker that actually killed. Three post-fix runs on three findings cannot separate eliminated from reduced. So the claim is that the class got re-specified, not that the reviewer stopped flipping on it. And if a condition report flips as often as the verdict did, the ratchet is a wash, and what’s left of this post is “decompose before you spend” — still true, still worth doing, and a much smaller claim than the one I’m making.

The difference between the two fixes is scope. An example pins one instance; anything shaped slightly differently is back to a guess. A procedure defines the policy, so it covers the cases you haven’t seen yet. A demo treats one instance; a decision procedure defines the policy. Re-specifying those two rules closed the specification gap for the class, using evidence I’d already paid for — that much I can show. Whether it closed the flicker is the question the paragraph above says I can’t answer at this n.

This is the same instrument as one I’ve argued for from a different angle: run the work on a model too cheap to do it, and where it breaks is a map of the judgment you haven’t encoded. The flip is that instrument without the second model. It runs on your production configuration, on every real diff, for free, and once you’ve ruled out your ruler and your pipeline, it points at the specific clause you never wrote. You just have to stop reading it as weather.

It also does something I’d previously filed as impossible. I’ve argued that a pool of models can’t recover what you never specified — every model verifies what you wrote down, so the part you left out is invisible to all of them at once, by construction. That still holds, and no amount of union or ensemble touches it. But it’s an impossibility about reaching the missing part, not about locating it. A flip with the effective input pinned locates it. You can’t get a model to supply the clause you never wrote; you can get it to show you where the clause should have been, and then you write it yourself. A hard ceiling turns into a work item.

Where this stands

I have not made the reviewer deterministic, and I want to be exact about what I have — which is less than I first wrote here. One flicker class stopped appearing after a truthful refute example, in three runs. One class got re-specified after I admitted the rule was wrong, and I can show the specification changed but not that the flicker did. One class was never a defect and I fixed my measurement instead. Note the shape of that list: one observed case each, three runs deep, so “eliminated” is a word I’ve taken back everywhere it appeared. The residue is real, it’s mostly in rules whose boundaries I haven’t finished writing, and finishing them is a long grind of specification and mechanisation, not one clever fix. Anyone promising you determinism in an afternoon is selling you a cache.

What generalises isn’t the fix. It’s the reading. When an agent gives two answers on an input you have genuinely held still, you have been handed a free, precisely-located pointer to the place your specification runs out — and the entire instinct of the field is to spend money making that pointer harder to see. Run it twice on purpose. Log the assembled prompt, not just the diff, so you know the input really was the same one. Diff the answers. Then sort the disagreements before you spend a day on any of them: throw out the ones that are your own ruler, throw out the ones that are your own pipeline, send the noticing failures to a detector, and what’s left — the verdicts that flipped on a trigger the system found both times — that’s your backlog. It’s shorter than the diff and it’s the only part that’s specification work.

How sure I am: The overlap figure, the decomposition, and the example experiment are metered runs of my own, genericised, and small n — two runs before and three after, on three findings, on one diff. The uncontrolled confound is effective-input drift, described in the body: I held the diff fixed and did not verify that context assembly, retrieval, tool output, base-repo state, model version, endpoint configuration, and cache state were also fixed. That makes 4-of-9 an upper bound on specification flicker, and it means the post’s central claim only holds once you’ve pinned the effective input — which I did not do. Treat that and the ratchet as the two places this could be wrong rather than merely thin. Treat the run figures as the shape of the thing and not as rates; the tier audit is mine too, but its denominator is a different number and a withheld one — I audited the whole tier and I can’t print how many rules that is, so “not one was safe to promote” is unsizeable, not small, and the small-n above doesn’t cover it in either direction. The refute result (1/2 → 0/3) is one finding that stopped appearing across three runs — not an elimination and not a measured effect size, because three runs can’t tell those apart. On the overlap specifically: 4-of-9 is the corrected figure and 3-of-10 was my first count, and I’d rather you saw both than trusted either — the correction came from my own re-read of how I keyed findings, not from an independent check, so the method of it is unaudited even though the arithmetic is plain. The 2.5x union figure and the 25–35% overlap are from my own earlier published work, deep-confirmed but not oracle-verified — and that 25–35% is a share of each round’s own findings while mine is a share of the two runs’ union, so the two are not on a common basis and I don’t have the per-run counts to put them on one. I’m claiming they see the same phenomenon, not the same number. The ratchet claim — that moving the guess from the verdict down to an observable condition is a real gain, because conditions are checkable against the code and some can be mechanised away entirely — is an argument, not a measurement, and it’s the load-bearing one in the post. I have not measured whether condition reports flip less often than verdicts did; if they don’t, the gain is smaller than I’m claiming and the honest residue is the decompose-first advice. Treat that as the post’s biggest open exposure. The claim that a flip with the effective input pinned locates a missing decision procedure is my take, mechanism-backed — the strongest evidence for it is that it corrected me and then correctly predicted why my two examples had failed, which is more than I can usually say for a framing. “Averaging buys consistency, not correctness” is an argument, not a measurement. The unavailability of temperature and seed is specific to the endpoint I was on; the reason greedy decoding wouldn’t have saved me anyway is borrowed and cited below, measured on their setup, not mine. The n=1 section is reasoning, not measurement: I have not measured human reviewer flip rates against model ones, and I can’t cite anyone who has. What the argument establishes is that the usual comparison — one model over N runs against one human over N sittings — doesn’t license the difference-in-kind conclusion drawn from it. It does not establish that the two rates are close, and they may not be.

Sources

• Cross-run finding comparison — my own instrumentation: two runs of the same rules pass over one real diff, 3 of 10 findings shared on the first count and 4 of 9 after correcting the counting artifact, decomposed into noticing-flicker, verdict-flicker, and that artifact — a cross-run effect of keying findings on exact line. Shares are over the union of the two runs. The diff was held fixed; the assembled prompt, retrieval, and model routing were not verified fixed, so this is an upper bound on specification flicker. Author’s own, genericised.
• Tier promotion audit — every detector-backed rule in the tier reviewed for whether its judgment step could be dropped in favour of posting the detector hit. Zero promotable; each had a genuine semantic exception (commented-out guard, provably-safe parse, contextually-fine pattern, coercion-changing rewrite). Author’s own, genericised.
• Worked-example experiment — three flickering findings mined as decision-boundary cases, two runs before and three after: one refute example 1/2 → 0/3 (stopped appearing; three runs can’t distinguish that from reduced); two confirm examples 1/2 → 2/3 and 1/2 → 1/3. Author’s own, small n.
• The correction — a cross-family review pass rejecting my “irreducible floor” framing in favour of “a flip means the rule lacks a decision procedure,” and the subsequent re-reading of both stuck cases as non-violations. Author’s own working notes; kept because the wrong version is load-bearing.
• Thinking Machines, Defeating Nondeterminism in LLM Inference (10 September 2025) — of 1000 completions sampled at temperature zero, “we generate 80 unique completions, with the most common of these occuring 78 times.” Greedy sampling is “theoretically deterministic” and yet “LLM APIs are still not deterministic in practice”; the paper puts the primary cause not on concurrency-plus-floating-point but on the fact that “the load (and thus batch-size) nondeterministically varies,” against a forward pass that “lacks ‘batch invariance’.”
• PEP 20, The Zen of Python, Tim Peters (2004) — “In the face of ambiguity, refuse the temptation to guess.” Written as advice for people writing code; it reads as a specification for how a rule should behave when the evidence isn’t there.
• Alex Ewerlöf, Coding Is Not Solved (blog.alexewerlof.com) — explicitly opinion-led, and cited for one span: “a human can be wrong consistently, but a model can be wrong about something it was right about,” read by the author as intrinsic and at least one breakthrough away. I take the accountability half of his case — a model has no skin in the game and can’t be punished — as correct, and untouched by anything in this post. The reliability half is the part I contest, and the n=1 reading of it is mine. Neither side of that disagreement is a measurement: his piece offers none on it, and neither do I.

---

Related: A Cue Finds More of What It Can Already See — the union-of-N measurement this post reclassifies as a coverage lever rather than a determinism one; Noticing Is Cheap, Judgment Is Scarce — a mechanised detector can’t flicker on noticing, because a scanner doesn’t have moods — which is an argument from construction, not a class I measured going away; Keep the Factory, Ship the Result — the decision procedure as an artifact, and the cheap-model probe this post’s flip-reading is a free alternative to; A Pool of Models Narrows the Overlap — It Doesn’t Erase It — why more models can’t recover what you never specified; A Big Rule Set Is an Attention Problem — mechanise the noticing, not necessarily the rule.