Intermittent Competence Looks Like Coverage

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman

A broad sandstone band sweeps across thirteen slate stones, implying every one was covered, while clay marks appear on only five of them and cluster at one end.

There are two ways an automated check can fail you. One is loud: it doesn’t run, you look at the output, and the zero is obvious. The other is a check that genuinely works, just not every time.

The second one is worse, and I’d rank it above every other failure I hit building an AI code reviewer. Not because it misses more. Because of how it looks.

BLUF

A check that fires a quarter of the time doesn’t look like a gap. It looks like a feature. You see it work once, conclude it’s covered, and stop asking.
I found mine by accident, while trying to establish whether I needed to build a check at all.
It clustered. Three for three on the dramatic cases, nothing at all on eight of thirteen. So the average hides the shape, and the shape is what matters.
The fix isn’t a better prompt. It’s taking noticing off the model. A deterministic check fires every time or never, and either way you find out which.
The reason to prefer the deterministic version isn’t cost. It’s that it doesn’t have moods.

How I found it

My reviewer reads a pull request and comments on what it finds. It had no check for behaviour a diff removes — a deleted call, a dropped guard, a catch block that quietly stopped reporting. Before writing that check, I wanted to know whether the model already caught removals on its own.

So I inverted my planted-bug fixtures. Swap base and head, and each diff becomes a removal of the behaviour the bug had added. Thirteen fixtures, three runs each.

The same two commits in both panels. Read in the written order, the diff adds the planted bug and a check for new behaviour sees it. Read in the reverse order, the same diff removes the behaviour, and nothing was looking for that.

The model flagged the removal in roughly a quarter of runs: ten across thirty-nine. And they clustered hard. Three for three on a deleted await store.flush(). Three for three on a removed error-swallow. Nothing, in any run, on eight of the thirteen.

The model notices when the deletion is dramatic.

Two honest limits on that. Not every removal deserves a flag, and I haven’t adjudicated which of the thirteen did — so treat that quarter of runs as a shape, not a rate. And thirteen fixtures at three runs is small. The clustering shows the misses aren’t random, and that’s the part that matters here.

Why catching a quarter of removals is worse than catching none

Zero would have been fine. I’d have seen the gap and built the check — that’s an afternoon’s work and I’d already decided to do it.

Catching a quarter of removals is worse than catching none because of what it does to your belief. You run the reviewer, watch it catch a dropped flush call, and conclude it handles removals. It demonstrably does handle removals; you just watched it. So you stop asking, and you never find out it’s silent on the eight cases that didn’t happen to be dramatic.

A silent miss is silent either way: a check that didn’t run, or a model that only sometimes looks. From the outside those are identical: no output. The difference is that the first one you’ll eventually notice, because zero findings looks wrong. Intermittent competence is a coin flip you’ve mistaken for a check.

And you can’t audit your way out of it by looking at what the reviewer found. Everything it found is evidence it works. The misses are the evidence, and they don’t show up anywhere.

The fix, and the reason for it

A deterministic base-to-head delta notices every removal, every run, and hands each one over as a candidate. The model goes back to the job it’s actually good at — deciding whether this removal matters.

I’d moved noticing off the model once before, to stop paying frontier rates for pattern-matching. That was a cost argument and it holds, but it turns out to be the lesser one. The scanner’s real advantage over the model isn’t that it’s free. It’s that it doesn’t have moods.

A parser that misses a case misses it every time, which means a fixture catches it once and forever. A model that misses a case three times in four will pass your fixture on the fourth run. A deterministic check doesn’t win here by being correct — a detector with a bug is wrong every time. It wins because consistency is auditable, and an intermittent check is not.

That generalises past code review. Any time you’re relying on a model to notice something instead of judging it, the question isn’t whether it can. It’s whether it does it reliably enough that your test of it means anything.

Which is why I’d take the parser even at the same price. The scanner’s real advantage over the model isn’t that it’s free. It’s that it doesn’t have moods.

How sure I am: The inverted-fixture run is mine and real — thirteen planted-bug fixtures, base and head swapped, three runs each, ten removal alerts across thirty-nine runs, clustered on the dramatic deletions and absent on eight of the thirteen. The corpus is unadjudicated: I haven’t ruled on which removals warranted a comment, so this establishes the clustering rather than a recall rate. The claim that a deterministic delta would catch all of them is reasoned from how a parser works, not measured — I haven’t built it and run the comparison. The generalisation in the last paragraph is argument, not evidence.

Related: Noticing Is Cheap, Judgment Is Scarce — where this came from, and what it cost to move noticing off the model. A Big Rule Set Is an Attention Problem — the same move, argued from attention, not reliability.