Shrink, Reorganise, Relocate: Why Synthetic Wins Fail on Real PRs

“There ain’t no such thing as a free lunch.” — Robert A. Heinlein

I spent a stretch trying to make an AI code reviewer cheaper without letting it miss more. Cost and recall, two dials pulling against each other. I pulled every lever I could find — shrank the context, split the diff, fused passes, routed to different models, swapped in a stronger reviewer, bolted on a deterministic scout. Most taught me the same thing, and I only saw it once I lined the results up. Every one was doing one of three things to the model’s workload — shrinking it, reorganising it, or relocating it — and only one of the three is free. The dangerous one is the middle: reorganising can pass every synthetic test and still lose real bugs.

BLUF: Bottom line up front

• Three things you can do to a model’s workload: shrink what it reads, reorganise how the work splits, or move work off the model into code. Only the third is free.
• Shrinking the input to save money costs recall. You cannot shrink your way to coverage.
• Reorganising moves cost around; it does not buy recall — and a synthetic test will happily say it did. A merged-sharding scheme passed a planted-bug fixture, then dropped real findings on real PRs.
• Moving the work off the model is the free win. A deterministic checklist scout — no per-token cost at all — lifted recall by 33 to 67 percent.
• A stronger deep reviewer is a real lever too, but you rent it. The more capable model let me cut the rulebook from 17 rules to 7 — capability paid by the token, not a free lunch.
• Try it: before you tune a prompt, find the slice a parser could do and move it off the model. Decide with real inputs over at least three runs, never one synthetic fixture.

One law behind every lever

Line the levers up and they sort into three buckets by where they touch the model’s job. You can shrink the input — trim context, drop rules, cap the diff. You can reorganise the work — shard a big diff, fuse two passes, route easy cases cheap and hard ones to the good model. Or you can relocate the work — hand a chunk of what the model was doing to deterministic code that runs for free.

The law is which bucket costs what. Shrinking trades recall for money. The one reorg lever I measured — merged-sharding — traded cost for cost and lost recall; I expect reorganising to net near zero on recall as a class, but that is an expectation, not a measured result. Relocation is the only move that buys recall without paying the model more. Call it Amdahl-flavoured intuition, not a mechanism: you only win by attacking the dominant term, and the dominant term is how much the model has to reason over, not how the reasoning is arranged.

“One law explains all of it” is my inference after the fact, not something I proved lever by lever. It is the cleanest story that fits the outcomes — a working theory with receipts, not a theorem.

Shrinking the input costs recall

The tempting cost lever is the input: fewer tokens in, smaller bill. So I cut context and capped the diff the deep pass would read. Recall fell every time, in the direction you would guess — the model cannot flag what it never sees. A capped diff leaves the tail of the change unread, and the tail is where the quiet bugs sit. The least surprising result in the exercise, and the easiest to disbelieve when the token count looks so good.

One caveat earns its own name: this is about cutting signal. Evicting waste is a different move — stale tool definitions, dead results, context the model has moved past. Treat the window like a cache with demand paging (evict the cold pages; page them back on a “fault” when the model reaches for one) and you reclaim that waste without the recall hit: one OS-framed system reports about a fifth of a session’s tokens are structural waste and pages them out with a negligible fault rate (Mason, 2026). The rule holds — you can’t shrink signal your way to coverage — but not all shrinking cuts signal.

Reorganising moves the cost — and synthetic tests lie

The clever levers are the reorganising ones, and this is where I got burned. I built a merged-sharding scheme: split a large diff into shards, review each, merge, share context across them to avoid paying twice. On a planted-bug fixture it looked great — caught the planted bugs, accounting came out ahead. I nearly shipped on that.

Then I ran it on real PRs and it dropped findings the un-sharded path had caught. The merge seam ate cross-shard issues: a bug whose two halves landed in different shards had no reviewer who saw both. The fixture could not surface that, because I had planted each bug inside one file. It measured what I built it to measure, which was not the thing that mattered. A synthetic test checks internal validity — did the mechanism do what I designed. Real inputs check external validity — does it work on the world. Reorganising levers pass the first and fail the second, because what they break is the seam between the pieces, and your fixtures live inside the pieces.

The only free lever: move it off the model

The one lever that paid was relocation. I took the “did we look at every rule on every changed line” job — pure noticing, no judgment — off the model and onto a deterministic checklist scout. A parser walks the change, forces coverage of every rule against every line, and shortlists the hits. No model, no per-token cost.

Recall went up 33 to 67 percent depending on the run. Not because the scout is smart — it is dumb on purpose — but because it is exhaustive where the model was sampling. A model left to scan a big rulebook quietly skips. The scout cannot skip; that is the whole point of moving the job to code. Free coverage, and the model’s attention freed for the calls only it can make.

A stronger model is a lever you rent

One more lever worked, and it is worth naming so nobody mistakes it for free. Swapping the deep reviewer for a more capable model let me delete rules — the rulebook went from 17 to 7, because the better model caught in judgment what those ten rules used to spell out. A genuine win, but capability rented by the token. It buys recall with money, so it belongs in the “costs something” column with shrinking, not the free column with relocation.

Synthetic for direction, real PRs for decisions

The merged-sharding miss left me a rule. Synthetic fixtures are for direction — fast, cheap, good for “is this pointed right.” Real inputs are for decisions — you do not ship a lever off a fixture, because the fixture cannot see the seams your design breaks. And measure recall as the union over at least three runs: one run undercounts, because the model misses different things each time and a single trial reads worse than the tool is.

That is the part I would carry to any pipeline. Find the noticing a parser could do and move it off the model — that is your free recall. Everything else costs, so price it, and decide it on the real thing.

---

How sure I am: Three results are measured runs of my own, genericised: the scout’s 33-67 percent recall lift, the rulebook dropping 17 → 7 under a stronger reviewer, and merged-sharding passing a planted-bug fixture then losing findings on real PRs. The cost figures are mostly from an offline estimator, not billing — treat them as close, not exact, and re-check against the primary write-up before quoting. The framing, “one law explains every lever,” is my inference after watching the outcomes, not a result I proved lever by lever; I would not be shocked to find a lever it does not cleanly sort. The methodology rule is a habit earned from the single anecdote where synthetic and real disagreed, not a controlled study.

Sources

• The lever ledger — my own instrumented runs of an AI code reviewer: scout +33-67 percent recall; rulebook 17 → 7 under a more capable deep reviewer; merged-sharding passed a synthetic planted-bug fixture and failed real-PR recall. Cost figures from an offline estimator, not billing. Genericised; numbers (verify to primary before publishing).
• Amdahl’s law (as analogy) — Gene Amdahl’s 1967 result that speedup is capped by the term you do not attack; here the dominant term is how much the model has to reason over, so relocation is the free win. An intuition pump, not a derivation.
• Internal vs external validity — the Campbell & Stanley research-design distinction: synthetic tests the mechanism (internal), real inputs test the world (external).
• Demand paging for context — Mason, The Missing Memory Hierarchy: Demand Paging for LLM Context Windows (2026, arXiv:2603.09023) — frames the context window as L1 cache with no paging, and reclaims stale context by eviction with fault-driven page-back (OS working-set theory). Reports ~22% of a session’s tokens are structural waste, and up to ~93% reduction in resident context, in one deployment — the measured case that evicting waste (not signal) is a shrink that need not cost recall. See also ACM: Agentic Context Management for Long-Horizon Tasks (Li et al., 2026, arXiv:2607.23809) — the same move made by the agent itself (lossless self-compression to external memory), qualitative. Both single-deployment / preprint — directional, not laws.

---

Related: Noticing Is Cheap, Judgment Is Scarce — the specific relocation move and where the bill lands. You Can’t Grade Your Own Misses — why a dropped finding never shows up in what you found. Fidelity Was a Vibe, Cost Was a Number — measuring the thing that matters, not the thing that’s easy.