Your Merged-PR History Is Survivorship-Biased Training Data

“The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted.” — John Tukey (1986)

In the Second World War, the Statistical Research Group was asked where to add armour on bombers, and handed the damage data from the planes that came back — most shot up on the wings, fuselage, and tail. Abraham Wald’s insight was to read that data backwards. The holes showed where a plane could take fire and still come back. The armour belonged where the survivors had no holes — the engines, the cockpit — because the planes hit there weren’t in the sample. They hadn’t returned to be measured.

If you’re training a code reviewer on your own merged-PR history, you’re armouring the wings. Your corpus is the bombers that came back.

BLUF

• Merged code is post-cleanup. By the time a PR merges, the violations a reviewer exists to catch have been removed — by the author, the linters, the reviewers. Train on that and the model would learn violations are rarer than they are, and under-flag in production. (I’m reasoning forward here — I investigated the data, I haven’t trained the model; see How sure I am.)
• If you squash-merge, that destroys the “before.” The reviewer saw an early commit; the fixes came in later commits; squash collapses all of it into one final commit on main. Reach for the pre-fix state and it isn’t earlier in the mainline history — it isn’t there at all. (Merge-commit or rebase workflows keep it; squash — in my experience most teams’ default — does not. The pre-squash commits linger on the PR platform, but recovering them is a separate, lossier job than reading the merged history.)
• So synthetic injection is the tempting fix. If the real dirty examples were deleted before you could record them, manufacturing them is one way to get a positive class — carefully, and not the only one.
• But you can’t over-inject. Flood the corpus with planted violations and you skew the base rate: the model turns trigger-happy, and precision can fall unless you calibrate for the deployment distribution. Synthetic fills the gaps; it doesn’t set the distribution.
• Distillation is bounded by the data, not just the teacher. A student can sometimes out-generalise a noisy teacher — but it can’t learn a violation the corpus never contained. The ceiling is coverage: survivorship caps the result before the first epoch.

The corpus is the survivor

A merged pull request is a success story. Whatever was wrong with it early — the missing guard, the leaky abstraction, the banned import — got fixed on the way in. That’s the point of review; it’s also what makes merged history the wrong place to learn what unreviewed code looks like. The examples a reviewer most needs to recognise are precisely the ones the merge process was built to erase.

Train a model to flag violations using a corpus scrubbed of violations and it learns the wrong base rate. It sees clean code labelled “shipped” and concludes the bad patterns are rare — so in production, faced with the dirty code it was supposed to guard, it shrugs. You didn’t teach it to catch violations. You taught it that violations don’t happen here.

And “scrubbed” is too kind. Merged code isn’t clean — it’s missing the violations that got caught, and still carrying the ones review missed: the subtle bugs nobody flagged. The lesson is wrong twice over — the easy violations look rare because they were removed, and the hard ones sit in the training set labelled shipped, good. You end up teaching the model to wave through exactly the subtle bugs that already slipped past a human, because your own data calls them clean.

Why the obvious fix isn’t one

The intuitive escape is to reach earlier in history: don’t train on the merged state, train on the code as the reviewer saw it — an earlier commit, before the fixes landed. The problem is that squash-merge throws that commit away. A review runs on some commit; the fixes arrive in later commits; and when the PR is squash-merged, the whole sequence collapses into a single commit on the main branch — the final, post-fix state. Mine your merged history and that squashed “after” is all that survived. The “before” the reviewer actually looked at isn’t earlier in the history — it isn’t in the history at all.

That also demotes a number that looks like the problem but is really a symptom: sample the merged history and the large majority of PRs read as single-commit (my count, on a thin ~30-PR sample, so read the share as directional) — because squashing made them single-commit. There’s no earlier commit to reach for; the reach-earlier fix has nothing to reach. (The pre-squash commits do linger on the PR branch until it’s deleted — but recovering those is a different job than “read the merged history,” and a force-push erases even them.)

So the fix everyone reaches for isn’t a base-rate fix at all. On a squash-merged history it’s a no-op: you can’t recover a “before” that the merge policy was built to discard.

So you inject — but injection has its own bias

If the real violations were deleted, manufacture them: take clean code and inject the patterns you need the reviewer to catch. On a survivor corpus this is the tempting shortcut to a positive class — but not the only source. Rejected PRs, PR revisions, bug-fix antecedents, and human-labelled positives all carry real violations you can mine first; synthetic injection fills what they leave uncovered.

The catch is the base rate again, from the other side. Inject too much and you risk inverting the problem: the model can learn violations are common and turn trigger-happy, recall up and precision down, unless you calibrate against a validation set that reflects the real prevalence — the deployment distribution, not the injected one. Class imbalance is manageable: oversample the positives, then calibrate the threshold. So over-injection is a risk to handle, not an automatic precision loss.

The point is that synthetic sets the variety, not the base rate you deploy against. Synthetic examples fill the variants the corpus lost; they don’t define how often violations occur. The real modern violations you can find are in-distribution gold; synthetic is the patch over the holes survivorship left, sized to match reality.

The ceiling you set before you start

The last limit is structural. Distil a reviewer — train a cheaper student to imitate a stronger teacher’s judgments — and it’s tempting to say the ceiling is the teacher: the student can’t out-review what it copied. That’s not quite right, and the exception is the point. A student can beat its teacher. A same-size student trained on the teacher’s outputs often generalises better (the “born-again” result); the usual explanation is that distillation smooths noisy labels.

What a student cannot do is learn a violation the training signal never contained. Born-again gains come from denoising labels you already have; they don’t conjure a positive class survivorship deleted. The real ceiling isn’t the teacher’s skill; it’s the coverage of the data. If the corpus never showed the pattern, no amount of training on it reliably puts it there. A pretrained model’s priors might still catch some unshown patterns — it saw the world before it saw your corpus — but you can’t count on catching what your own data never recorded, and it’s hard to measure that gap from your own output. “Cheaper and just as good” is a claim you measure, not assume — but “learns the violations we never recorded” isn’t even on the table.

The point

Before you train on your own history, ask what your history systematically threw away. For a code reviewer the answer is exact: it threw away the examples the reviewer is for. The merged diff is the returning bomber, and the bug you most need to catch is the one that kept a plane from coming back. You can still build the corpus — but only if you count the missing holes, inject to fill them without faking the base rate, and remember that a student usually can’t out-review the teacher it was trained to copy. Usually. Born-again nets are the exception.

---

How sure I am: I investigated the training data for a reviewer I planned to build — I have not trained or distilled the model yet. So the checkable, lived part is the data: the corpus census I actually ran — roughly 248,000 closed PRs scanned — and the squash-merge finding. Everything downstream — that a model trained on it would learn the wrong base rate and under-flag, that over-injection would invert it, that distillation would cap at the data’s coverage — is reasoned prediction, not a measured result. The core of the “you can’t reach the before” argument is a policy fact — squash-merge leaves only the post-fix state in the merged history — not a measured percentage; the single-commit share (a thin ~30-PR sample) is a downstream symptom of squashing, so the argument no longer leans on that number. The distillation coverage ceiling is an argued limit: I’ve granted that students can out-generalise teachers (born-again networks) and narrowed the claim to what the data can’t contain — still not something I’ve measured here. The Wald story is history — told per the sober version, not the dramatized “Wald corrected the brass” one — and the analogy is mine.

Sources

• My own training-corpus notes — corpus census (~248,000 closed PRs), a small-sample commits-per-PR estimate (flagged as thin), and the survivorship / synthetic-injection reasoning. Working documents, genericised.
• Abraham Wald / the Statistical Research Group (WWII) — the returning-bombers case, the canonical survivorship-bias example. Framed per the sober history (the AMS’s The Legend of Abraham Wald), not the dramatized “Wald corrected the brass” telling.
• Hinton, Vinyals & Dean, Distilling the Knowledge in a Neural Network (2015) — distillation transfers a teacher’s behaviour to a student.
• Furlanello et al., Born-Again Neural Networks (2018) — a student can out-generalise its teacher, which is why the ceiling here is framed as data coverage, not teacher skill: distillation denoises labels you already have; it can’t add a class the corpus lost.
• John Tukey, Sunset Salvo (1986) — data plus a desire for an answer doesn’t guarantee a reasonable one.

---

Related: You Can’t Grade Your Own Misses (the reviewed state is a specific earlier commit — there you pin it to score misses; here squash destroys it, so you can’t even train on it), Teaching the Reviewer (freezing judgment into a tool — this is the training-data version, and its trap), Your Codebase Is the Prompt (models regress toward what they’re trained on; a scrubbed corpus regresses toward “nothing’s wrong”), Noticing Is Cheap, Judgment Is Scarce (what to mechanise vs learn).