Appendix — a worked code case

Most of the receipts in this essay come from making the essay: evals, summaries, prose. Fair criticism — so here’s one that’s only code, from a tool I was building with the model to find the cheapest model a given prompt could run on. Two bugs the model wrote, both of which I would have shipped, neither of which the obvious checks caught.

The bug that passed every check

The tool had a config panel: click “Tune,” a panel opens, you pick the models and the steps, it runs. The model wrote it cleanly. handleTune() set configuring = true to open the panel. Separately, the dropdown menu it lived in had an onInteraction(false) handler that reset configuring = false when the menu closed — also correct, on its own.

Both pieces were right. Together they weren’t. Opening the panel closed the menu; closing the menu fired onInteraction(false); that reset configuring back to false; the panel never appeared. What you saw was a button that did nothing when you clicked it, or flashed “Done · 0 models tried” with no config in sight.

It passed the type-checker. It passed the linter. It passed a read-through review — because each piece is correct in isolation, and the bug lives only in the interaction between two handlers you have to hold in your head at the same time. The diff looked fine. I approved it. What would have shipped is a feature whose main button silently does nothing.

One thing caught it: running it and watching. “0 models tried, nothing happening” is not something a checker can see. It is something a person sees when the screen doesn’t match what the code claims. The fix was three lines — drop the menu-close reset, reset on dialog close instead.

The bug that would have thrown away every win

The same tool had to decide whether a cheaper model was good enough. The model wrote the obvious gate: does the candidate hit 100% on the eval? Clean, simple, passed its tests.

It would have shipped a tool that lies. A rewrite that took a weak model from 60% to 90% does not hit 100%, so the gate scored it a failure and discarded it — and the tool reported “no cheaper model works,” confidently, while throwing away real improvement after real improvement. Every test passed. The tests encoded the same wrong bar the code did, so they agreed with it.

Caught the same way: watching it run and seeing the 90% results dropped on the floor. The fix was not a better model or a smarter gate. It was changing the question — from “did it hit 100%” to “did it beat the model we’re replacing, and never apply a worse one.” A different bar, not a better implementation.

Why this is in the essay

Both bugs were AI-written. Both looked correct. Both passed every check the AI could run on itself. Both would have shipped. What caught them was not a sharper review — it was contact with the running thing, the one check made of different material than the code.

Type-checkers and linters catch what is wrong in the code. They are silent on what is wrong about it — a wrong interaction, a wrong bar — because judging that needs taste, and taste is the one thing a checker built from the same model as the code cannot supply. The mechanical checks caught the mechanical bugs and stayed quiet on the two that mattered. Automate what reduces to a rule; keep a person in contact with the running thing for everything that doesn’t.