LLM Bug-Fixers Damage Correct Code: 29.3% vs 6.2%

Every autonomous repair harness shares the same promise: the agent loops until it finds no more bugs, so the code you ship is code that survived scrutiny. A fresh preprint from UnlikelyAI and the University of Warwick runs that loop until it breaks — and the breakage lands on the code that was already correct.

The problem: blind iteration with no ground truth

Automated program repair tools increasingly run unsupervised: the LLM reviews code, patches it, reviews the patch, patches again — until it reports the code clean. The critical setup here is blind iteration: the agent sees only the current file, never its history. That's not a strawman. It's what happens when past states blow the context window, when a multi-agent pipeline delegates, or when a human rubber-stamps every diff. The question: does the loop converge to correctness, or somewhere worse?

The method: bug-fixing as a dynamical system

The authors treat the fixer as a Markov chain over code states — apply the model, feed the output back in, up to 100 turns, under greedy decoding and sampled decoding (τ=0.7). They seed the loop with verified-correct and verified-incorrect competitive programming submissions (CodeContests+, ~40 LOC median C++), then measure two transition probabilities: the repair rate α (incorrect → correct) and the damage rate β (correct → incorrect). Two edit modes are compared: whole-file rewrites and search/replace blocks (SRBs) — the atomic-edit format used by Aider, SWE-RL, and Agentless. Models: Gemini 2.5 Flash-Lite and Qwen2.5-7B-Instruct.

flowchart LR
    A["Correct code
(passes all tests)"] -->|"damage β
up to 42.4%/step"| B["Patched
(now broken)"]
    B -->|"repair α
0.4–10.1%/step"| A
    A -->|"pseudo-bug claim"| C["'Fix' applied"]
    C --> A
    A -.->|"cycle"| A
            

The results: damage beats repair, everywhere

The headline numbers, from correct starting code with Gemini 2.5 Flash-Lite at τ=0 using SRB edits: 6.2% repair rate versus 29.3% damage rate — the model breaks working code nearly five times more often than it fixes broken code. Qwen2.5-7B is uglier: 4.1% repair against 42.4% damage. The pattern holds at τ=0.7 and with whole-file edits (where the rates are at least comparable rather than inverted). The loop then settles into attractors: pseudo-bug-fixing cycles, where the same change is added and removed indefinitely, including length-2 cycles that oscillate between a passing and a failing state forever. SRBs make all of this worse — more cycles, longer cycles, more degeneration — plausibly because the model can't rewrite toward its internal "canonical" version of the algorithm and keeps "correcting" code that merely doesn't match its template.

The mechanistic part is what elevates this beyond a benchmark complaint. A difference-of-means probe on Qwen2.5-7B finds a "buggy code" direction — cross-validated AUC approaching 1.0 around layer 19-22 — and logit-lens analysis shows it is not just reading the Y/N verdict token off the unembedding. Steer along it negatively (γ=−0.5) and the edit loop stops entirely: all correct submissions preserved, none repaired. Steer positively and the model edits more — and hallucinates search blocks that don't exist in the file. The model has an internal bugginess representation, and on clean code it fires falsely. The false positives aren't a prompting artifact; they're the same internal circuit that finds real bugs, misfiring.

Limitations the abstract doesn't mention

The scale is small: 20 problems, 40 submissions each, single-file C++ averaging 42 lines — nothing like a 5,000-file production repo, and the largest model tested is a Flash-Lite tier. The strongest steering results are from one 7B model. And the "blind" setting is the paper's own construct; real repair agents like RepairAgent carry history, which may break the pure Markov dynamics. The damage-rate number you quote in a design meeting should come with all three caveats.

Why someone building things should care

Two takeaways for anyone wiring LLMs into review or repair pipelines. First, "no bugs found" is not a stopping condition — it's a coin flip weighted against you. An unanchored loop on clean code is a degradation engine; gate it behind test suites, cap the iterations, and treat a claimed fix on already-passing code as hostile by default. Second, the steering result points at a cheap deployment-time control: a probe on internal activations could gate the editing loop with a confidence signal the model can't talk itself out of — a circuit breaker for pseudo-bug fixing. This also connects backward: it's the false-positive twin of the correlated-error and self-testing failure modes we've covered before. The loop's self-assessment is the failure surface, again.