LLMs Retrofit Exception Handling: 85.9% pass@1

The problem: happy-path code ships without armor

Code generation benchmarks obsess over whether a model can write a function that works. Almost nothing measures whether it can add the boring scaffolding that keeps a function from detonating in production: the null check that throws IllegalArgumentException, the guard clause before the throw, the try/catch that converts a crash into a handled state. UT Austin's new paper calls this Exception Related Code (ERC) and proposes a task nobody has formally benchmarked before: retrofitting — given existing code with no error handling plus tests that specify the exceptional behavior, generate the missing guards so the tests pass. This is the codebase-maintenance reality: most industrial code didn't get its error handling written yesterday; it gets bolted on after the incident.

Method: context engineering, not fine-tuning

EXCODER (Exception Coder) wraps an off-the-shelf LLM in a context-engineering pipeline that fuses static and dynamic program analysis. Static analysis extracts the call graph and type information around the target method; dynamic analysis runs the exceptional-behavior tests and feeds back what actually threw, where, and with what. The LLM receives that structured context rather than raw source. The benchmark is refreshingly surgical: 304 real methods from 75 GitHub Java projects, with the authors systematically deleting the existing ERC so the ground truth is code that humans actually wrote and shipped — not synthetic mutations nobody would produce.

flowchart LR
    A[Method sans ERC] --> C[Context builder]
    B[Exceptional Behavior Tests] --> D[Dynamic analysis: run, capture throws]
    A --> E[Static analysis: types, callgraph]
    E --> C
    D --> C
    C --> F[LLM]
    F --> G[Generated guards]
    B --> H{EBTs pass?}
    G --> H
            

Results: strong, but read the k

With Qwen 2.5 Coder 32B, EXCODER hits 85.92% pass@1, 86.18% pass@5, and 86.51% pass@10 — 12.6 to 13.2 percentage points over the raw-LLM baseline. The headline number is good. The interesting number is the spread: 0.6 percentage points from pass@1 to pass@10. That flatness is the receipt the abstract doesn't comment on. When pass@10 ≈ pass@1, sampling more doesn't help — the remaining ~14% failures are systematic (the model consistently picks the wrong exception type, or can't construct the guard condition), not stochastic noise you can vote your way out of. The same pattern showed up in yesterday's bug-fixer analysis on Dispatch: agents confidently damage what they don't understand. Here the confident-retrieval regime buys you 86% and a hard ceiling; the last 14% will need reasoning or feedback loops, not temperature.

Second receipt: the baseline gap is 12.56 points, meaning context engineering — zero gradient updates — accounts for more than half the total performance. The 73.4%-ish raw baseline versus 85.9% is a direct measurement of how much structured analysis input is worth on a task the models were never explicitly trained for.

Limitations the abstract plays down

The tests are the oracle, and that cuts both ways. Exceptional Behavior Tests fully specify which exception to throw and when — so this measures spec-following under test supervision, not the harder problem of deciding what exceptional behavior a method should have. That discovery problem (where IdeaAMBIG showed models miss 90%+ of spec gaps) remains untouched. Second, Java-only: exception taxonomy and checked/unchecked culture differ wildly in Go, Rust, or Python, and nothing here transfers for free. Third, 304 methods is small enough that the 0.59-point pass@1-to-pass@10 gap could hide real per-method variance. And the authors' own manual inspection found imperfect generated code — passing tests is a floor, not idiomatic-ERC certification.

Why builders should care

If you're building code-migration, lint-remediation, or "harden this service" agents: this paper is your template. Retrieval-style context engineering over static+dynamic analysis gets you the first 86 points for free on a task with no fine-tuning budget. But budget honestly: the flat pass@k curve says retry-and-vote loops are dead weight here. Spend your engineering on the failure mode that matters — a verification signal that distinguishes "wrong exception type" from "missing guard entirely" — because that's where the remaining 14% lives, and sampling won't reach it.