LLMs Miss Implementation-Critical Spec Gaps: 9.6% Recovery

Everyone is pointing coding agents at research repos and asking them to "implement this paper." Yale's new IdeaAMBIG benchmark (arXiv:2609.10539) puts a number on what that actually costs: the best of 13 evaluated LLMs recovers only 9.6% of implementation-critical defects in research-method specifications on real-world instances. The model that can't find the gap can implement the wrong thing with total confidence.

The problem: specs read fine and implement wrong

A research idea can be novel, coherent, and plausible, and still leave the method under-specified. Which variant of the loss? What happens to edge-case inputs? Which of two contradictory details in the paper wins? IdeaAMBIG formalizes this as codification readiness: can a competent implementer or coding agent construct the intended method without making unsupported assumptions? The benchmark contains 660 evidence-grounded instances — 163 real-world gaps harvested from reproducibility reports and GitHub issue threads, and 497 synthetic gaps injected into otherwise codification-ready references. Each gap has a supported resolution, so grading is grounded rather than vibes-based.

The method: three capabilities, one bottleneck

The evaluation isolates three capabilities. Codification-readiness assessment: does the spec suffice as written? Defect localization: find the implementation-critical gap, given only the spec. Clarification action generation: propose a resolution, but here the model is handed the annotated defect. That asymmetry is the experimental design's sharpest move — it splits "can you find it" from "can you fix it."

flowchart LR
    A[Spec + codebase] --> B{Defect localization
spec only}
    B -->|9.6% found| C[Clarification
given the defect]
    B -->|missed| D[Silent wrong assumption]
    C -->|80.6% resolved| E[Correct implementation]
            

The results: finding is the bottleneck, not fixing

The 80.6% clarification success rate against 9.6% defect recovery is the headline. Given the defect, models are good at proposing reasonable clarifying questions or resolutions. Without it, they walk past the gap. The oracle study makes the ceiling explicit: supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Almost all of the failure mass lives in detection.

The comparison nobody should miss: this inverts the usual agent-failure story. We keep benching agents on execution — tool use, long horizons, multi-step coding — and they keep getting better. IdeaAMBIG measures the step before execution: noticing that the spec itself is broken. That capability is barely improving across 13 frontier models, and the authors report it's the bottleneck in every model evaluated.

Limitations the paper glides past

First, 163 real-world instances is small; the 9.6% figure carries wide error bars, and the 497 synthetic gaps may be systematically easier to spot than organically-argued GitHub complaints — the real-world number is the honest one. Second, "recovery" is graded against supported resolutions, which penalizes models that flag a defect differently than the annotation scheme intended; near-miss localization is invisible in the Macro Defect Recovery Rate. Third, the benchmark tests spec-reading in isolation. A real implementer runs the code, hits the ambiguity as a crash or silent wrong result, and iterates. IdeaAMBIG measures one-shot detection, which is the hardest and least agentic configuration.

Why builders should care

If you're wiring agents to reproduce papers or implement methods from internal specs, the failure mode isn't incompetent coding — it's confident coding against an underspecified contract. The practical fix falls straight out of the 9.6%/80.6% asymmetry: don't ask the agent to both find and fix. Force a pre-implementation spec audit that produces explicit clarification questions before any code is written — the oracle study says resolving even human-supplied questions moves codification-readiness from 14% to 98%. And keep a human in that loop; the model won't reliably raise its hand on its own.