ExecCritic: Bad Agent Tests Cut SWE-bench 61.2% to 57.3%

Everyone assumed execution feedback can only help a coding agent. A new paper called ExecCritic puts a number on the case where it actively hurts — and the number is the receipts. On SWE-bench Verified, holding the repair agent fixed, adding tests written by a base-model Test agent dropped the resolved rate from a no-test baseline of 61.2% to 57.3%. Bad tests aren't neutral overhead. They're a four-point tax.

The problem: the agent grading its own homework

The standard agentic repair loop uses tests to check patches. The failure mode the paper names is structural: when the same trajectory writes both the patch and the test, the two can share the same misunderstanding of the issue. The patch encodes the wrong behavior; the test validates the wrong behavior. Errors agree, and execution feedback becomes false confidence — worse than no feedback, because the agent now has an authority figure endorsing its mistake. This is the same structural finding as the 2,080-run self-verification study from earlier this week, but ExecCritic does something that paper didn't: it treats the failure as a training problem, not just a measurement problem.

The method: separate roles, frozen tests, closed doors

ExecCritic's scaffold has three moves. First, role separation: a Test agent independently generates repository-native tests, and a Repair agent fixes source code from their execution feedback without touching the tests. Second, a fail-closed harness qualifies and freezes the tests — tests that don't run, or pass vacuously, don't enter the loop. Third, role-specific RL: both agents are Qwen-3.5-35B-A3B trained separately — "Learn to Test" teaches the Test agent to produce tests that discriminate correct from incorrect patches (measured as Base-to-Gold pass discrimination), and "Test to Improve" teaches the Repair agent both direct resolution and feedback-guided revision.

flowchart LR
    A[Issue] --> T[Test agent]
    T --> F[Fail-closed harness
qualify + freeze tests]
    F --> R[Repair agent]
    R --> E[Execute frozen tests]
    E --> R
    R --> P[Patch]
    style F fill:#27272a

The results: the test agent is the bottleneck

The ablation is the paper. Holding the Repair agent fixed, the quality of the Test agent determines whether feedback helps or hurts: base-model tests → 57.3% (below the 61.2% no-test baseline); GPT-5.6-sol-written tests → 65.3% (above it). Same repair agent, an eight-point swing driven entirely by who wrote the tests. Post-training the Qwen Test agent lifts its Base-to-Gold discrimination from 22.2% to 62.2%, and the composed pair of post-trained Qwen agents reaches 72.6% on SWE-bench Verified — 11.4 points over the no-test baseline, with no stronger model or oracle feedback at evaluation time.

Limitations the abstract doesn't mention

Three things deserve skepticism. First, the model comparison is asymmetric: the headline "test quality decides the verdict" claim leans on GPT-5.6-sol as the good test writer, so part of the gain may be a stronger model, not the scaffold — though the 72.6% from two same-sized Qwen agents partially answers that. Second, everything is Qwen-3.5-35B-A3B; the RL recipe's transfer to other backbones is untested. Third — and this is the one that will bite people who skim the abstract — agent-written frozen tests are still agent-written. The Test agent is trained to discriminate, not to be correct; a test suite that systematically encodes a plausible-but-wrong reading of an issue would fool the discriminator metric exactly as well as the repair agent. The paper freezes tests to prevent repair-side gaming, but nothing in the design audits the tests against the issue text.

Why someone building things should care

If your agent harness uses self-generated tests as a confidence gate, measure the counterfactual before trusting it — run the same patches with tests and without, and check which resolves more. If you build evaluation harnesses, the ordering here matters more than the model: qualify (fail-closed), freeze, then let a separate role consume the feedback. And if you train agents, "Learn to Test" is a genuinely underexploited RL target — 22.2% to 62.2% on test discrimination is a bigger relative jump than most repair-side recipes report. The test writer, not the coder, is the cheapest place to buy points.