SWE-Gate: Passing Functional Tests Is Not Enough — 34.3%

Every coding-agent benchmark you have seen this year shares one assumption: if a patch makes the tests pass, it is correct. SWE-Gate is a new 303-instance benchmark built to break that assumption — and the numbers are not flattering. Among 644 repairs that pass functional tests, 221 — 34.3% — still violate the review constraints a human maintainer would enforce.

The Problem: Test-Passing Overestimates Coding Agents

Repository-level benchmarks like SWE-bench grade agents on a binary signal: do the repo's functional tests pass after the patch? That signal was designed for bug-fix verification, but real software development has a second gate no test captures — the pull request review. Maintainers reject patches that add dependencies the issue didn't ask for, rename public APIs, touch unrelated files, or solve the problem in a way the spec forbids.

SWE-Gate's authors argue this blind spot inflates every functional-only score: an agent can "solve" an issue while failing the actual acceptance criteria of the task. You cannot measure what you never test.

The Method: Review Constraints Become Tests

The team mined real pull-request review comments from 75 open-source Python repositories and synthesized 303 repository-level repair instances around the constraints those reviews enforced. Every instance ships with two decoupled test suites — functional tests and constraint tests — plus a non-compliant patch and a gold patch.

That split is the whole trick: an agent is now scored twice, once on "did it fix the issue" and once on "did it respect the review constraints." Four LLM backends of varying capability ran the same coding-agent scaffold.

flowchart LR
    A[PR review comments
75 Python repos] --> B[303 repair instances]
    B --> C[Functional tests]
    B --> D[Constraint tests]
    C --> E[Agent patch]
    D --> F[Gate score:
fix + constraints]
    E --> F

The Numbers: 221 of 644 Test-Passing Fixes Fail

Across all four backends, 644 repairs passed the functional tests. Of those, 221 (34.3%) failed the review constraints. Roughly one in three patches a functional-only evaluation would call "correct" does not satisfy the full repair specification.

That gap persisted across every backend regardless of capability level — it is a property of the evaluation, not a weakness of one model. The authors publish the full replication package, including code, data, and experimental results, on GitHub.

Limitations

• Python-only, 75 repos. Review norms differ wildly across ecosystems; C++, Java, or TypeScript shops may see different gap sizes.
• Synthetic instances. Constraints are extracted from real comments, but the synthesis step can lose the tacit context of an actual review thread.
• Constraints ≠ merge probability. Complying with extracted constraints is not the same as a maintainer actually clicking merge.
• One scaffold, four backends. Scaffold design interacts with constraint compliance; results could shift under other harnesses.

Why It Matters for Builders

If you are shipping a coding agent, functional pass is your floor, not your ceiling. A third of the "solved" issues in SWE-Gate were solved wrong. The fix is mechanical: add a second evaluation gate that checks spec compliance — diff-level lints, forbidden-touch rules, dependency budgets — before you trust or ship an agent patch.

And if you are designing the next benchmark: test-pass-only scoring is a known-bad proxy, and SWE-Gate now has a number attached to it. Agents are being graded on the easy half of the job.