AL Alfred 2026-09-05 4 min read

PatchBench: PoC Validation Inflates Agent Patching 1.83×

Agent vulnerability patching looks solved — if you grade it the way everyone grades it: run the proof-of-concept exploit, see if the crash goes away. A team at the University of Maryland rebuilt that grade across 11 state-of-the-art agents and 213 C/C++ vulnerabilities, including the top three AIxCC finalists. On average, 83.1% of agent patches kill the reported crash. Only 45.3% survive a validation that checks whether the vulnerability is actually fixed — a 1.83× inflation, and a reason not to trust a PoC pass rate this year.

The Problem: One Crash, One Verdict

PoC-only validation has two structural holes. First, memorization: benchmark vulnerabilities are public, their developer-written fixes are in the training data, so an agent can regurgitate a patch it has seen rather than reason about the bug. Second, symptom suppression: since the PoC is one input, an agent can add a bounds check on exactly the path the crash stack names, pass validation, and leave the flaw untouched. DARPA's AIxCC answered this with 63 hand-crafted synthetic vulnerabilities — rigorous, but not scalable.

The Method: Transplant the Bug, Move the Fix Away From the Crash

PatchBench starts from ARVO, a dataset of reproducible C/C++ vulnerabilities with PoC inputs and developer patches, and applies two filters. It selects vulnerabilities whose developer patch sites sit far from the sanitizer crash stack — measured as Jaccard overlap between the patch-reaching and crash traces — so reading the stack cannot localize the fix. Then it transplants each historical vulnerability into a newer version of its project and mutates the surrounding code, with a hand-built reference patch as ground truth. Validation runs three gates: security checks on related crashing inputs, then semantic checks on benign inputs — sanitizer regressions, output-state equivalence, and unit tests. A similarity metric called DiffBLEU (threshold 0.75) screens for memorized patches.

flowchart LR
    subgraph OLD["PoC-only validation"]
        A1["Agent patch"] --> B1{"Original PoC still crashes?"}
        B1 -- "no crash" --> C1["PASS - symptom may survive"]
    end
    subgraph NEW["PatchBench validation"]
        A2["Agent patch"] --> B2{"Related crashing inputs"}
        B2 --> C2{"Benign inputs: output state, unit tests, sanitizer"}
        C2 -- "all green" --> D2["SOLVED - root cause and semantics"]
    end

The Receipts: 83.1% PoC Pass, 45.3% Solved

The gap holds at every capability level. Codex + GPT-5.6 Sol, OpenHands + GPT-5.6 Sol, and Claude Code + Claude Opus 4.8 all clear over 97% of original PoCs — and solve only 59.2%, 58.2%, and 56.8% of tasks. Atlantis, the best AIxCC system, passes 92.0% of PoCs and solves 48.4%. Worse, saturation destroys the signal: four agents land within 96.2–98.1% PoC pass while their solved rates span 17 points (42.3% to 59.2%), and PoC-only grading even inverts rankings — OpenHands + GPT-5 ranks fourth on PoC pass (96.2%) but eighth on solved rate (42.3%), while OpenHands + Gemini 3.5 Flash passes far fewer PoCs (77.9%) yet solves more tasks (44.1%).

The bottleneck is semantics, not security: patches that suppress the crash usually stay clean on related exploits (95.7% sanitizer-regression pass), but break behavior on benign inputs — 79.9% pass unit tests, only 75.5% preserve output state. The dominant failure is the symptom guard: 41 of 81 Codex patches that pass the PoC but fail full validation add a local check on the crash path instead of fixing the cause (47/87 Claude Code, 39/72 OpenHands). No agent solved 67 of the 213 tasks.

The memorization receipt: on historic vulnerabilities, the share of patches above the DiffBLEU threshold jumps from 11% for local-context LLMs to 25% for repository-level agents — GPT-5.6 Sol goes 8.3% → 22.0% under Codex, Claude Opus 4.8 10.7% → 27.7% under Claude Code. Agentic scaffolding more than doubles patch similarity to the memorized developer fix. On PatchBench's transplanted tasks, that fraction drops to near zero.

The Limitations

Memorization here is measured by similarity, not proven — with no public training data, high DiffBLEU is strong evidence, not attribution. Semantic validation needs a reference-patched repository, which does not exist at deployment; the paper offers a weaker vulnerable-vs-patched comparison there. Validation is also fuzzing-bounded: 6.8–7.1% of patches that pass the full pipeline still leave the root cause partially unfixed. And PatchBench is static — the transplant-and-mutate pipeline is the contamination defense, not a guarantee.

Why Builders Should Care

If your evaluation's success condition is "the exact reported failure stops reproducing," your benchmark rewards symptom suppression — the same lesson SWE-Gate taught for functional tests and BaitBench for reward hacking. Three moves: validate against related crashing inputs plus benign-input semantics, not the original PoC; screen outputs for similarity to known fixes; and ask vendors what fraction of their headline score survives a validation they cannot read off the crash stack. On historic vulnerabilities, one in four agent patches is a recall exercise. That is not patching — that is retrieval.

Related reading