al Alfred 2026-09-15 4 min read

VLoc Bench: Best Agent Scores 0.229 F1 Finding Vulnerable Code

We've spent two years celebrating agents that fix vulnerabilities. SWE-bench-style leaderboards reward the model that writes the passing patch. This paper asks the question underneath: can the agent even find the broken code in the first place? The answer, measured across 27 models: mostly no. The strongest system posts a 0.229 File F1, and on 38.4% of tasks, not a single evaluated model returns a correct localization.

The problem: detection benchmarks never tested the search

Cybersecurity evaluations for LLM agents cluster around three verbs — detect, reproduce, repair. All three implicitly assume the hard part, navigation, is solved. It isn't. Real security work on an unfamiliar 300-file repository means reading a CWE description, forming a search strategy, and walking the codebase until you've isolated the affected files. Nobody had measured that step in isolation, which means every "agent found and fixed the CVE" headline carries an unexamined assumption about localization ability.

How VLoc Bench isolates it

The benchmark contains 500 real-world vulnerabilities from 290 repositories spanning six package ecosystems and 147 CWE categories. Each task is a matched pair of repository snapshots: one taken immediately before a security fix, one immediately after. The agent gets only the CWE description and read-only terminal access. On the vulnerable snapshot it must name the affected files; on the patched snapshot it must correctly conclude the vulnerability is gone. That second condition is the clever bit — it tests whether the agent can tell "I searched and found it" apart from "I searched and it isn't there," a distinction benchmarks almost never draw.

Twenty-seven LLMs and four static-analysis tools run under one common agent interface, so the comparison is apples-to-apples rather than leaderboard folklore.

flowchart LR
    A[500 real CVEs
290 repos, 6 ecosystems] --> B[Vulnerable snapshot
CWE desc + read-only shell]
    A --> C[Patched snapshot
same setup]
    B --> D["Must return
affected files"]
    C --> E["Must say:
not present"]
    D --> F[File F1 scored]
    E --> G[False-report rate scored]

Results: the receipts

The number that should rewire how you read agent-security demos is the third line. A model scoring high on vulnerable snapshots but hallucinating findings on clean code isn't a locator; it's a generator of plausible-sounding file paths. If your security pipeline feeds its output to a human triager, that model's confident wrong answers cost real analyst hours.

Limitations, stated plainly

File-level F1 is coarse — a vulnerability that lives in one function gets scored across whole files, which flatters models that dump large file lists. The pre/post snapshot structure assumes the fix commit is a clean marker of the vulnerability's location; real fixes often touch adjacent hardening code. And the paper measures localization, not exploitation — a model that finds the file still has to understand the bug to be useful in a repair loop. The authors also don't publish per-ecosystem breakdowns in the abstract-level results, so whether one package ecosystem (say, npm's dependency-heavy repos) drives the failure mode is an open question.

Why builders should care

If you're shipping an agent that triages security alerts, don't trust its "clean" verdicts. The 38.4% zero-coverage figure means a fifth-to-half of real vulnerabilities in the wild are effectively invisible to current models — so absence of a finding is not evidence of absence of a bug. And if you're building evals: the pre/post snapshot pattern here is the right template for any capability you suspect is being masked by a downstream skill. Measure the search before you measure the fix.

The deeper reading: detection, localization, and repair are three separate capabilities, and the industry has been benchmarking only the bookends. The middle step — where the agent actually does security work — is where it's weakest, and nobody noticed because no one was scoring it.

Related reading