AL Alfred 2026-09-05 4 min read

Paper-Code Discrepancy Detection: Dude Lifts F1 18.7%

Every serious ML paper ships with code — and almost none of it gets independently checked. Peer review doesn't have the capacity: submissions outscale the humans who could diff a release against its own abstract. LLM-based discrepancy detection was the obvious answer, but single-agent readers have a structural problem — bounded context and one-sided reading. Dude, accepted to EMNLP 2026, reworks the pipeline as a dual-detection multi-agent system, and the numbers justify the complexity: recall and precision improve by up to 22.8%, F1 by up to 18.7%.

The Problem: Single-Agent Detectors Can't See Both Sides

Paper-code discrepancy detection means finding places where a paper's claims and its published code disagree — a missing ablation, an evaluation that doesn't match the described setup, a metric computed differently in code than in prose. Existing LLM-based approaches run a single agent over both artifacts, and that fails twice. First, context: a real paper plus a real repository exceeds what one pass can hold, so the agent reads one side shallowly. The result is low recall — the discrepancies that matter get missed. Second, the fix that seems obvious — split the work across agents — introduces a failure of its own: granularity asymmetry. Papers describe intent at paragraph level; code implements it at statement level. Let two agents loose without aligning those levels, and they over-interpret each other's claims and over-report, and false positives explode. That's the trap Dude's design is built around.

The Method: Dual Detection, Negotiation, Then Filtering

Dude runs two specialized detectors in parallel: one reads the paper language, one reads the code language. The core contribution is what happens between and after them. A granularity-aligned negotiation forces the two agents to reconcile their findings at a common level of abstraction before anything is compared — a paper-level claim is matched against the code region that implements it, not against stray statements. Then a two-stage salience-filtering mechanism drops low-value or low-confidence reports before anything reaches a human. The intent is explicit: attack the false-positive problem that naive multi-agent designs create, not just add another reader.

flowchart LR
    P[Paper Text] --> A1[Paper-Side Agent]
    C[Code Repository] --> A2[Code-Side Agent]
    A1 --> N[Granularity-Aligned Negotiation]
    A2 --> N
    N --> F[Two-Stage Salience Filtering]
    F --> V[Verified Discrepancy Report]

The Results: Recall and Precision Up 22.8%, F1 +18.7%

On real-world paper-code discrepancy datasets, Dude beats the baseline single-agent paradigms on both axes at once: recall and precision improve by up to 22.8%, lifting F1 by up to 18.7%. The important shape of these numbers is that recall and precision move together. A naive multi-agent system typically buys recall at the cost of precision — more eyes, more noise. Dude's negotiation-plus-filtering stack is specifically what lets it raise both, which is exactly the claim to stress-test when this lands in a review pipeline.

The Limitations: Faster and Cleaner, Not Infallible

The gains are real but bounded. Results are reported with "up to" hedges, so improvement varies across datasets — the negotiation and filtering machinery earns its keep unevenly. False positives are reduced, not eliminated, and salience filtering can in principle suppress a true positive that reads as low-salience. The system inherits the context problem it was built to escape: for very large repositories, each agent still has to fit its half in context, so the deepest discrepancies can fall outside both windows. And detection is only detection — Dude flags disagreement; it doesn't adjudicate which side is right, so a human still has to make the call. Finally, dual agents plus negotiation cost tokens and orchestration overhead per paper, which matters if you're running this at submission scale.

Why Builders Should Care

Three reasons. First, if you build research tooling — reproducibility audits, review assistants, evaluation brokers — Dude's architecture is a directly stealable pattern: parallel detectors, granularity-aligned negotiation, salience filtering. The false-positive lesson generalizes past papers: any multi-agent verdict pipeline that merges opinions at mismatched granularities amplifies noise. That's the same failure mode the LLM judge reliability audit found on shared endpoints — instruments that look stable until you measure them. Second, paper-code verification is quietly becoming a real job category as agentic reviewers scale past human capacity; Dude is the current bar. Third, it pairs with what we already know about verification gaps: functional tests aren't enough for code agents (SWE-Gate), and citations need independent reproduction (Perplexity audit). Discrepancy detection is the same war on a different front — the artifacts themselves.

Related Posts