LLM Judge Omission Blindness: 0.50–0.63 on Missing Facts

Ambient AI scribes draft clinical notes for real patients, and the audits keep returning the same verdict: their dominant failure mode is omission — facts the encounter established that never make it into the note. The standard safety net is an LLM judge: a second model reads the note against the transcript and flags problems. New work from Fox, Markham, Lail and Karotsieris asks a question the field had quietly skipped: can the judge even see an omission? Across eight judge designs, the answer is essentially no — paired discrimination on missing content lands at 0.50–0.63, against 0.79–0.94 for added or altered content. And the fix turns out not to be a better prompt, but a different task: list the facts, then check them off.

The Problem: Judges Verify Presence, Not Absence

Prior audits couldn't answer this because the public corpora are structurally unusable: clinician reference notes and transcripts are "materially discrepant," so there is no trustworthy answer key for what a note should contain. The authors build their own — 500 single-error note pairs drawn from audited fact sheets, 298 where a named fact is certainly absent from the note and 202 added-or-altered controls. Single-error pairs matter: they isolate the judge's ability to catch one injected problem without conflating failure modes. The setup mirrors the QA loop already deployed in production clinics — which is precisely the point.

The Method: Eight Judge Designs, Then a Task Restructure

The study runs eight judge designs — different models, prompts, voting schemes, and a GEPA-evolved prompt — and measures paired discrimination: does the judge rank the flawed note below its clean twin? 0.5 is a coin flip. On added or altered content, judges land at 0.79–0.94. On omissions, 0.50–0.63 — statistically indistinguishable from chance for most designs. On single notes, no design flags omissions more often than it flags perfect notes. Wording changes, voting, and prompt optimization only move the operating point; the signal isn't there to extract.

flowchart LR
    A[Transcript + note] --> B[Vanilla LLM judge]
    B --> C[Added/Altered: 0.79-0.94]
    B --> D[Omissions: 0.50-0.63]
    A --> E[Step 1: list facts transcript establishes]
    E --> F[Step 2: check each fact against note]
    F --> G[Detects 24.6-36.9% at 2.7-6.2% false alarms]
    style D fill:#7f1d1d
    style G fill:#14532d
            

Recovery comes from restructuring the task: list the facts the transcript establishes, then check each fact against the note. Two implementations converge on it independently. A per-fact pipeline flags the missing fact and its severity at 2.7% false alarms. A GEPA-evolved single-call prompt does the same list-then-check in one shot.

The Results: Near Chance on Absence, and a 10x Cheaper Fix

The numbers separate the two failure modes cleanly. Omission detection at 0.50–0.63 paired discrimination is a coin flip wearing a lab coat — while the same judges catch 0.79–0.94 of fabricated or altered content. The restructured task changes the economics: the pipeline detects 24.6% of omissions at 2.7% false alarms and names the missing fact; the single-call prompt detects 36.9% vs 24.6% (p=0.002) at 6.2% false alarms and roughly a tenth of the cost per note. A physician author validated 70 items, and where the two routes disagreed, sided with the pipeline on 10 of 10.

Limitations: Absence Is Fundamentally Harder to Verify

The fix is a partial recovery, not a cure: even the best design still misses the majority of omissions (63.1% at best), and false alarms at 6.2% are non-trivial for production where notes are reviewed at volume. The benchmark is clinical-domain and constructed — 500 single-error pairs from audited fact sheets, cleaner than real transcripts where errors compound. GEPA-optimized prompts risk overfitting this specific benchmark, and no cross-domain test shows the list-then-check trick transfers to law, code review, or support logs. There is also an evidentiary asymmetry the paper surfaces rather than solves: a note can prove it contains a fact, but can never prove one isn't missing — verifying absence is structurally the harder direction, for humans and models alike.

Why Builders Should Care

Any eval that only checks "is present content wrong?" is blind to "is required content missing?" — and that blind spot is exactly where the dominant failure mode of AI scribes lives. The lesson generalizes well past healthcare: RAG answers, agent task logs, code review, contract summaries, support transcripts. Three rules fall out. First, structure evals as list-then-check when completeness matters — decompose the source into checkable claims before grading the output. Second, hallucination detectors are not omission detectors; measuring fabrication tells you nothing about what was dropped. Third, if your judge is a prompted second model, validate it against an absence-labeled set like this one — a judge that scores at chance on missing content isn't a safety net, it's theater.