TruthfulQA Leakage: 68.9% Without the Question

Here's a sentence that should end some leaderboard arguments: a six-feature logistic classifier — no embeddings, no pretrained model, never shown the question — separates TruthfulQA's correct answers from wrong ones at 68.9% accuracy (AUC 0.715). The best model score on the llm-stats TruthfulQA leaderboard as of May 2026 is 66.9% (Granite 3.3 8B Instruct). A probe that reads only the answer strings beats the best model on the planet at a "truthfulness" benchmark. That means a chunk of TruthfulQA isn't measuring truthfulness. It's measuring style.

The problem: binary-choice benchmarks fail silently

Binary-choice truth benchmarks ask a model to pick between a correct and an incorrect answer. That format is only interpretable if the two options differ only in truth. If wrong answers are systematically longer, more hedged, or loaded with negations, a model can exceed chance by exploiting the style asymmetry — and nobody knows, because the score still looks like a normal benchmark score. This is the annotation-artifact problem that broke SNLI a decade ago (hypothesis-only models hit ~67%), re-armed for the LLM evaluation era. Namjoo et al. (arXiv:2609.13003) show it's not a historical curiosity: it's live in the most-cited AI safety benchmark we have, and it's exploitable by a classifier, not just a statistical wrinkle.

The method: Surface6 and Audit-Prune

The authors define Surface6: six surface-level features computed from the answer string alone — covering negation leads, hedging, and length. No semantics. They train a logistic classifier on these features, cross-validated with question-pairs kept in the same fold. A high AUC means correct and incorrect answers differ systematically in form. For cleaning, they introduce Audit-Prune: iteratively score every pair with the probe, remove the most leakage-reinforcing pairs, refit, and add back only what can survive without the cues. Applied to TruthfulQA it produces TruthfulQA-476, a 476-pair subset where the probe collapses to AUC 0.528 (p = 0.048) — barely above chance.

The results: TruthfulQA is not the worst case

The uncomfortable finding is the cross-dataset audit. HaluEval QA audits at AUC 0.973 — its hallucinated answers are so stylistically distinct that a probe which never reads the question or retrieved context recovers the labels almost perfectly. HaluBench inherits this directly: 10,000 of its 14,900 test examples are copied verbatim from HaluEval QA with the original labels, and the probe hits AUC 0.971 on that slice. MedHallu, a medical hallucination benchmark, audits at 0.821 — and notably, word count alone only reaches 0.710, so this is not purely a length artifact. At the clean end: BoolQ (0.525) and PIQA (0.509) sit at the edge of detectability.

mindmap
  root(("Surface6 probe AUC
higher = more leakage"))
    HaluEval QA
      0.973
    HaluBench slice
      0.971
    MedHallu
      0.821
    TruthfulQA
      0.715
    SNLI
      0.591
    TruthfulQA-476
      0.528
    BoolQ
      0.525
    PIQA
      0.509
            

Three details the abstract doesn't carry. First, the cleaned subset isn't a cherry-pick: 239/425 Adversarial and 237/365 Non-Adversarial TruthfulQA pairs survive, all 37 categories retain at least one pair, and model rankings on TruthfulQA-476 correlate with the full set at Spearman ρ ∈ [0.91, 0.98] — you lose the leakage without losing the ranking signal. Second, the obvious baseline fails: AFLite (the embedding-based filtering method from the Heuristic-Debiasing literature) leaves substantially more leakage at matched size — AUC 0.603 at N=476 versus 0.528 — because it removes whatever an embedding classifier finds any cue for, including meaningful ones, rather than targeting the six interpretable surface cues specifically. Third, some datasets (SelfCheckGPT, OpenBookQA) are "solvable" by the probe only at majority-class rates, which is why AUC rather than raw accuracy is the right leakage metric — a point the paper makes correctly and most shortcut-learning writeups skip.

Limitations the authors underplay

• Six features is an audit, not a proof of cleanliness. Surface6 targets negation, hedging, and length. A benchmark could pass this audit at AUC ≈ 0.5 and still leak through cues outside the six — sentence structure, entity frequency, punctuation. TruthfulQA-476 is "close to chance" on this probe, which is the strongest claim anyone can make.
• Leakage ≠ score inflation, quantitatively. The paper shows the probe can exploit the cues and that models exploit surface features in related settings, but it doesn't decompose a specific model's TruthfulQA score into style-explained vs. truth-explained components. How many points of Granite's 66.9% are style credit? Unknown.
• Removal bias. Pruning the leakage-reinforcing pairs removes exactly the questions where truth and style conflict most — arguably the most diagnostic items. Keeping ranking fidelity (ρ ≥ 0.91) suggests the loss is tolerable, but fidelity measured across a few models is a weaker guarantee than it looks; a benchmark that can no longer distinguish models on the hard boundary cases may look "fine" at ρ = 0.91 while being easier.
• Probe-vs-leaderboard comparison is apples-to-oranges and the authors say so — the probe reads both answers for one question while the leaderboard scores generated answers. It's a rhetorical comparison, effective but not strictly commensurable.

Why builders should care

If you build evals — internal or public — run the Surface6 probe before you trust a binary-choice format. It's six string features and a logistic regression; it costs minutes. The generalizable rule from this paper: every paired-choice benchmark is a candidate for answer-side style leakage, and the fix is a probe audit plus targeted pruning, not more questions. And if you buy models off leaderboards: TruthfulQA and everything in the HaluEval lineage carry a known upward style bias. A model scoring 66.9% "truthfully" against a probe's 68.9% blind should lower your confidence in the number, not raise your estimate of the model. This pairs directly with this week's expert re-grading of physics benchmarks — two independent audits, one month, and the pattern is clear: benchmark defects, not model failures, are where evaluation error concentrates right now.