Physics Benchmarks Understate GPT-5.6: 47.3% to 78.7%

Somewhere along the way, "frontier models can't do physics" became a citation-grade fact. It shows up in the Artificial Analysis Intelligence Index (2026), it shows up in leaderboards, it shows up in procurement decks. A team of 25+ physics faculty and grad researchers across Yale, and other institutions — people who run the actual benchmarks — just audited the receipts, and the receipts don't hold. Most cases scored as model failures were grader failures.

The problem: who grades the grader?

The paper (arXiv:2609.13009) starts from a tension practitioners feel daily: low reported physics scores on leaderboards versus physicists' lived experience that frontier models are genuinely useful. The gap could mean models are overhyped in practice — or that the benchmarks are wrong. Third-party indices aggregate these scores, so a broken benchmark propagates everywhere, silently. This is the same failure class Dispatch has covered in evaluation before (LLM judges failing a 0.90 Spearman gate), but with a twist: here the grader is a static reference solution, not a model.

The method: expert adjudication, not re-scoring

The team took six widely used physics benchmarks — HLE-Physics, CMT-Benchmark, CritPt, UGPhysics, PRISM-Physics, and PHYBench — restricted to text-only problems with verifiable final answers, and had subfield experts review three things for every disputed item: the problem statement, the reference solution, and the model's response. The output is a four-way classification: genuine model error, grader error, incorrect reference solution, or ambiguous/underspecified question. Then experts fixed the benchmark: correcting wrong reference solutions and dropping unanswerable questions. Scores were recomputed on the retained subsets.

The results: 31.4 points of non-model error

For GPT-5.6-Sol, mean@4 rises from 47.3% to 78.7% on HLE-Physics (+31.4 points) and from 61.0% to 87.2% on CMT-Benchmark (+26.2). Corrected pass@4 hits 94.4% on the 54 retained CritPt challenges. UGPhysics, PRISM-Physics, and PHYBench rise substantially too. The authors' verdict: these benchmarks were near-saturation all along — the models were better than the measurement.

mindmap
  root(("HLE-Physics
GPT-5.6-Sol mean@4"))
    As reported
      47.3%
    After expert audit
      78.7%
    Sources of error
      Grader errors
      Wrong reference solutions
      Ambiguous questions
      Genuine model errors
            

The number that matters most isn't in the abstract's headline: the failure mass decomposes into benchmarking issues versus genuine model errors, and the paper reports that most audited "incorrect" cases were the former. A 31-point swing on a leaderboard-grade benchmark is not noise; it's a systematic measurement defect that every downstream consumer — including the indices that price models — inherited uncritically.

Limitations the authors underplay

• Selection on retention. Corrected scores are computed on "retained" subsets after experts excluded flawed questions. Exclusion is defensible item-by-item, but it means the 78.7% is a conditional score on a filtered benchmark — you cannot directly compare it to the 47.3% full-set number without an audit trail of what was dropped and why.
• Text-only, verifiable answers. This deliberately excludes multi-part derivations, figure-dependent problems, and experimental design — the hard parts of real physics. "Near-saturation" applies to closed-ended tasks only.
• Experts correcting reference solutions is also a human-judgment process. The paper validates models against expert judgment; it does not (and can't) validate the corrected key itself at scale. The symmetrical risk — experts too charitable to fluent model answers — isn't quantified.
• One model in the headline. The audit covers frontier models broadly, but the splashiest numbers are GPT-5.6-Sol's. Weaker models may genuinely fail more often, which would make the benchmark look worse than it is.

Why builders should care

If you're choosing models off a leaderboard that includes physics or science reasoning, treat any score as an upper bound on the benchmark's validity, not the model's. The actionable shift: when your own domain evaluations show your team's real experience diverging from published scores, the divergence is now more likely to be the evaluation's fault. Budget for expert audits of your evals the way this team audited these — one round of human adjudication recovered 31 points of signal that was sitting there all along. And for anyone building agentic science tools: the bottleneck on verifiable physics tasks is no longer model capability, it's answer key quality.