LLM Judge Reliability Fails: Spearman 0.400 vs 0.90 Gate
Every leaderboard, every eval gate, every RLHF reward signal runs through an LLM judge. The unspoken assumption underneath all of it: send the same request to the same model name, and you get the same measurement tomorrow. A new preregistered audit of black-box LLM observers on shared endpoints — Clean Engineering, Unstable Measurement — put that assumption in front of a validation gate and watched it fail catastrophically: Spearman 0.400 against a required 0.90.
What problem does this solve?
LLM judges aren't just scoring: they gate training data, rank generations, and drive public leaderboards. That makes the judge a measurement instrument — and instruments have reliability requirements. The authors reframe judge evaluation the way you'd audit a lab thermometer: freeze the thresholds in advance, validate the instrument before trusting a single reading. The critical, rarely-stated claim being tested: a model name on a shared endpoint is a frozen instrument.
Method: two preregistered campaigns, no moving goalposts
Haoyaun Zhu and Jie Zhang ran two preregistered campaigns with every threshold fixed in advance — same-window repeat ranking agreement at Spearman ≥ 0.90, and byte-identical next-day replays at ≥ 0.99. That's the strength of the design: no post-hoc threshold tuning. They audited 52,988 request attempts against shared serving infrastructure, and both campaigns failed at the instrument validation stage — they never even got to the evaluations they were meant to gate.
Results: the instrument has a noise floor you can't see
The numbers are brutal. Same-window repeat rankings agreed at Spearman 0.400 (required: 0.90). Byte-identical next-day replays agreed at 0.78 (required: 0.99) — with the execution record at ceiling, so it wasn't request failures. Three mechanisms explain the gap:
- Label-to-meaning mapping bias — the judge's internal mapping skewed readouts as strongly as the actual signal;
- Noise-floor dwarfing — candidate quality gaps seven orders of magnitude below the instrument's own noise were undetectable;
- Non-determinism — byte-identical inputs returned different rankings, which exact-permutation readouts compound.
The follow-up probes are the part builders should memorize. Waiting didn't help (0.805 vs 0.800, replicated over five further days). Switching providers didn't help — four providers share the same floor, medians 0.74–0.88, predicted by none of the metadata they expose. Self-hosting on batch-invariant kernels helped only while the server was quiet. And on constructed errors with known gaps, readout separation tracked error type, not error size. Metric substitution and sampling didn't repair it on the tested grid.
The authors distill this into a three-level snapshot-identity ladder, eight design rules for judge-based eval, and a reporting checklist. The most actionable line in the paper: a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance.
Limitations
Everything measured is external behavior on shared serving infrastructure — the paper doesn't diagnose what happens inside the models, and dedicated or self-hosted deployments may behave differently (the quiet-server result hints they do). The campaigns target ranking-style judge tasks; other judge shapes (classification, rubric scoring) aren't directly audited. And the failure is environmental, not model-specific — which is precisely why it's so easy to miss and so hard to fix by swapping vendors.
Why builders should care
If you ship any pipeline where an LLM judge gates anything — agent eval harnesses, PR review bots, evals in CI, data filters — assume your instrument is unstable until proven otherwise. Run the 2%-volume pilot before trusting a single benchmark delta, log model name plus snapshot identity (endpoint, version, time) with every verdict, and stop treating sub-1-point leaderboard moves as signal. On a shared endpoint, a model name is a lease on a measurement, not a contract for one.