There Is No Neutral Harness
What if every LLM leaderboard you've read this year is, in a literal statistical sense, manufactured?
A new preprint from V. S. Raghu Parupudi drops exactly that claim, and backs it with 114,608 correctness bits across 12 models, 3,679 items, and 26 harness configurations. The paper is titled "There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items" (2608.21382), and its central result is devastating: the harness selects the winner, not the model.
The problem
Multiple-choice benchmarks fix the questions and the correct answers — but they don't fix the evaluation harness. The order of options, the wording of the prompt, whether the answer is read from generated text or from per-option likelihoods: all of these are left up to the evaluator. Previous work noted that these choices cause aggregate score variance, but nobody had resolved the effect down to individual items. Parupudi does exactly that, introducing the fragility grid: a matched design where only the harness varies while items, weights, and greedy decoding stay frozen.
The method
Twelve open-weight instruction-tuned LLMs from four families (Gemma, Llama, Qwen, and Mistral) answer the same 3,679 items from four standard benchmarks — ARC, HellaSwag, MMLU, and TruthfulQA — under 26 equally defensible harness configurations. Every model-item-configuration triple is recorded as a single correctness bit. The result is a tensorial data structure that can be sliced any way: by model, by item, by configuration dimension.
mindmap
root((Fragility Grid))
12 models
Gemma4
Llama 3.x
Qwen 2.5
Mistral
26 harness configs
Scoring choice
Option order
Prompt template
Answer extraction
4 benchmarks
ARC
HellaSwag
MMLU
TruthfulQA
3,679 items
The results
The numbers speak for themselves:
- Score is a band, not a point. gemma4-31b scores between 31% and 89% depending only on the harness configuration. That's a ~58 point swing from changing prompt wording and whether you read log-probs or generated text.
- Four of the 12 models can reach rank #1 under some configuration. The harness doesn't just nudge scores — it determines who tops the chart.
- Config-fragile items carry 95.7% of the gap between adjacent models on the leaderboard. The items that separate one model from the next are precisely the ones that wobble under configuration changes.
- Item discrimination correlates with fragility at r = 0.28 (95% CI 0.25–0.30). Benchmark compression methods — which select the most "discriminating" items — actively retain the most configuration-sensitive ones, magnifying the problem.
- The scoring choice is the load-bearing axis. Whether you use log-probability of the correct token vs. greedy decoding of the generated text matters more than option order or prompt wording.
The limits
The study is scoped to 12 open-weight models and 4 standard benchmarks. It doesn't cover API-only models (GPT-4o, Claude), where the harness question is even harder because you can't access per-token log-probs. The identified "fragility items" depend on the particular models tested — a fragility grid for a different model set would likely surface different unstable items. And the paper doesn't propose a fix beyond transparency: publish a fragility grid alongside every leaderboard so readers can see the band, not just the point.
Why this matters
If you build or buy AI systems, this paper is a splash of cold water. Every leaderboard claim you see — "Model X beats Model Y by 2.3% on MMLU" — should now carry an asterisk: under one particular harness configuration, one of 26 equally valid ones. The implication isn't that benchmarks are useless. It's that they measure something far more entangled than model capability. They measure the joint distribution of the model and the evaluator's unstated assumptions.
For practitioners: stop optimizing for leaderboard position. For researchers: publish fragility grids, not single numbers. For everyone else: the next time someone tells you their model is "top of the leaderboard," ask which harness got it there.