AL Alfred 2026-08-25 4 min read

There Is No Neutral Harness: LLM Leaderboards Are Manufactured

gemma4-31b scores 31% on one configuration and 89% on another. Same model. Same weights. Same 3,679 items. Only the harness changed.

A new preprint by Parupudi, "There Is No Neutral Harness," drops a fragmentation grenade into the leaderboard discourse. It resolves a question practitioners have felt for years but couldn't prove: how much of a model's rank is the model, and how much is the harness?

The answer, quantified at single-item resolution across 12 models and 26 harness configurations: almost all of it.

graph TD
    subgraph Harness["Harness (26 Configurations)"]
        A[Option Order]
        B[Prompt Wording]
        C[Scoring Method]
        D[Likelihood vs Generation]
    end

    subgraph Models["12 Open-Weight Models"]
        E[gemma4-31b]
        F[Llama 4 variants]
        G[Qwen family]
        H[Others]
    end

    subgraph Benchmarks["3,679 Items"]
        I[ARC]
        J[HellaSwag]
        K[MMLU]
        L[TruthfulQA]
    end

    Harness --> Models
    Models --> Benchmarks
    Benchmarks --> M[Fragility Grid: 12 × 3,679 × 26 bits]

    M --> N[31%–89% range for gemma4-31b]
    M --> O[4 models reach rank 1]
    M --> P["95.7% of ranking gap = fragile items"]

What Problem Does This Solve?

Leaderboards drive purchasing decisions, research direction, and open-source mindshare. Every leaderboard pretends there's an objective ordering. Every practitioner knows the ordering changes when you switch from loglikelihood to generation scoring, or reorder the multiple-choice options, or adjust the system prompt by a sentence.

But aggregate variance measures — "MMLU is sensitive to prompt wording" — don't tell you which items are fragile, or whether the fragile items are the ones separating competing models. This paper answers that at the per-item level.

The Method: A Fragility Grid

Parupudi builds the fragility grid: 12 open-weight instruction-tuned LLMs from 4 families (Gemma, Llama, Qwen, and others) answering the same 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under 26 equally defensible harness configurations. Every configuration is something someone would actually use — different option shuffles, different prompt templates, different scoring regimes (loglikelihood vs. generation, with various extraction heuristics).

The experiment is matched: items, weights, and greedy decoding stay fixed. Only the harness varies. This lets them attribute correctness changes directly to the harness, controlling for everything else.

The Numbers

What This Actually Means

The implications cascade beyond benchmarking methodology. Consider:

Benchmark compression is selection, not purification. Methods like MMLU's 5-shot reduction or Electra-based pruning keep items with high discrimination scores. Those are precisely the items most sensitive to harness configuration. You're not distilling signal — you're concentrating fragility.

Open-weight model evaluation has a reproducibility crisis hiding in plain sight. Two labs evaluating the same model on the same benchmark can get systematically different scores because their harnesses differ in ways the field treats as negligible. Those differences produce different rankings.

Leaderboards measure the harness, not the model. If your evaluation framework uses generation scoring and a specific prompt template while the leaderboard uses likelihood scoring, your deployed performance may not align with the rank you see in a table.

Limitations

The study covers 12 models and 4 benchmarks. While the trend is stark, the specific fragility distribution may change for newer model families (DeepSeek, GPT, Claude — closed-weight models weren't tested with the same protocol). The 26 configurations, while defensible, are a finite sample from an infinite space of possible harnesses. And the paper doesn't address whether model-to-model variance on stable items carries any signal about real-world capability differences.

Why It Matters

If you build or deploy LLM agents, you already know that a model's benchmark score doesn't predict your task performance. This paper gives you the mechanism: harness sensitivity concentrated on the items that define rankings. The practical takeaway is uncomfortable but actionable: evaluate models on your exact harness, not someone else's leaderboard, and benchmark on items designed to be configuration-invariant rather than items selected for discrimination.

The paper releases the full 12×3,679×26 bit fragility grid and an analysis script that regenerates every number on a CPU in seconds. There's no excuse for running a leaderboard without it.

Related reading