Serving Route Beats Model ID: 77.38 to 82.54 in Benchmarks

Every AI benchmark you've read this year — every leaderboard, every "X model scores Y" headline — is scoring the wrong thing. That's the finding of IBIB (arXiv:2609.10494), a measurement-protocol paper from Stenstrom, Vasantharajan, and Sathianathan published yesterday. The authors audited 18 benchmarks used to evaluate enterprise AI systems, and all 18 score an advertised model identifier. None score what an enterprise actually deploys: a serving route — the weights plus the serving infrastructure, precision, output contract, and harness that together determine what a user experiences. The paper treats this mismatch as measurement error and does something about it.

The problem: capability is a system property, not a checkpoint property

The paper's core observation is almost embarrassingly obvious once stated: usable capability depends jointly on weights, serving route, precision, output contract, and harness. Two deployments of "the same model" can behave differently because one runs at a different precision, uses a different tool-call parser, or wraps the model in a different harness generation. Every existing benchmark collapses that stack into a single label — the model name — and reports one number. That number silently attributes to the weights whatever the serving stack did.

The method: make the measurement error reportable

IBIB's answer is a three-part protocol called IB2. First, a gold-blind capability-binding preflight: before any scored task runs, the route must prove it can execute the evaluation contract — with evaluators blind to which system is which, so you can't game the gate. Second, a reliability-inclusive scoring rule: failed responses stay in the denominator. Third, structurally score-blind adjudication: the human or process that resolves ambiguous outputs never sees the scores. Crucially, the reference instantiation — 128 locked tasks, 987 assertions across document, spreadsheet, chart, tool, and database work — stays sealed. "The procedure is the artifact, not the corpus." That's a deliberate inversion of how benchmarks usually work, and it's the right call: a public corpus gets absorbed into training data, and your benchmark becomes pretraining trivia.

mindmap
  root((IB2 Protocol))
    Capability Binding Preflight
      Gold-blind
      Route must execute eval contract first
      Identical weights failed distinct predicates
    Reliability-Inclusive Scoring
      Failures stay in denominators
      Excluding them changes the ranking
    Score-Blind Adjudication
      Structural, not trust-based
    Sealed Task Corpus
      128 locked tasks
      987 assertions
      Procedure is the artifact

The results: the identifier hides real spread

Across eleven systems, four results stand out, and the third is the one that should change how you read every leaderboard:

• Availability is measurable. Two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate — while a third run passed. The advertised model identifier exposed none of this.
• Discrimination is not uniform. Four of seven suites saturate under a six-system band, and the spread comes almost entirely from governed database work and multi-tab joins. So IB2 reports interval-backed resolution groups, not ranks — and two of the nominal five-label output's four cuts fail multiplicity adjustment. Translation: some leaderboard ranks are statistically fiction.
• The serving arm moved a score by 5 points. The same declared model revision and precision measured 77.38 on one serving arm and 82.54 on another — a paired interval of [0.11, 10.60]. The arms differ in access mode, harness generation, and serving tool-call parser. No model-identifier-based benchmark could have detected this, because it would have called both runs the same number.
• Reliability inclusion changes conclusions. Excluding failed responses from denominators flips the point ordering of the systems. That means the common practice of reporting scores over successful attempts only isn't a rounding choice — it decides who wins.

The limitations, stated honestly

The paper flags its own confound on the 77.38→82.54 result: the two arms differ in access mode, harness generation, and serving parser simultaneously, so you can't attribute the 5-point gap to any single component — and it admits harness generation is a property of its own evaluator, not the endpoint. The paired interval [0.11, 10.60] is wide enough that the effect could be near-zero or enormous; the honest summary is "route matters, size unknown." The evaluation scope is enterprise document/database work, so extrapolation to frontier-reasoning benchmarks is untested. And there's a circularity worth watching: a protocol this opinionated about correct serving becomes a vendor's tool for arguing its own route is the "real" one. The sealed corpus mitigates gaming but also makes the protocol harder to independently replicate than an open benchmark would be.

Why builders should care

If you're choosing a model by leaderboard, you're reading a score attached to a name, not to the thing you'll deploy. The 77.38 vs 82.54 result means the serving route you pick — precision, parser, harness version — can be worth more than a model-generation upgrade. The practical IB2 takeaway: benchmark your route, not the checkpoint. Run the capability-binding preflight on your own stack before trusting any external number, and never let a "pass rate" exclude the failures — that exclusion is where the ranking flips.

Verdict: this is a protocols paper, not a model paper, and it lands the receipts angle perfectly — 18 of 18 audited benchmarks share the same measurement error, and the paper quantifies what that error costs (5.16 points on one pair, ordering changes on another). It also dovetails with what Dispatch has measured independently: harness variation swings agent behavior more than prompting does. The identifier was never the system.