QuoteBench: The Score That Lied

Every coding agent benchmark I've seen reports a single number: pass rate. The model issued commands, the commands ran, the task was completed — X%. Clean, comparable, actionable.

It's a lie. And QuoteBench (Li et al., 2608.13547) proves it with surgical precision.

What's Actually Broken

When an LLM coding agent generates a Bash command, that string passes through a serialization, wrapping, and re-parsing pipeline before the shell ever sees it. The model thinks it's writing ls -la. The shell receives something subtly different — unescaped quotes, mangled pipes, an extra layer of shell interpolation.

The paper isolates this problem with a clean experimental design. They take a set of 56 one-shot agent tasks derived from real incidents (14 families) and cross the generation contract (what the model outputs) with the execution transport (how the shell receives it). They introduce one deliberately unescaped parser boundary — a single quote added in transport — and replay the same model outputs through it.

The results are damning.

The Numbers

Across eight same-window model configurations, replaying the exact same model reply through the altered transport drops success rates by 55.4 to 73.2 percentage points. That's not the model getting worse. That's the same output failing because the boundary between generation and execution is hostile.

When the parser boundary is disclosed to the model (letting it adapt its generation strategy), six of eight configurations recover 30.4 to 60.7 points. The other two recover zero or even negative — meaning the model cannot adapt to the boundary at all.

The headline figure: GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The net score looks fine. The underlying reality is a catastrophe neatly cancelled out.

Architecture of the Deception

mindmap
  root((Agent Evaluation))
    What we report
      Matched Pass Rate
      Single Number
      Model Ranking
    What gets hidden
      Generation Contract
        Shell escaping
        Quoting rules
        Format spec
      Execution Transport
        Serialization layer
        Wrapper injection
        Re-parsing errors
      Operating Point
        Temperature
        Top-p
        Max tokens
    What matters
      Generation skill
      Boundary adaptation
      Final-state validation

Why This Matters for Builders

If you're building agents that execute shell commands — and most coding agents do — QuoteBench tells you three uncomfortable things:

1. Your eval numbers are inflated. The matched pass rate you're reporting is a composite of genuine generation skill and the accidental compatibility between your model and your transport. Change the transport, the number changes, but the model didn't.
2. Frontier saturation is an artifact. The paper shows raw generation is nearly saturated at the frontier. What separates models today is not how well they generate commands — it's how well they adapt to boundary conditions they shouldn't have to adapt to.
3. Model rankings are deployment-dependent. QuoteBench found one unambiguous rank reversal among 26 comparable model pairs, with four more sitting on single-task margins. The "best" model depends on which parser is in front of it.

Limitations

The benchmark is limited to 56 one-shot tasks. Real agent workflows involve multi-turn interaction, file state, and context accumulation — all of which could either amplify or mitigate the boundary problem. The paper also focuses on a single parser artifact (an unescaped quote boundary). Real deployment pipelines may have multiple compounding issues. And the recovery experiment (disclosing the boundary) assumes the model can change its generation strategy, which not all can.

The Bottom Line

QuoteBench's recommendation should be standard practice: every evaluation of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator. A matched score is not an intrinsic property of the model — it's a property of the whole pipeline.

We're benchmarking the wrong thing. Time to fix that.