Mercury 2.5 Benchmarked: 1,107 Tok/s Claim vs Reality

Inception Labs shipped Mercury 2.5 today — what it calls the largest diffusion LLM ever trained — claiming 1,107 tokens per second and parity with GPT-5.6 Luna (Low), Gemini 3.5 Flash-Lite, and Claude Haiku 4.5. Claims are cheap. I pointed the claim at the API and measured: 413–704 billed tok/s end-to-end through OpenRouter, up to 76% of output tokens are hidden reasoning, and every call that hit a length cap returned an empty string — billed in full. Here's the scoreboard.

What launched: diffusion goes production

Mercury 2.5 is a diffusion LLM — it generates token blocks in parallel instead of one token at a time. Today's release claims a 40% intelligence jump over Mercury 2, a 260K context window, tunable reasoning, parallel tool calls, and schema-aligned JSON. List price is $0.20/M input and $0.75/M output; launch pricing is 80% off at $0.04/$0.15. Inception says 1,107 tok/s "on widely-available NVIDIA GPUs," and OpenCall, a voice-agent company, reports P50 response latency under 0.2s in production. The headliner number, 1,107 tok/s, is a decode-rate claim measured on Inception's own infrastructure.

The benchmark: four models, same prompts, $0.04 total

I ran the same two workloads through OpenRouter's chat API against all four models Inception names as peers: a "write me a 600-word technical explainer of vector databases" long-form run, and a one-sentence factual answer for latency. Streaming, client-side timestamps, three to five runs per condition. Total spend, per OpenRouter's own billing: $0.04.

Long-form, finished runs only (finish_reason: stop), medians of 3 runs each:

On short answers the gap narrows but Mercury still leads: median 0.97s end-to-end over 14 runs, worst case 1.31s. Luna's median was 1.33s with a 6.4s outlier; Haiku's median 1.71s with an 11.2s outlier. If you're routing latency-sensitive traffic, Mercury's tail is the calmest I measured today.

Did the 1,107 tok/s claim survive contact with OpenRouter?

No — I got 37–64% of it. But the claim isn't fake; it's a different measurement. Inception counts decode rate on their own API. I measured everything a paying caller experiences through a proxy: queueing, TTFT, network. Even so, the direction of the result holds — on visible characters per second, Mercury at minimal reasoning effort delivered ~1,630 chars/s, roughly 2.3x Gemini 3.5 Flash-Lite (~720), 4x Luna (~400), and 4.7x Haiku (~345) on identical prompts. It is the fastest end-to-end text generator I've benchmarked through OpenRouter, and the cheapest by a mile: $0.00015–0.00018 per 600-word answer versus $0.0027 for Gemini.

You can also see the diffusion in the stream. Autoregressive models drip: Luna's answers arrived in ~1,300 SSE chunks, Haiku's in ~450, Gemini's in ~44. Mercury's arrived in 1–5 chunks — one run landed as blocks of 383, 457, 454, 9, and 32 characters. The parallel-block architecture isn't marketing; it's visible in the wire format.

Every capped call returned empty — and billed me anyway

Here's the part that should worry anyone wiring this in today. Across my session, all 22 runs where I recorded finish_reason: "length" came back null or a useless fragment, and all 24 runs that finished naturally ("stop") delivered content. Zero exceptions in either direction. One capped run billed 1,975 output tokens and returned 117 characters. The rule is brutal in its simplicity:

graph LR
    A["Request with max_tokens"] --> B{"Generation hits the cap?"}
    B -->|"No, finish=stop"| C["Content delivered"]
    B -->|"Yes, finish=length"| D["content: null"]
    D --> E["Tokens billed in full"]

Minimal repro — this returned empty content and charged $0.0003:

# model: inception/mercury-2.5 via OpenRouter, max_tokens=2048
# finish_reason: "length", completion_tokens: 1981, cost: 0.00029899
# choices[0].message.content: None   <-- 1,981 tokens, zero text

I can't attribute the fault cleanly between Inception's API and OpenRouter's translation layer from outside. Doesn't matter for you: through OpenRouter, never let a Mercury call cap. Set max_tokens generously above expected output and treat any finish_reason of "length" as a billed failure. Check the finish reason before you bill your users.

The $0.15/M price has a catch: hidden reasoning tokens

Mercury 2.5 reasons by default — medium effort — and bills the thinking as output. On a one-sentence answer I was charged 409 completion tokens for roughly 20 visible ones: completion_tokens_details.reasoning_tokens: 409. On the 600-word explainer, 1,728–2,478 of 2,391–3,268 billed tokens were hidden — up to 76% of what you pay for is invisible. That's why Mercury's "cheap" output price is really ~$0.55–0.75/M at default effort on long-form, and up to ~$0.75/M-equivalent on short answers where hidden tokens dominate.

The tunable knob is real, though. Docs list reasoning_effort: instant, low, medium (default), high. Through OpenRouter, "instant" gets rejected (their enum is minimal/low/medium/high/…), but minimal and low pass through and cut latency roughly in half: 4.5–7.9s down to 2.3–3.3s, and cost to $0.00015–0.00018 per answer. Even at minimal there's a floor of ~250 hidden tokens per call. Also: strict JSON schema failed 0-for-3 through OpenRouter, and every response starts with a stray "\n\n". Quality sanity checks — 3/3 on deterministic logic probes (trick question, 37×43, modular arithmetic) — suggest the intelligence claim isn't hollow for routine work, though one HN tester reports tool use and agentic coding lag the frontier.

Verdict: the speed is real, the edges are sharp

Mercury 2.5 is the first diffusion LLM I'd actually route traffic to: fastest end-to-end and ~15x cheaper per answer than its quality peers, with the steadiest latency tail. But ship it with three guards: never cap max_tokens at the model's expected output length, log and refund on finish_reason "length", and set reasoning_effort explicitly or you're donating three-quarters of your output bill to invisible deliberation. The 1,107 tok/s headline is a lab number; 400–700 billed tok/s through a proxy is the field number, and the field number is still enough to win. Diffusion LLMs just stopped being a research curiosity. They're a price-performance weapon with a loaded footgun in the API.