The TTS Bottleneck Isn't Compute — It's the Verifier

Test-time scaling (TTS) is the strategy of the moment. Generate more candidates, search harder, refine longer — and the model gets better. It works on math. It works on code. The industry has been treating this as a general law: more inference compute → better output.

Romano et al. ran the experiment nobody else ran. They controlled for compute across five TTS families (best-of-N, tree search, refinement, fusion, and self-consistency) on five open-ended benchmarks (medicine, law, finance, general chat, creative writing) — a domain set where verifiers are not built into the problem. The result reframes the entire TTS conversation.

graph TD
    subgraph "TTS Pipeline"
        A[Input prompt] --> B[Generate N candidates]
        B --> C[Select/refine/synthesize]
        C --> D[Final output]
    end
    subgraph "What Works"
        E[Best candidate in pool] --> F[Improves steadily with compute ✓]
        B --> E
    end
    subgraph "What Breaks"
        G[Reward model selection] --> H[ρ ≈ 0.12 correlation with quality]
        G --> I[Selection = near-random]
        J[Tree search] --> K[Diversity collapse]
        L[Refinement] --> M[Works on 1/5 benchmarks]
    end
    style F fill:#1e3a5f,stroke:#3b82f6
    style H fill:#7f1d1d,stroke:#ef4444
    style I fill:#7f1d1d,stroke:#ef4444
    style K fill:#7f1d1d,stroke:#ef4444
    style M fill:#7f1d1d,stroke:#ef4444

What the paper actually found

The authors decompose each TTS method's token budget into exploration (generating diverse candidates) and exploitation (converting the pool into a final answer). The decomposition is the key to everything:

• Exploration works. The best candidate in the pool improves steadily with compute across all five benchmarks. More candidates → better ceiling. No surprises.
• Exploitation fails. The step that converts that rich candidate pool into a single output is broken. Reward models — the gatekeepers of selection — correlate at only ρ ≈ 0.12 with true output quality. That's not a little broken. That's effectively random selection.
• Tree search amplifies the failure through diversity collapse — the search paths converge before they've explored the space, so the pool is narrower than it looks.
• Refinement (iterative improvement of a draft) helps on exactly one of five benchmarks. Its apparent gains elsewhere are confounded by the fact that the refinement model is also generating from scratch — the improvement comes from the fresh generation, not the refinement.
• Fusion (synthesizing across candidates) consistently beats single-sample baselines, but still recovers only ~40% of available quality. The ceiling is real.

Why this matters

The implication is uncomfortable for anyone deploying TTS in production: the candidate pool is not the bottleneck. Choosing from it is.

Most teams optimize the wrong thing. They add more candidates, widen the search, scale up generation — and quality barely moves. The paper explains why: you're pouring compute into the part of the pipeline that already works (exploration) while the part that's broken (exploitation) stays broken. If your reward model can't tell good from mediocre at ρ ≈ 0.12, no amount of candidate scaling will help. You're sampling from a good distribution but selecting from a noisy one.

The numbers

• Reward model correlation: ρ ≈ 0.12 with true quality across state-of-the-art generators. Statistically indistinguishable from zero in practice.
• Fusion ceiling: ~40% of available quality recovered even with optimal synthesis. The rest is lost to the verifier bottleneck.
• Refinement signal: Works on 1/5 benchmarks. Elsewhere the apparent gain is an artifact of the refinement model generating fresh output, not improving the draft.
• Scope: five TTS families (best-of-N, tree search, iterative refinement, self-consistency, fusion) × five open-ended benchmarks (medicine, law, finance, chat, creative writing) — domains where automatic verification isn't solved.

Limitations

The paper's scope is open-ended generation. On math and code — where verifiers are cheap and reliable (execution-based) — TTS works well, and the authors acknowledge this. The bottleneck is specifically about domains where quality is subjective, multi-dimensional, or hard to formalize. There's also a question of whether the fusion ceiling is a fundamental bound or a limitation of the particular synthesis methods tested. Finally, the compute-normalized comparison controls for token budget, but different TTS methods have different latency profiles — fusion may recover ~40% of available quality but be the most expensive method in wall-clock time.

What this means for builders

If you're deploying TTS today, this paper suggests a clear priority: fix your verifier before you scale your generator. The common instinct — throw more candidates at the problem — treats the wrong variable. The candidates are fine. The selection is broken.

For teams building agentic systems, the implication is sharper. If your agent generates and evaluates its own outputs (self-critique, self-consistency, iterative refinement), the verifier is the model. The ρ ≈ 0.12 gap is a best-case scenario with external reward models; with self-verification it's likely worse.

The paper's coda is worth quoting directly: "The candidate pool is not the bottleneck — choosing from it is." That's the sentence that should be on the wall of every team building inference-time scaling.