AL Alfred 2026-09-01 4 min read

RLVR Solution Collapse: 67% Lost at First Step, Not During Reasoning

Reinforcement learning with verifiable rewards (RLVR) has become the default method for sharpening LLM reasoning. Pass@1 scores go up, chain-of-thought traces get longer, and the field celebrates. But there's a cost that shows up only when you look under the hood: the model's solution space — the set of distinct ways it can reach an answer — contracts dramatically.

A new paper from Zhou and Li (arXiv:2608.29188) pinpoints exactly where this contraction happens. Spoiler: it's not where most people would look.

What Problem Does This Solve?

RLVR's most visible limitation is that it makes test-time scaling less effective. As the policy sharpens, it puts probability mass on fewer reasoning paths, so the returns of majority voting or beam search diminish. Prior work documented this effect but treated it as a monolithic property of the trained policy — "RLVR kills diversity, full stop."

This paper decomposes the problem. Is the policy failing to initiate alternative solution strategies, or is it failing to execute them once started? The distinction matters because the remedies are different: early versus late intervention in the reasoning trace.

How They Isolated the Signal

The authors use the Countdown task, a mathematical puzzle where the model must combine a set of numbers with arithmetic operators to reach a target. The key design choice is exhaustive enumerability: every valid solution path can be pre-computed, and each path is uniquely identified by its first operand-and-operator pair — the entrance family. This lets the authors track exactly which families the model can access versus execute.

They train two setups — PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct — and measure per-token likelihood shifts across the reasoning trace with family-level granularity.

mindmap
  root((RLVR Diversity Collapse))
    Entrance
      First operator choice
      11x-16x likelihood drop
      Cannot initiate alternatives
    Execution
      Downstream reasoning intact
      Supplied prefix completes 0.018->0.212
      Can finish what it doesn't start
    Remedy
      Late-layer interpolation
      +37% coverage, 0% accuracy loss
      Staged SFT->DPO->RLVR preserves entropy

Results: The Numbers That Matter

Three findings stand out:

1. The collapse is at the entrance. Solution coverage falls by up to 67% under RLVR, halving even on problems the model solves across all training checkpoints. But per-token likelihood shifts are 11x–16x larger before the first arithmetic operation than during downstream reasoning. The model stops thinking of alternatives, not thinking through them.

2. The execution machinery is intact. When the authors supply an unselected entrance prefix (essentially whispering "start with 12 × 3" instead of "12 + 5"), completion rates in low-access families jump from 0.018 to 0.212 under PPO — over an order of magnitude recovery. The model can finish paths it could not initiate on its own.

3. Targeted intervention works. Late-layer parameter interpolation between the RLVR checkpoint and an earlier checkpoint increases solution coverage by 37% with zero loss in pass@1. Surface-level prompting fails to recover diversity, but the right weight-space maneuver succeeds cleanly.

The phenomenon replicates across six math benchmarks with 7B and 14B models. Crucially, it's not an inevitable byproduct of reasoning optimization: a pure SFT baseline preserves more than double the solution coverage, and staged SFT→DPO→RLVR pipelines retain early-step entropy throughout.

Limitations

The analysis is grounded in Countdown, a domain whose solution space is small enough for exhaustive enumeration. Whether the entrance-locking mechanism transfers to open-ended reasoning tasks, code generation, or long-horizon planning remains unestablished. The intervention (late-layer interpolation) is also demonstrated at relatively small model scales (3B–14B). The paper itself notes that surface prompting fails to recover diversity, suggesting the problem operates at the level of learned policy geometry rather than token-level instruction following.

Why Should Builders Care?

If you're training reasoning models with RLVR, this paper gives you a specific diagnosis and a concrete fix. The diagnosis: check whether your model's diversity collapse is concentrated in the first few tokens (it likely is). The fix: late-layer interpolation with an earlier checkpoint recovers coverage without hurting accuracy — and it costs nothing to try.

More broadly, the result suggests that the training recipe matters as much as the training algorithm. SFT-first pipelines retain solution diversity; direct RLVR on a randomly initialized policy does not. Anyone building for robustness — where multiple reasoning paths to the same answer signal reliability — should treat entrance diversity as a first-class metric alongside pass@1.

The paper's headline result — "reasoning breadth is lost at the door, not inside the room" — is the kind of crisp finding that reframes how we think about a problem. Worth reading the full preprint.

Related Reading