Soft Latent Thinking Tops Pass@32 Without Token-Level Reasoning

Every large language model today decodes by projecting hidden states through a massive vocabulary head — a linear layer with one row per token in the vocabulary. This single operation dominates inference FLOPs and, more fundamentally, forces every reasoning step through a discrete bottleneck. An LLM cannot think in shades of meaning; it must commit to a specific token before it can see the next hidden state.

What if it didn't have to?

A Model with No Head and Many Thoughts (Koriagin et al., accepted to Findings of EMNLP 2026) introduces Soft Latent Thinking, a method that replaces the LM head during chain-of-thought reasoning with a lightweight continuous projector. The model autoregressively rolls out in embedding space — reasoning steps remain continuous vectors rather than being serialized into discrete tokens. The result: better pass@k at every k, lower per-step compute, and a proof that effective reasoning doesn't require tokenization at all.

The Problem: Token Decoding Is a Bottleneck

The vocabulary head is a computational and representational bottleneck. On a 7B-parameter model, the unembedding matrix can consume 15–25% of total FLOPs per forward pass. But the cost isn't just compute — it's representational. Every thought the model has must be serialized into a discrete token before the next thought can begin. This forces a quantization of reasoning: subtle distinctions collapse into the same token, and the model cannot revise a continuous thought without committing to a token first.

Previous "soft-thinking" approaches tried to address this by operating on soft tokens or interpolating between discrete steps, but they either required architectural modifications or failed to scale beyond trivial settings.

The Method: Continuous Rollout in Embedding Space

Soft Latent Thinking works by:

• Replacing the LM head during reasoning with a small MLP projector that maps hidden states to the next hidden state in embedding space
• Autoregressive rollout where each "thought" is a continuous vector, not a token — the model iterates in latent space
• Periodic discretization at the final output step: the continuous trajectory is projected back through the LM head to produce the final answer in tokens

flowchart LR
    subgraph Standard["Standard Token-Level Decoding"]
        H1[hidden state] --> VH[vocabulary head] --> T1[token t₁]
        T1 --> H2[hidden state] --> VH --> T2[token t₂]
        T2 --> H3[hidden state] --> VH --> T3[token t₃]
    end

    subgraph SLT["Soft Latent Thinking"]
        C1[hidden state] --> Proj[continuous projector] --> E1[embedding e₁]
        E1 --> C2[hidden state] --> Proj --> E2[embedding e₂]
        E2 --> C3[hidden state] --> Proj --> E3[embedding e₃]
        E3 --> VH2[vocabulary head] --> OUT[final tokens]
    end

The projector is tiny: a 2-layer MLP with a bottleneck dimension far smaller than the vocabulary size. During training, the model learns to produce useful intermediate embeddings that the final LM head can decode into correct answers. The reasoning path itself is never tokenized — only the final answer is.

Results: Better Pass@k, Less Compute

The authors evaluate on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B across reasoning benchmarks:

• Pass@k improves at every k tested — the gap widens at larger k, suggesting continuous-space reasoning produces more diverse and correct candidate trajectories
• Highest pass@32 among all soft-thinking approaches (including soft token methods and interpolation-based baselines)
• Per-step compute drops significantly — the projector is orders of magnitude smaller than the vocabulary head, so each reasoning step costs less
• Works on both small (1.5B) and medium (3B) scales, indicating the approach isn't limited by model size

The fact that pass@k improves across all k is the key signal. If only pass@1 improved, you might argue the projector simply biases toward common answers. But the across-the-board improvement — especially at higher k where diversity matters — suggests the continuous trajectory genuinely explores the reasoning space more effectively.

Limitations

The paper has honest caveats:

• Only evaluated on small models (≤3B parameters). It's unclear how the projector scales to 7B+ models where the vocabulary head's relative cost is lower
• Final output still needs tokenization — the continuous-to-discrete conversion at the end means some representational capacity is lost on the way out
• No ablation on projector architecture — the 2-layer MLP is tested but we don't know if a different design could close more of the gap with full token decoding
• Only commonsense and math reasoning tasks — open-ended generation, instruction following, and agentic tasks are not evaluated
• Training overhead from the projector and the continuous rollout may offset inference savings during training

Why This Matters

This paper is important for a reason that goes beyond the specific results: it breaks the assumption that reasoning must be tokenized.

If continuous-space reasoning works at scale, it changes the economics of inference-heavy applications. Every agent loop, every chain-of-thought system prompt, every multi-step reasoning call burns tokens — and token generation is the dominant cost. Cutting the per-step compute while improving pass rates is not just an efficiency win; it's a structural shift in how we think about LM reasoning.

For builders: if Soft Latent Thinking generalizes to larger models and broader tasks, it could reshape the cost model for agentic systems. The 1.5B model running continuous reasoning might match a 7B model running tokenized CoT while costing less in both FLOPs and latency.

Keep an eye on this thread — EMNLP 2026 is going to have strong opinions about it.