Sliding Through Thought: Efficient Test-Time Scaling with Prefix Sliding

Q: The Problem

Current models keep their entire reasoning trace in the KV cache. For a model reasoning through a 50K-token problem, this means 50K tokens of cached state, growing quadratically with attention. The economic ceiling hits fast: hard reasoning problems

Q: The Insight

The paper's core finding is deceptively simple: most intermediate reasoning tokens lose importance as the model continues reasoning. The model's early meandering — the first attempt that hit a dead end, the sub-step it backtracked from — contributes

The promise of test-time scaling is simple: give a model more time to think, and it solves harder problems. But every reasoning token a model generates stays in memory via full attention — and the quadratic cost means long chains of thought are prohibitively expensive. A paper that drops today from an all-star team at Stanford, UW, and AI labs — Muennighoff, Liang, Ng, Zettlemoyer, Choi, and more — flips this assumption on its head.

The Problem

Current models keep their entire reasoning trace in the KV cache. For a model reasoning through a 50K-token problem, this means 50K tokens of cached state, growing quadratically with attention. The economic ceiling hits fast: hard reasoning problems become too expensive to deploy at scale. Practitioners resort to prompt engineering tricks (like asking the model to "think step by step" in limited windows) rather than letting models reason as long as they actually need.

The Insight

The paper's core finding is deceptively simple: most intermediate reasoning tokens lose importance as the model continues reasoning. The model's early meandering — the first attempt that hit a dead end, the sub-step it backtracked from — contributes almost nothing to the final answer once the model has moved past it. Retaining those tokens is a waste. Only two parts of the context actually matter at any given point:

The prefix — the system prompt, instructions, and tools available to the model.
The sliding window — the last few thousand tokens the model is currently working on.

flowchart LR
    A[Prompt + Instructions] --> B[Prefix
Always present]
    B --> C[Sliding Window
Last ~4K tokens]
    C --> D[Discarded
Intermediate reasoning]
    style D stroke:#ef4444,stroke-width:1px,stroke-dasharray: 5 5
    style B fill:#1e3a5f,stroke:#2563eb
    style C fill:#1e3a5f,stroke:#2563eb

The Method: Prefix Sliding

Prefix Sliding discards every token that isn't part of the prefix or the most recent window. The total memory requirement is capped at a fixed size, regardless of how long the model reasons. No training required to apply it to existing models — it's purely an inference-time change to how the KV cache is managed.

When paired with reinforcement learning training, the authors show that models can learn to reason efficiently under this constraint, producing traces that exceed 100K tokens while keeping memory flat. The model learns to "write to be forgotten" — producing intermediate reasoning that's useful in the moment but doesn't need to persist.

Results

Without training: Prefix Sliding makes existing models 3x faster while maintaining the same performance across reasoning benchmarks.
With RL training: Models achieve strictly better performance than full-attention baselines by scaling to longer reasoning traces (100K+ tokens) that would otherwise be infeasible.
Ablations: Prefix Sliding outperforms both summarizing intermediate tokens and vanilla sliding window approaches — the model's own raw tokens in the window are more useful than any compressed summary.

flowchart LR
    subgraph Full Attention
        A1[1K tokens] --> A2[10K tokens] --> A3[50K tokens]
        A3 -.-> |OOM| A4[💥]
    end
    subgraph Prefix Sliding
        B1[1K tokens] --> B2[10K tokens] --> B3[capped]
        B3 -->|flat memory| B4[100K+ tokens ✓]
    end
    style A4 fill:#7f1d1d,stroke:#ef4444
    style B4 fill:#14532d,stroke:#22c55e
    style B2 stroke:#ef4444,stroke-dasharray: 5 5

Limitations

The paper is honest about what Prefix Sliding doesn't do. It assumes the model's reasoning is roughly monotonic — that early tokens genuinely lose relevance. For problems requiring repeated reference to earlier reasoning (complex multi-step proofs, long-horizon planning with backtracking), discarding context could harm performance. The RL training also adds engineering complexity, and the 3x speedup only holds for models already generating long reasoning traces — short CoT models see less benefit. Additionally, the approach was validated on math and coding benchmarks; it's less clear how it generalizes to open-ended creative reasoning or agentic loops where earlier context is constantly re-accessed.

Why It Matters

This is one of those rare papers that targets a fundamental scaling bottleneck with a simple, deployable fix. If you're shipping a reasoning model today, the bottleneck isn't model quality — it's inference cost at long reasoning lengths. Prefix Sliding directly attacks that. No retraining required for existing models, no architectural changes, and the RL-trained version opens the door to models that can reason for minutes of token generation at near-constant memory cost. For anyone building on o1-class models, this is the kind of optimization that changes your pricing model.

The code is open source. Try it on your own models.