Here's the standard approach to reasoning models: you give them a fixed token budget, they think for that many tokens, and you pay for every single one regardless of whether the problem needed it. Easy problem? Full chain of thought. Trivial lookup? Full chain of thought. "What's 2+2?" — the model writes a paragraph about addition before answering.
A paper that dropped on arXiv yesterday — "Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation" by Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet — fixes this with an approach so clean it's surprising nobody published it exactly this way before.
The Problem With Fixed Reasoning Budgets
Reasoning language models trained with reinforcement learning (like DeepSeek-R1, QwQ, or any of the chain-of-thought fine-tunes) almost always operate under a fixed token budget. The model is trained to produce a certain number of reasoning tokens per response, and it does — regardless of whether the question is "what is the capital of France" or "prove the Riemann hypothesis by induction."
This isn't just inefficient. It's structurally wasteful. The model over-computes on easy problems (wasting inference dollars) and under-computes on hard ones (producing shallow reasoning when depth is needed).
The Method: A Learned Router, No Extra System
The paper's approach is elegant: make the reasoning effort a learned decision rather than a fixed hyperparameter. The model outputs, as its very first token, one of three modes:
- NoThink — answer immediately, no chain of thought. For trivial lookups and direct retrievals.
- Short — brief reasoning (capped at ~128 tokens). For moderate difficulty.
- Long — extended reasoning (capped at ~3,000 tokens). For hard problems requiring multi-step logic.
graph LR
subgraph "Input Problem"
Q[Query]
end
subgraph "First Token Decides"
Q --> C{Learned Router
One GRPO step}
C -->|NoThink| NT["Answer immediately
< 50 tokens"]
C -->|Short| ST["Brief reasoning
~128 token cap"]
C -->|Long| LT["Extended reasoning
~3,000 token cap"]
end
subgraph "Outcome"
NT --> R["Accuracy: 0.782
Mean: 2,811 tokens
41% savings"]
ST --> R
LT --> R
end
The key: this choice is learned inside the Group Relative Policy Optimization (GRPO) training loop with no separate router module. They use a shaped reward that makes each mode worthwhile at a different response length, combined with hard per-mode token caps that keep the modes distinct. No external classifier, no separate system — the model itself learns to budget its own cognition.
The Numbers
Trained on a 1.5B parameter distilled model on MATH, the three modes emerge without collapsing to a single choice. The results are concrete:
- Accuracy: 0.782 vs baseline 0.796 on held-out MATH500. A 1.8% drop in accuracy for a 41% reduction in tokens. In most production settings, that tradeoff is worth taking.
- Mean response length dropped from 4,796 to 2,811 tokens — a 41% reduction. Your inference bill just got cut nearly in half.
- 76% token reduction on GSM8K — with higher accuracy than fixed-budget baselines at the same response length. The model correctly identifies easy grade-school math problems and skips the theatrics.
- Transfer without retraining. The policy generalizes to other benchmarks. The biggest savings occur where problems are easiest — exactly where you want savings.
The brief modes end up more accurate than Long mode on the problems they choose to handle, which means the router isn't random: it's actually sorting problems by difficulty. Short and NoThink modes get activated for problems the model is confident about, and it turns out confidence correlates with correctness.
Limitations
The 1.5B parameter scale is a limitation — results on frontier-sized models (70B+) may differ, though the mechanism should transfer. The three-mode taxonomy (NoThink/Short/Long) is arbitrary; the optimal split probably varies by domain and user preference. The paper also doesn't address the latency penalty of the routing decision itself (the model still generates one token to decide, though this is negligible compared to saved reasoning length). And the hard token caps mean the model can't dynamically extend reasoning when it's close to solving a hard problem — it hits the Long cap and stops, even if a few more tokens would close the gap.
Why Builders Should Care
This is one of those papers where the idea is so straightforward you'll wonder why you weren't already doing it.
Inference cost is the bottleneck for reasoning models. The most capable models (OpenAI's o1/o3, DeepSeek-R1, Gemini Thinking) are expensive precisely because they burn tokens on reasoning. A 41% reduction at near-identical accuracy is not incremental — it's transformative for any application that routes to thinking models at scale.
The approach requires no architectural changes. You don't need a separate router, classifier, or orchestrator. The decision is learned as part of the existing training loop. If you're already fine-tuning a reasoning model with RL, adding this is a reward shaping change, not a system redesign.
It works out of the box on new domains. The transfer result (trained on MATH, works on GSM8K without retraining) suggests the router learns a meta-skill: estimate problem difficulty and budget accordingly. This is the kind of capability that makes a model smarter about its own limits, which is a step toward models that know what they don't know.
Three modes is enough. The paper shows you don't need a continuous spectrum of reasoning effort. Discrete buckets (none, some, lots) capture most of the benefit. This simplifies both training and deployment: you can predict and control the latency distribution of your model's responses.
If you're running reasoning models in production — or building the next generation of them — this paper belongs on your shortlist. The 41% savings alone justifies the read.
- Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation — Kassenaar, Yang, François-Lavet, Aug 2026