RL for LLMs is having its reckoning. GRPO exploded onto the scene because it ditched the critic — the separate value network that made PPO memory-heavy and finicky to train. But GRPO bought that memory efficiency at a cost: it needs multiple rollouts per prompt to estimate advantages. Multiple rollouts means multiple forward passes, which means more compute, and the advantage signal weakens in long-horizon tasks.
SAPO (Liang et al., arXiv 2608.19842) asks: what if you could have both — the memory efficiency of critic-free methods and the sample efficiency of PPO? The answer is both simpler and more clever than I expected.
The Clever Trick
The core insight is hiding in plain sight. An autoregressive LLM already produces a sequence of token-level hidden states. Some of those tokens are "reasoning" tokens; some are "value" tokens; some are "action" tokens. The standard approach is to train a separate critic network — doubling the memory footprint — because the policy and value functions are supposed to be different.
SAPO says: they're not that different. Both the policy and value function operate on the same hidden representations. So why not produce both from the same backbone, at different token positions? The autoregressive structure gives you the separation for free — produce the action distribution at the action token boundary, and produce the value estimate at a causally distinct prior position in the same sequence.
mindmap
root((SAPO Architecture))
Shared Backbone
Single LLM
No separate critic
Token-level separation
Policy Head
Action distribution at action boundary
Standard PPO objective
Value Head
Value estimate at prior position
SARSA objective
Advantage
Lambda-returns
Batch normalization
Single rollout only
This isn't a trivial change. The paper demonstrates that a single autoregressive backbone can serve dual purposes without interference — the key is establishing distinct causal boundaries for the policy and value outputs within the same sequence generation.
The Numbers
SAPO was tested on Qwen2.5-1.5B and Qwen2.5-7B across ALFWorld (household task completion) and WebShop (online shopping). Against PPO and GRPO:
- +15.1 points mean improvement over PPO across both models and both benchmarks
- +12.1 points mean improvement over GRPO — the method everyone's been excited about
- 33.2% faster per-iteration runtime vs PPO (no critic to forward-pass)
- Stable training — no advantage collapse on long-horizon tasks
The stability result matters more than the raw performance lift. GRPO's multiple-rollout strategy dilutes the advantage signal on long trajectories, creating a pernicious failure mode where the model stops learning halfway through a complex task. SAPO's single-rollout design with trajectory-level generalized advantage estimation (GAE) + batch normalization keeps the signal clean across the entire trajectory.
What This Means for the Post-Training Landscape
The RL-for-LLM training stack has been stabilizing into a few recognizable patterns: PPO (reliable but heavy), GRPO/DAPO (light but wasteful with rollouts), REINFORCE variants (simple but high variance). SAPO carves a new niche.
The autoregressive sharing trick is the kind of innovation that looks obvious in retrospect — much like how GRPO's "drop the critic" insight now seems inevitable. But implementing it correctly requires careful architectural design: the policy and value outputs must operate at causally distinct token positions, sharing parameters through the backbone but optimizing independently through their respective objective functions (PPO + auxiliary SARSA). Get that wrong and you get interference. Get it right and you erase the memory overhead of the critic without multiplying your rollout budget.
I suspect this pattern will generalize. Any architecture that generates token sequences autoregressively — which is essentially all production LLMs — can layer in a shared value head at virtually zero incremental memory cost. The question is whether it works at scale (70B+ models) and on longer-horizon tasks than ALFWorld. That's the next experiment I want to see.
Limitations
Two stand out. First, the experiments are on Qwen2.5-1.5B/7B — solid models, but a long way from the 70B/405B scale where memory optimization really matters. The computational savings compound with model size, so the results could be even better at scale, or the shared backbone could introduce interference we don't see at 7B.
Second, the evaluation is on interactive benchmarks (ALFWorld, WebShop), not on the pure reasoning tasks (math, code) where GRPO has shown the strongest gains. SAPO's trajectory-level GAE is designed for interactive settings where actions have long-range consequences. Whether it translates to chain-of-thought reasoning optimization is open.
Why Builders Should Care
If you're training or fine-tuning LLMs with RL — and the industry is rapidly converging on this being the default post-training paradigm — the compute-to-quality ratio of your training pipeline is now a product differentiator. SAPO offers a genuine pareto improvement: better results, less memory, faster iteration. Three things that don't usually come in the same package.
The single-rollout design is particularly relevant for anyone doing multi-turn agent RL, where the rollout cost scales with the number of interaction rounds. GRPO's multiple-rollout budget multiplies the cost per training step; SAPO doesn't. For agentic post-training pipelines, that's a decisive advantage.
- SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning — Dayang Liang, Lang Feng, Bo An, Yunlong Liu (Aug 2026)
mindmap
root((Training Method Comparison))
PPO
+ Value function (critic)
+ Sample efficient
- High memory
- Complex tuning
GRPO
- No value function
+ Memory efficient
- Multiple rollouts
- Advantage collapse
SAPO (this)
+ Shared value-head in backbone
+ Single rollout
+ Memory efficient
+ Sample efficient
- Not tested at 70B+ yet