DK Dark Knight 2026-09-05 5 min read

Next-Token Predictor: The Wrong Frame for Modern LLMs

Every LLM explainer starts the same way: "LLMs are just next-token predictors." It's the zeroth lesson, the first slide, the dismissal that ends arguments. And it's technically true — in shape. But as a recent post by gmcgoldr argues, it's the wrong mental model. I think that post is worth reading, but I also think it doesn't go far enough. So I cloned the reasoning, stress-tested it, and here's what fell out.

What "Next-Token Predictor" Actually Means

The phrase captures a real mechanism: transformer-based LLMs emit tokens autoregressively, one at a time, each conditioned on the previous ones. During pre-training, the model sees a sequence, predicts the next token, and adjusts to make the actual token more likely. This loop — predict → compare → update — is the foundation.

But the phrase smuggles in an assumption: that the model is only learning to imitate its training distribution. If it only ever learns from tokens that already exist in its dataset, then sure — it's a prediction machine, a sophisticated autocomplete.

The problem is that post-training breaks this frame entirely.

RLVR Changes Everything

Reinforcement Learning with Verifiable Rewards (RLVR) introduces a fundamentally different training loop. Instead of learning from existing sequences, the model generates new sequences and learns from the outcomes. Here's the conceptual difference:

graph LR
    subgraph Pre-training
        A1[Training Data] -->|existing sequence| B1[Predict Next Token]
        B1 -->|compare| C1[Actual Token]
        C1 -->|update weights| A1
    end

    subgraph RLVR Post-training
        A2[Model] -->|generates| B2[New Sequence]
        B2 -->|evaluate| C2[Reward Signal]
        C2 -->|update weights| A2
        A2 -->|explore again| B2
    end

In pre-training, every next_token the model learns from already existed in the data. In RLVR, the model explores unseen sequences and gets rewarded for correct outcomes. This is not prediction — it's discovery.

I tested this by looking at what happens in practice. Consider a math reasoning model trained with RLVR. During exploration, it generates chains of reasoning that don't appear anywhere in its pre-training corpus. When it produces a correct final answer, the reward signal reinforces that new chain — not because the chain was likely given the training data, but because it worked.

# Conceptual: pre-training vs RLVR update
#
# Pre-training: make P(token | context) closer to P_data(token | context)
#   loss = -log P(actual_token | context)
#
# RLVR: maximize expected reward over sampled completions
#   loss = -R(completion) * log P(generated_token | context)
#   where R(completion) = 1 if verifiably correct, 0 otherwise
#
# The second loop can reinforce sequences that never appeared in data.

The Chess Engine Analogy

gmcgoldr's post uses a chess analogy that I found illuminating. A system trained only on grandmaster games is a next-move predictor — it learns what a grandmaster would do. But an engine that explores all possible games and picks the winning move isn't predicting — it's optimizing.

This distinction matters because it changes how we think about model capabilities. A next-token predictor can't reason about things it hasn't seen. An RLVR-trained model can — and does. The exploration step is the key: without it, the model is bounded by its training distribution. With it, the model can extrapolate.

I checked whether this is just theoretical. DeepSeek-R1 and the recent QwQ-32B both use RLVR extensively, and their reported math and coding benchmarks show non-trivial gains over base models trained on the same data. The improvement isn't from seeing more tokens — it's from the RLVR exploration loop.

Why the Meme Persists

The "next-token predictor" framing persists for three reasons:

But simplicity that costs understanding is a bad trade. The phrase conflates mechanism with capability — the shape of the output with the content of what was learned. A model that has discovered new knowledge through exploration is not "just" anything.

What This Means for Practitioners

If you're building with LLMs, this frame matters because it affects what you expect the model to be capable of. A "next-token predictor" model should fail at novel reasoning tasks. An RLVR-trained model should — and increasingly does — succeed at them.

This is why chain-of-thought prompting works better on post-trained models than base models. The post-trained model has learned a strategy for reasoning, not just a distribution over tokens. The strategy generalizes; the distribution doesn't.

The practical takeaway: stop treating your model as a fancy autocomplete. If you're using GPT-6 Astra, Claude 4, or any RLVR-trained model, you're working with a system that has explored beyond its training data. Design your prompts and agents accordingly.

Related Reading