AL Alfred 2026-09-01 4 min read

On-Policy Distillation Is a Myth: +35.41 AIME24

Here is a question that should unsettle anyone post-training an LLM: what if the teacher in on-policy distillation is a placebo? Purdue's Yi Ding and Ruqi Zhang ran the controlled experiment the field forgot to run (arXiv:2608.31046). The answer is uncomfortable. Teacher token-level supervision is wrong up to 50.6% of the time at scale, students improve identically whether that noisy supervision is kept, removed, or trained on alone, and a single constant negative advantage matches the teacher's signal. Then they ship the conclusion as a method: OPSA, teacher-free self-improvement that gains +35.41 Avg@32 on AIME24 (263% relative improvement) on a 1.7B model.

Why On-Policy Distillation's Teacher Is Suspect

On-policy distillation (OPD) is the trending alternative to reinforcement learning with verifiable rewards (RLVR): a larger teacher scores every token of the student's own rollouts and emits dense advantages for a KL-regularized update, promising token-level signal that sparse outcome rewards lack. The catch: those rollouts are sampled by the student, so they are off-policy for the teacher — it is grading text from a distribution it never produced. Whether that supervision even carries information was an open question.

Ding and Zhang quantified it. Defining noise as a negative advantage on a correct answer token, or a positive one on an incorrect token, a 4B teacher is wrong 30.6% of the time (20.4% of correct trajectories penalized on their answer tokens; 40.8% of incorrect ones rewarded). Bigger teachers are worse: 34.7% noise for a 30B-A3B teacher, 50.6% for a 235B-A22B teacher. The largest teacher even stamps negative advantages on 97.8% of correct boxed answers. Yet students converge to the same performance whether noisy trajectories are filtered out — or kept in, or trained on exclusively. The supervision content is noise, and the student doesn't care.

OPSA: Teacher-Free Entropy-Adaptive Training

So where do OPD's real gains come from? Token-level attribution shows learning concentrates on the student's low-log-probability tokens — high-probability tokens contribute nothing. And here is the kill shot: swapping teacher-provided advantages for a single fixed negative advantage on those low-probability tokens matches OPD's performance. OPD works by suppressing tail tokens. The teacher is doing none of the work.

OPSA (On-Policy Self-Adaptation) makes that mechanism explicit. At high-entropy positions — reasoning "forks" where the student is genuinely uncertain — it assigns stronger negative signal to the low-probability tail, pruning those branches, and evenly redistributes the mass across head tokens. The loss touches only the 20% of policy-sampled tokens with the lowest log-probabilities. No teacher, no verifiable reward, no hints, no reference answers: just entropy statistics on the student's own logits.

flowchart LR
    subgraph OPD["OPD (with teacher)"]
        A1[Student rollout] --> B1[Teacher scores off-policy tokens]
        B1 --> C1[30.6%-50.6% noisy advantages]
        C1 --> D1[Suppresses low-prob tokens anyway]
    end
    subgraph OPSA2["OPSA (no teacher)"]
        A2[Student rollout] --> B2[Entropy-adaptive negative advantage]
        B2 --> C2[Prune tail, rebalance head tokens]
        C2 --> D2[Sharper policy: +35.41 AIME24]
    end

Results: +35.41 AIME24, Doubled Pass@32

On Qwen3-1.7B trained on DAPO-17k (non-thinking mode), OPSA takes AIME24 Avg@32 from 13.44 to 48.85 (+35.41, +263%) and Pass@32 from 40 to 80 — a doubling. AIME25: 9.69 → 35.31 (+264%). HMMT25: 5.73 → 23.33 (+307%). It beats every labeled baseline on AIME24 Avg@32 — GRPO 33.96, OPSD 33.33, OPD 32.08, TTRL 19.90 — by up to +16.77 over OPD, and averaged across the three math benchmarks it beats the best RL baseline by +11.04 Avg@32 and +8.89 Pass@32.

It also scales with the model. Qwen3-4B jumps 23.33 → 62.08 on AIME24 Avg@32 (+166%); Qwen3.5-9B, already strong at 76.35, still gains +11.46; and thinking-mode Qwen3-1.7B improves 46.56 → 52.50. Out-of-domain gains are real but modest: MBPP+ +1.20, GPQA-Diamond +4.48.

Limitations

Experiments top out at 9B dense models — there is no evidence yet at frontier scale or on mixture-of-experts architectures. OPSA reshuffles the student's existing probability mass: heavily post-trained models with already-sharp, low-entropy distributions may have little left to gain, and it does not expand the exploration frontier (thinking-mode Pass@32 gains stay modest). The headline math gains are in-domain, and the numbers are one group's harness (slime) self-reported, though code and Hugging Face links make reproduction feasible.

Why Builders Should Care

Three takeaways. One: if you run any teacher-based distillation, run the control first — a fixed negative advantage on low-probability tokens. If it matches your teacher, you are paying (in teacher FLOPs and KL overhead) for a placebo. Two: OPSA is close to the cheapest post-training signal that exists — entropy statistics on the student's own logits — and the appendix shows it cold-starts GRPO, so it works as a warm-up before RLVR. Three: this is a generational warning about supervision quality: for OPD-style dense signals, bigger teachers moved the noise floor up, not down. Measure the signal before you trust it.

Related reading