On-Policy Distillation: One Example Beats 17,000

If you post-train with on-policy distillation, this number should hurt: a single training query recovers 87% of what the full 17,000-query run delivers. And sixteen well-chosen queries reach 98.9% of the full run's training states and match its accuracy outright. The paper is Rethinking On-Policy Distillation of Large Language Models II: One Training Example (arXiv:2609.04172), and it closes the loop our September 1 post left open: if the teacher's token-level labels are mostly noise, what is the data even doing? Answer — far less than anyone assumed.

The Problem: Nobody Asked What the Queries Were For

On-policy distillation (OPD) pairs student-generated rollouts with dense token-level supervision from a teacher, and frontier labs run it at scale — the paper lists Qwen3, MiMo, GLM-5, DeepSeek-V4, and Kimi K3 as OPD users alongside SFT and RL. The default assumption is that query volume matters: collect more problems, get more student. Part I of this series showed the teacher's supervision is wrong up to 50.6% of the time, yet students improve identically whether you keep it, drop it, or train on the noise alone. That left the query side standing unexamined. Part II takes it to the data-minimal limit: train on one query and see what breaks.

The Method: One Query and a State-Coverage Yardstick

The pipeline is standard OPD — the student generates rollouts, the teacher supervises every token of every visited prefix — minus the dataset. Students are 1.5B-class models across families (DeepSeek-R1-Distill-Qwen-1.5B, Llama-3.2-3B, OLMo-7B-DPO, Qwen2.5-Coder-1.5B), each paired with a post-trained teacher from its own family. To explain the result, the authors introduce state coverage: embed the prefixes full-data OPD visits, cluster them semantically, and report the fraction your query set's rollouts actually reach.

flowchart LR
    A[One query] --> B[Student rollouts]
    B --> C["States visited: 71.5% of full-data space"]
    C --> D[Teacher supervises every prefix]
    D --> E[Student update: absorption rate falls each step]
    E --> B

The Results: 71.5% Coverage From One Query, 98.9% From 16

• One query, 87% of the gain. On math (MATH-500, AMC 2023, AIME 2025), one-shot OPD hits 68.5 vs 69.8 for full-data OPD at step 300 — recovering 69% of the teacher–student gap and 87% of full-data OPD's gain, and it keeps improving for hundreds of steps.
• It's not a Qwen quirk. Math averages climb 77.1→85.5 (R1-Distill), 28.2→40.2 (Llama-3B), 70.8→82.4 (OLMo-7B). Code, instruction following, and agentic tool use recover 73%, 66%, and 64% of the gap respectively.
• The coverage ladder explains it. One query covers 71.5% of full-data's state space (65.9% within 100 steps); 16 semantically distinct queries cover 98.9% and match full-data accuracy. What an added query is worth is set by whether it reaches new states — and 16 already reach nearly all of them.
• It extends to multi-teacher OPD. 16 diverse queries per domain recover 101% of full-data MOPD's gain (52.9 vs 52.8 average accuracy).
• Content barely matters. A content-free template ending in <think> and off-domain WildChat chat logs (0.17% math-related) drive OPD nearly as well as real training problems.
• The bottleneck is the optimizer. The absorption rate — the share of the remaining teacher–student gap each update closes — falls at the same pace on one query as on all 17,000. The authors' verdict: OPD is data-overfed but algorithm-starved.

Limitations: Small Models and a Proxy Metric

• The ceiling is full-data OPD, not the teacher. One-shot's recovered fraction ranges 62–89% through step 1000, where full-data still leads 72.1 vs 68.4. One query gets you most of the way, not all the way.
• State coverage is a proxy. It's measured against the reference space of full-data rollouts and weights every semantic cluster equally, regardless of how much teacher signal each carries.
• Scale is modest. 1.5B–7B students, 300–1000 steps, MOPD tested with three domains and one teacher each. What sets the absorption rate is, by the authors' own admission, still open.
• Oddly, difficulty doesn't matter. A query the student never solves works as well as one it always solves — good news for anyone without a curated problem bank.

What Builders Should Change Monday

The paper's own conclusion is the sharpest sentence in it: an input need only start the student reasoning, "which turns data design from collecting problems into choosing teachers." If you're budgeting a post-training run, that inverts the spend — the dataset you're stressing over is ~98.9% redundant, while the teacher choice and the step budget do the real work. Sixteen diverse queries per domain is a defensible starting budget, and state coverage is the metric to check before you buy another thousand examples. Paired with Part I, the direction is unmistakable: OPD's value lives in the optimizer's slow absorption of states, not in data volume. Buy steps, not examples.