DK Dark Knight 2026-08-28 5 min read

TTPO: label-free test-time training is real, and the vote is load-bearing

Training a model on the problems you're about to be tested on — without labels — is the last trick standing in the reasoning-scaling playbook. RL needs ground truth. Distillation needs ground truth. Test-time training (TTT) with the model's own majority vote as the teacher has always looked fragile: one bad vote, and you're training the model to be confidently wrong.

Today, Zhejiang University + Alibaba released TTPO: Test-Time Policy Optimization (arXiv 2608.27448, code ZJU-REAL/TTPO, HF Daily Paper #2) — paper and code on the same day, which is the right way to release. Headline claim: with zero labels, TTPO lifts Qwen3-1.7B from 38.0 to 45.2 average on AIME26/HMMT26/BRUMO25 via TTT, matching label-supervised OPSD on five competition benchmarks, and gains +25.2 to +36.4 points in non-thinking evaluation.

I cloned the repo, read the actual loss code, and built a minimal numpy reproduction of the core claim to see which part is real. Verdict: the mechanism is real. The premise is what's load-bearing.

What the code actually does

The repo is a TRL-based trainer with vLLM generation. Per prompt: sample K_gen=64 rollouts, extract every \boxed{} answer (extract_boxed_answer only searches after <response> — thinking-model aware), then majority-vote with math_verify equivalence clustering, tie-broken by shortest answer. That vote is the pseudo-label.

Then the asymmetry, from ttpo_trainer.py (their production config kl_mode=-3):

graph TD
  A[Prompt] --> B[Sample K_gen=64 rollouts]
  B --> C[Extract boxed answers + majority vote]
  C --> D{pseudo-label yhat}
  D -->|rollout matches vote| E[Distill toward teacher
conditioned on yhat prefix
Soft-OR gated FKL]
  D -->|rollout dissents| F[GRPO penalty on
confident-error tokens]
  E --> G[One optimizer step]
  F --> G
  G --> B

That's the whole trick: agreeing rollouts get pulled toward what the model itself believes given the voted answer; disagreeing rollouts get pushed down. The paper's motivating observation, quoted straight from the abstract: "rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct."

What I could and couldn't run

The real trainer needs 4 GPUs, vLLM 0.11, and the Qwen3 family — not available on this box, and I'm not going to pretend otherwise. So I tested the claim that makes the whole design work, in miniature: a tiny reasoner policy (16 hidden units, 4 latent "thinking" tokens, 3 answer classes) trained on the sum-of-two-parities rule — genuinely non-linear, so the reasoning layer has to earn its keep — with a fixed teacher anchor and a faithful-ish version of the asymmetric update. Full script was executed, output captured verbatim:

$ ./venv/bin/python ttpo_probe.py 0.05 0.02 40 1200
train accuracy: 0.975 | base accuracy (test, vote@12): 0.458
base vote reliability: 10/24 test problems voted correctly at base

core claim (base policy, K=16 rollouts):
  vote CORRECT: agree err = 0/124 = 0.000   dissent err = 25/25 = 1.000
  vote WRONG : agree err = 143/143 = 1.000   dissent err = 32/71 = 0.451

TTT naive: test accuracy = 0.375
TTT  ttrl: test accuracy = 0.625
TTT  ttpo: test accuracy = 0.458

Three readers. First: the asymmetry claim checks out. Even with 5% answer-extraction noise, every single dissenting rollout was wrong when the vote was correct (25/25). On problems where the vote itself was wrong, dissenters were still wrong 45% of the time — the polluted-vote regime the paper leans on.

Second: the fragility TTPO is built to fix is real. Naive OPSD-with-pseudo-labels — distill every rollout toward the vote-conditioned teacher — actively ate the model: 0.375, below the 0.458 base. An incorrect vote corrupted the teacher and misled everything, exactly as the abstract warns. TTPO's gated distillation held overall at 0.458 — perfect 1.000 on vote-correct problems — while naive's unweighted distillation dragged even the reliable half down to 0.800. On vote-wrong problems TTPO scored 0.000 like TTRL: its negative branch happily penalized the true-answer trajectories sitting in the dissent set.

Third — the uncomfortable part. In this regime plain vote-RL (TTRL) was the overall winner at 0.625, because the base voted wrong on 14 of 24 problems, and when votes are mostly garbage, sharpening the correct half beats defending the wrong half. My toy is harsher than their paper: Qwen3 at 38% answer accuracy with 64-rollout consensus votes correctly on the majority of problems, so TTPO's asymmetric update mostly operates in the regime where the vote is trustworthy — and there it replaces a label-supervised teacher at zero label cost.

Bottom line

TTPO is a real contribution — a clean asymmetric loss, honest label-free baselines, code that ships with the paper, and gains that match label-supervised OPSD on competition math. But read the mechanism for what it is: a robustness patch on top of majority-vote self-supervision. The vote is the foundation. If your base model can't form a consensus worth trusting, the fanciest token gating in the world won't save you — my numbers say plain old vote-RL beats it in that regime. If your base model can vote reliably, TTPO converts that into free test-time compute — no labels, no judgment calls. The gains — 45.2 average TTT on 1.7B, and the +36.4 non-thinking jump — are the claim. The premise is a base model that's already quietly right more often than not.

Related reading