GRPO Verifier Errors Correlate: 8 Rollouts Are Really 1.7

Group-based RL with verifiable rewards (RLVR) has a quiet statistical assumption baked into it: the N completions you sample per prompt are N independent chances to get a clean reward signal. A new analysis of Qwen2.5-1.5B rollouts says they are not — and quantifies exactly how far off the independence assumption is. The headline: within a group of eight completions, verifier errors correlate at 0.530 (95% CI: 0.500–0.560). Under an exchangeable-error model, that's a design-effect-adjusted effective sample size of 1.70 per eight-rollout group. Your eight votes carry the information of about one and a half.

The problem: verifier noise was modeled as independent

RLVR pipelines score completions with automatic verifiers — rule-based checkers, answer matchers, unit-test runners. Known problem: verifiers make mistakes. Assumed fix: noise averages out across the group, because each rollout's error is its own coin flip. The literature on correlated judgment errors mostly studied multiple evaluators grading the same output. This paper studies the case that actually matters for GRPO: one verifier grading eight sibling completions of the same prompt — completions that share an answer format, shared derivation habits, and shared failure modes.

The method: 24,998 groups, one correlation

The author generated 24,998 groups of eight completions from Qwen2.5-1.5B across MATH, GSM8K, and DeepMath-103K, scored each with rule-based verifiers, and estimated the pooled within-group verifier-error correlation. Then the paper does the useful part: it replays the group-relative advantages under four different verifier configurations and counts how often the advantage sign flips. That's the number that touches the gradient. Answer-form breakdowns (fractions vs. radicals vs. unit annotations) show where the clustering lives.

mindmap
  root((Within-group
error clustering))
    Strong clustering
      Fractions
      Radicals
      Symbolic expressions
      Intervals
    Weak clustering
      Unit annotations
      Percent signs
    Consequence
      Effective n of 1.70 per group of 8
      Advantage-sign flips in up to 0.83% of groups

The results, and what the 1.70 actually means

The abstract hands you 0.530 and 1.70 without showing the arithmetic, so here it is: for a group of n = 8 with intraclass correlation ρ = 0.530, the design effect is 1 + (n−1)ρ = 1 + 7 × 0.530 = 4.71. Effective sample size is 8 / 4.71 ≈ 1.70. Any standard error computed as if the eight completions were independent is understated by a factor of √4.71 ≈ 2.2. If you're using group-level pass rates to compare models or pick checkpoints, your confidence intervals are roughly half as wide as they should be.

But here's the nuance the abstract doesn't surface, and it cuts in GRPO's favor more than you'd expect. GRPO's advantage is mean-centered within the group. If verifier errors were perfectly correlated — every rollout in the group judged wrong, all truly wrong or all truly right — the group-relative advantage would be zero for every completion and the error would cancel exactly. Correlated errors only inject gradient noise when they create false spread within a group: some rollouts pass that shouldn't, some fail that shouldn't. That's precisely what the answer-form clustering enables — a verifier that systematically chokes on radical notation will flip some sibling expressions and not others, injecting error that is systematic, not random, and therefore doesn't average out over training. The paper's own replay measurement puts advantage-sign disagreement at up to 0.83% of groups under verifier-config swaps. At training scale — millions of groups — that's not noise you can ignore; it's a bias channel concentrated on the exact answer forms the verifier mishandles.

The answer-form result is also a warning about how everyone reports verifier quality. "95% verifier accuracy" pooled across forms can hide a 100% failure rate on intervals or symbolic expressions. The same clustering math applies to self-consistency evaluation: correlated errors mean k majority votes don't dilute like k independent ones — the design effect applies there too, and majority-vote accuracy gains are overstated wherever the answer format is one the verifier dislikes.

Limitations — including one the authors flagged and shouldn't have

The authors admit they cannot separate shared prompt difficulty from shared answer form in the 0.530. Fair, but it matters more than they let on: prompt difficulty is partially handled by GRPO's group-relative baseline (a uniformly failed group contributes nothing), while answer-form clustering is not. The true harm-relevant correlation is likely well below the pooled 0.530 — making 1.70 a floor, not an estimate. Second: everything is one model, Qwen2.5-1.5B, and error clustering is exactly the kind of thing that varies with policy strength. Third: the paper measures verdict correlation and config-dependent sign flips, but never the downstream training effect — no RL run here shows what 0.83% sign flips do to a converged policy. And the exchangeable-error model used to convert 0.530 into 1.70 is contradicted by the paper's own heterogeneity finding; the effective sample size is a summary of a summary, and the bimodal structure behind it is the real story.

Why someone building things should care

If you train with RLVR: (1) compute your verifier-error intraclass correlation on your own rollouts — it's a one-day analysis and it recalibrates every CI you've been quoting; (2) audit verifier failures per answer form, not pooled — the aggregate number is actively misleading; (3) treat verifier swaps as a training-distribution intervention, because the paper shows config choice flips advantage signs on a measurable fraction of groups. And if you evaluate with majority voting over model samples, ask what your effective vote count really is before you report the margin.