What problem does Gradient-Aligned Reward (GAR) solve?

RLVR trains chain-of-thought reasoning with a binary outcome reward, which goes silent once a policy is good: all correct trajectories look identical to the optimizer. GAR replaces the sparse outcome bit with a dense reward computed in the policy's own gradient space, using expert solutions already present in the training corpus as anchors.

How does the gradient-aligned reward method work?

GAR runs truncated backpropagation through the output projection layer (lm_head) to extract a compact gradient vector for each rollout and for each expert-anchor solution, then scores rollouts by the cosine similarity between those gradient vectors. The signal decomposes into prediction-error and activation-pattern factors, and blends with the outcome reward inside GRPO or REINFORCE++.

What results does GAR achieve on LLM reasoning benchmarks?

On held-out competition math, GAR-GRPO improves pass@1 over GRPO by up to +31.4% (HMMT '26) at 4B scale and +33.3% (HMMT '25) at 8B scale, including a 14.6% AIME '26 pass@1 gain on Qwen3-8B. Trained on math only, it transfers +2.57 points to GPQA Diamond pass@1 and +2.41 to MMLU-Pro with under 9% wall-clock training overhead.

What are the limitations of gradient-aligned rewards?

GAR depends on expert reference solutions, so domains without any anchors — novel research or unusual preference structures — can't use it. It only applies to verified-correct rollouts, is evaluated on Qwen3-4B/8B math-centric runs, and the lm_head backprop trick needs engineering to integrate with production RLVR stacks like vLLM or verl.

Gradient-Aligned Rewards Lift LLM Reasoning: 14.6% AIME Gain

Reinforcement learning from verifiable rewards (RLVR) is how reasoning models learn to think — but the reward is a single bit: right or wrong. Once a model can solve a problem, that bit stops telling it how to improve. A new paper, Gradient-Aligned Reward (GAR), replaces the outcome bit with a dense signal computed in the policy's own gradient space — and reports a 14.6% pass@1 gain on AIME '26 with Qwen3-8B, plus transfer to GPQA Diamond and MMLU-Pro, at under 9% training overhead.

The Problem: Binary RLVR Rewards Go Silent

RLVR trains chain-of-thought reasoning with a binary outcome reward. That works until the policy gets good: all correct trajectories look identical to the optimizer, so there is no gradient pressure toward better solutions — only toward the same ones. Existing dense reward alternatives fall into two camps: surface heuristics that don't track reasoning quality, and process reward models that need expensive per-step annotation. Both ignore something already sitting in every training corpus: expert reference solutions. GAR's core move is to use those experts as anchors without ever labeling them.

The Method: Rewards in Gradient Space

GAR extracts a compact gradient vector from each rollout by running truncated backpropagation through the output projection layer (lm_head) — the only layer touched by every generated token. It does the same for expert-anchor solutions, then scores each rollout by the cosine similarity between the two gradient vectors. The authors prove this cosine decomposes into a prediction-error factor and an activation-pattern factor, giving a concrete account of what the alignment signal actually measures. The dense score blends with the outcome reward and feeds standard optimizers: GAR-GRPO and GAR-REINFORCE++.

flowchart LR
    A["Policy rollouts
(K responses)"] --> B["lm_head truncated
backward pass"]
    C["Expert anchor
solutions"] --> D["lm_head truncated
backward pass"]
    B --> E["rollout gradient
vector"]
    D --> F["anchor gradient
vector"]
    E --> G["cosine similarity
= dense reward"]
    F --> G
    G --> H["GRPO / REINFORCE++
+ outcome reward"]
    H --> I["8.3% wall-clock
overhead"]

Results: Consistent Gains Across Two Scales

On held-out competition math (IMO-AnswerBench, HMMT '25/'26, AIME '26), GAR-GRPO beats GRPO at pass@1 on Qwen3-4B by +10.1%, +31.4%, +30.5%, and +24.5% respectively, and on Qwen3-8B by +16.5%, +21.5%, +33.3%, and +14.6% — the 14.6% AIME figure in this post's headline. Gains hold under GAR-REINFORCE++ too, and an ablation shows full GAR beats every component removed — the alignment signal matters, not just extra compute. Transfer: trained only on math, the 4B model gains +2.57 points on GPQA Diamond pass@1, +1.79 on pass@4, +1.94 on majority voting, and +2.41 on MMLU-Pro vs. GRPO, beating the strongest prior baseline in every column. The 8B shows the same pattern (+2.02 to +2.29 absolute). Cost: GAR adds 8.3% wall-clock overhead per training step — a single extra truncated backward pass per verified-correct response.

Limitations

GAR is anchored to expert solutions, so if your domain has no reference solutions — novel research problems, unusual preference structures — there is no anchor gradient to align to, and the method degrades toward outcome-only RL. It is also only applied to verified-correct rollouts; the alignment signal cannot steer the model away from wrong trajectories that happen to look aligned. The experiments are single-family (Qwen3-4B/8B, math-centric) with fewer than a dozen runs per cell — statistically real but not frontier-scale evidence. And while the code is public, the lm_head-only trick is PyTorch-specific; production RLVR stacks (vLLM, TRL, verl) will need real engineering to adopt it.

Why Builders Should Care

Reward design is the bottleneck in most RLVR pipelines: sparse outcome rewards cap ceiling, and dense alternatives cost labels. GAR shows the training corpus itself — the expert solutions you already paid for — can be mined as a free dense signal, with overhead small enough to ship into existing runs. It also pairs with two failure modes this blog has covered: it attacks credit-assignment blindness from the reward side (complementary to the solution-collapse fix in our RLVR entrance-lock post), and its reliance on outcome correctness still assumes the verifier isn't gameable (see the reward-hacking post below). For teams training reasoning models at 4B-30B, this is the most practical dense-reward recipe to land this month.