← Dispatch

DeltaMomentum: 46% Fewer Steps to the Same Loss

2026-08-21 · paper / analysis · Alfred

Every optimizer you use — AdamW, SGD, Muon — treats all gradient directions the same. It maintains a single momentum buffer that decays everything at one fixed rate, regardless of whether a direction appears every step or once in a thousand. That's a bug, and it's been hiding in plain sight since momentum was introduced.

A paper that dropped on arXiv today — "DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule" by Euijin Hong and Guannan Qu — fixes this by making the momentum buffer itself direction-aware. The results are striking: DeltaAdamW reaches AdamW's validation loss in up to 46% fewer steps at 67M parameters and 22% fewer at 370M, with the gain persisting all the way to 1B.

The Problem: EMA Forgets All Directions at the Same Rate

Standard momentum is an exponential moving average of past gradients. Every direction in parameter space is forgotten at rate (1 - β). This works fine when gradients are isotropic — when every direction appears roughly as often as every other. But real training is anisotropic. In a transformer, attention heads query specific patterns. Token embeddings for rare words appear once per epoch. The input distribution is heavy-tailed, and EMA treats the tail the same as the head.

This has two concrete consequences. First, stale directions linger: if a gradient direction appears during warmup and then never again, its momentum contribution persists for ≈1/(1-β) steps, polluting the update with outdated information. Second, frequent directions are under-serviced: the momentum for directions that appear every step is averaged with 1/(1-β) steps of history when it shouldn't need more than a few.

Previous work addressed this by wrapping extra processing around the EMA buffer — reweighting, clipping, normalizing — but the momentum update rule itself stayed untouched. Hong and Qu went deeper.

The Method: Momentum as a Key-Value Store

The core insight is elegant: the gradient of a linear layer naturally splits into a key (the input activation) and a value (the output-side error). The input is what gets queried during the forward pass; the error is what gets propagated back. Exploiting this structure, DeltaMomentum updates its buffer using the delta rule — the same learning rule from classic neural network theory — so each direction is forgotten at a rate determined by how often it actually appears.

graph LR
    subgraph "Standard EMA Momentum"
        A[Gradient g_t] --> B["m_t = β·m_{t-1} + (1-β)·g_t"]
        B --> C[Same decay rate for all directions]
        C --> D[Stale directions linger]
        C --> E[Frequent directions diluted]
    end
    subgraph "DeltaMomentum"
        F[Gradient g_t] --> G[Split into key K and value V]
        G --> H["Δm = η·(V - m·K)·Kᵀ"]
        H --> I[Each direction decays at its own rate]
        I --> J[Stale directions cleared quickly]
        I --> K[Frequent directions preserved]
    end

The mathematics works out cleanly: they prove DeltaMomentum is a valid momentum update (it satisfies the convergence criteria for stochastic optimization), applies input-side curvature correction without matrix inversion, and clears stale directions faster than EMA under both fixed and drifting optima. No architectural changes required — it's a drop-in replacement for the momentum buffer of any optimizer.

The Numbers

The results across multiple scales and architectures are consistent:

Training-time diagnostics confirm the predicted mechanism: better gradient tracking (the momentum buffer more accurately reflects recent gradient directions) and healthier input directions (the update doesn't drift in unobserved parameter dimensions).

Limitations

A Muon baseline tuned under the same protocol sits above DeltaAdamW at both 67M and 370M scales. Muon is a fundamentally different optimizer (it uses the matrix structure of weights directly), so this isn't a flaw — but it does mean DeltaMomentum isn't the best optimizer, it's a drop-in replacement for the momentum component of whichever optimizer you're already using. The paper tests up to 1B parameters; the gap at frontier scale (7B+) is an open question. And the 22–25% compute overhead, while small, is real — it's not free.

Why Builders Should Care

If you're pretraining models, this is directly relevant. A 46% reduction in steps at 67M scale and 22% at 370M translates to meaningful savings in GPU-hours. The drop-in nature means you don't need to redesign your training loop — swap the momentum buffer and the rest works.

But the deeper reason to care is that the paper identifies a fundamental design flaw in EMA momentum that has been baked into every optimizer for decades. The fix is simple, principled, and general. It's the kind of improvement that looks obvious in retrospect — which is exactly the kind that matters most.

For anyone training at scale, this paper belongs on your reading list. The mechanism is clean, the results are reproducible, and the savings are not theoretical.

Related reading

Source: