Mixtures-of-Experts Overfit Repeated Data: 4x vs Dense 8x

The compute-optimal playbook has two standing rules. When data runs short, repeat it — Muennighoff et al.'s data-constrained scaling laws made ~4 epochs nearly free. When compute runs short, go sparse — MoE architectures dominate every recent efficiency frontier. A new Stanford study (arXiv:2609.11917, Jha, Li, Leskovec, Liang) shows these two rules collide, and it puts exact numbers on the collision: dense models tolerate 8x repetition with minimal degradation; MoE models start degrading at 4x and dramatically underperform dense after 32x. The MoE efficiency advantage has an expiration date, and it's denominated in unique tokens.

The problem: two standard playbooks, one hidden interaction

Data repetition and architectural sparsity are usually studied in isolation. The influential data-constrained scaling laws were fit on dense models on a single corpus (C4); the one prior MoE data point (Xue et al. 2023, a 16-expert T5) hinted that total parameters — not FLOPs — drive multi-epoch degradation, but from a single configuration. Nobody had swept the joint space. That matters now because the industry's default answer to the data wall is "repeat with an MoE": sparse models are precisely what you deploy when you want more capability per FLOP, and repetition is precisely what you do when unique tokens run out. If the two interact badly, a lot of training budgets are priced on a false assumption.

The method: a compute-matched grid over sparsity × repetition

The team trained compute-matched dense and MoE transformers at 80M, 200M, and 1B active parameters (up to 8.5B total), holding the total token budget fixed at each scale and varying the repetition rate R from 1 to 64+. MoE configurations spanned 8 to 128 experts with granularities from 1/2 down to 1/32 (sparsity 2–32), all matched to the dense baseline's active-parameter count. Data domains — DCLM web crawl, peS2o academic text, StarCoder code, Wikipedia — ran both solo and as mixes with per-domain repetition rates. A replicate with 4x more data (80 tokens-per-active-param instead of 20) confirmed the trends aren't a small-budget artifact.

The results: the crossover, and what sets it

Three findings carry the paper. First, the crossover: at 80M active parameters, dense models repeat data over 8x with minimal degradation, MoEs visibly degrade at 4x, and beyond 32 repetitions MoEs cede their all-unique-data advantage entirely, falling well below dense. Second — and this is the result I'd underline — the repetition penalty tracks total parameters, not active parameters. A 200M dense model's degradation curve sits between those of 80M-active MoEs with 158M and 244M total parameters. Sparsity buys compute efficiency and pays for it with memorization capacity: every extra expert is extra substrate for caching repeated tokens, and the router carves up the data stream so each expert sees only its shard.

Third, the mechanism. Router decisions ossify early — top-1 routing stability exceeds 0.9 for the second half of training, and ossifies faster under repetition. Expert knockout cost (the loss increase from zeroing one expert) rises 1.1x from R=1→32 at 16 experts and 2.3x at 128: repetition pushes experts from redundant toward specialized, and specialization correlates with overfitting. Tellingly, dropout reduces knockout cost without de-ossifying routing — so the overfitting lives in what the expert functions memorize, not in the router itself.

mindmap
  root((Repetition rate R))
    Dense 80M
      Safe through 8x
      Degrades after
    MoE 80M active
      Degrades from 4x
      Underperforms dense after 32x
      Worse with sparsity
        Tracks TOTAL params
          200M dense ≈ MoE 158–244M total
    What helps
      Dropout p 0.4 → MoE wins at 64x
      FFN/expert output masking
      Unrepeated similar domain mixed in
    What does nothing
      Weight decay
      Gradient norm clipping
      Router jitter
            

Two receipts deserve their own sentence. On data quality: one pass over raw DCLM-pool beat 32 repetitions of strictly filtered DCLM-baseline — filtering shrank the pool 40x, and uniqueness won. On mixing: degradation stayed confined to the repeated domain, and an unrepeated, semantically similar domain (web crawl alongside academic text) had a regularizing effect — but code mixed with web text got no such protection.

Limitations the paper doesn't dwell on

Scale, first. The largest model here is 8.5B total parameters — thirty to eighty times smaller than today's frontier MoEs. The authors show the response depends only weakly on the token budget, but nothing demonstrates parameter-scale invariance, and prior dense-side work (Kazdan et al. 2026) found repetition damage grows with model scale — which, if it transfers, makes the frontier risk worse than these curves suggest. Second, evaluation is held-out cross-entropy; no downstream benchmarks appear in the main results, and loss can overstate or understate task-level damage. Third, the training regime is exact duplication; organic corpora contain near-duplicates, which may behave differently. And the rescue comes at a price: p=0.4 dropout meaningfully hurts performance at low repetition, so it's a trade you make only when you know your R is high.

Why builders should care

If you're planning a multi-epoch run on a sparse model, treat "MoE vs dense" as a data-audit question, not an architecture preference: estimate your unique-token budget and repetition rate first, and only reach for sparsity if R stays below roughly 4x at your scale. Treat existing data-constrained scaling laws as dense-only until they're refit — this paper is the evidence they misprice MoE. If you're already committed and repetition is unavoidable, the working interventions are all output-dropping regularizers (dropout, FFN/expert output masking); weight decay, gradient clipping, and router jitter — the tools everyone reaches for — did nothing here. One extrapolation that's mine, not the authors': synthetic and distillation-heavy corpora are structurally high-repetition regimes — a teacher's outputs are far more self-similar than web crawl. If the tax scales with total parameters, frontier MoEs trained on teacher-generated data are sitting in exactly the untested quadrant. The regularization result says you can push the crossover out — to 64x and beyond. It also says, plainly, that nothing beats unique data. Masking buys time. It doesn't buy a refund.