What is distance generalization?

Generalization when inter-token distances change between training and inference while context length stays fixed — e.g., a model trained to copy tokens across 15–25 token delays tested at unseen delays.

Does NoPE beat RoPE and ALiBi on distance generalization?

In the paper's delay-copy experiments (T=256, 8-layer transformers), NoPE achieved the best out-of-distribution recall and RoPE the worst, matching prior length-generalization findings that causal transformers learn positions dynamically.

How does training-data diversity affect distance generalization?

Increasing training delays from 5 to 50 improves out-of-distribution accuracy, but sub-linearly — the OOD-to-in-distribution ratio falls as diversity grows, so diminishing returns are the rule.

NoPE Beats RoPE on Distance Generalization: d=5 to 50

arXiv:2609.11913, posted Sept 10. Delay-copy tasks, three position-encoding schemes, one uncomfortable conclusion for anyone who treats RoPE as load-bearing infrastructure.

The problem: length generalization isn't the only OOD axis

Length generalization — train on short contexts, test on long ones — has been studied to death. But there's a second, rarely isolated axis: distance generalization. Keep context length fixed; change the inter-token distance between "where the information sits" and "where it must be recalled." A model trained on retrieval delays of 15–25 tokens gets probed at delays it has never seen. This is the skill behind every pointer-chasing task: code where a variable is used 60 lines after definition, RAG with variable citation offsets, agentic memory reads at arbitrary offsets.

The method: delay-copy tasks, fixed context

The authors (Nevermann et al.) build two synthetic tasks on top of a task-switching framework: full delay-copy (C) and selective delay-copy (S), where only even numbers get copied. Sequences pivot between tasks, context stays at T=256, 8-layer / 8-head / d=512 decoder-only transformers, trained under three position-encoding regimes: RoPE, ALiBi, and NoPE — no positional encoding at all. Training delays sampled from a preset range; inference evaluated at every unseen delay. Teacher-forced accuracy isolates recall from generation compounding.

Results: RoPE is the problem, not the solution

Three findings that don't appear in the abstract:

1. NoPE wins, RoPE loses — again. All three schemes hit high in-distribution accuracy, but out-of-distribution NoPE beats both explicit schemes, with RoPE the worst. This is the third independent confirmation (following the length-generalization line the authors cite) that causal transformers learn relative positions dynamically. The mechanism isn't in the weights' positional prior — it's learned at attention time.

2. Data diversity has diminishing returns. Varying the number of training delays from 5 to 50 shows out-of-distribution recall improving, but sub-linearly: the ratio of OOD to in-distribution performance actually falls as you add more distances. Seeing 50 delay values does not buy you 10× the generalization of seeing 5. Whoever told you "just diversify the training distribution" owes you a scaling curve.

3. RoPE transfer is actively harmful. When the full-copy and selective-copy tasks are trained jointly with different delay ranges, RoPE's OOD curves decay faster than single-task training — the second task is a pure distractor. ALiBi and NoPE show positive transfer at moderate OOD distances but negative at larger ones. The prior transfer-learning result everyone quotes (auxiliary tasks help) does not survive contact with explicit position encodings.

mindmap
  root((Distance generalization))
    Encoding
      NoPE: best OOD
      ALiBi: mixed transfer
      RoPE: worst + distracts
    Data diversity 5→50 delays
      OOD improves
      But sub-linear
    Joint tasks
      RoPE: negative interference
      NoPE/ALiBi: positive then negative

Limitations the authors don't dwell on

Toy scale. 8 layers, 512 dims, T=256, synthetic integer vocabularies. Nothing here says a 100B-param RoPE model would improve by deleting RoPE — modern LLMs co-adapt their rotary baselines during pretraining in ways an 8-layer model never has to.
Teacher forcing only. Accuracy is computed with gold prefixes; free-form generation can compound errors differently. The authors admit this is deliberate isolation, but it means "recall ability" ≠ "usable retrieval in a chat loop."
Two tasks. C and S differ in one bit (even-only copying). Whether the negative-interference result holds for semantically rich auxiliary tasks is open — and their citation contrast with prior work suggests the answer depends heavily on task relatedness.

Why builders should care

The practical takeaway isn't "remove RoPE." It's that positional structure is a learned skill, and you can't assume a model trained on one spread of retrieval distances handles a different spread. If you fine-tune a model on short-context tool traces (RAG with snippets, code with tight definitions) and deploy it on long-offset retrieval, the failure mode is precise: the model locates the anchor but miscounts the offset. The fix the data points to isn't a different encoding — it's training-time diversity of the distances themselves, accepted with sub-linear returns, and keeping auxiliary distance-heavy tasks away from RoPE-based models unless you can afford to re-bake the base.

One more receipt for the pile: we already showed attention sinks, not RoPE, carry context anchoring in a Sept 9 post. This paper is the distance axis of the same argument. The positional-encoding stack is less load-bearing than the ecosystem pretends.