al Alfred 2026-09-09 4 min read

Attention Sinks Not From RoPE: 0.92 Sink Survives Ablation

A Tohoku University team (Kiya et al.) published an intervention study yesterday that does what most attention-sink papers don't: it ablates the folk explanation instead of adding another correlational observation. The folk explanation is RoPE. The receipt says otherwise — randomize the first token's key-side rotation and the sink barely flinches.

The problem: everyone blamed the wrong suspect

"Attention sinks" — the first token soaking up disproportionate attention — and "massive activations" — enormous hidden-state norms at that same position — have been known since StreamingLLM (Xiao et al., 2024) and Sun et al.'s massive-activations work. They matter for two practical reasons: massive activations wreck low-bit quantization (the LLM.int8 outlier saga), and streaming/KV-eviction schemes build their whole design around keeping the sink token alive. The community's running theory leaned on positional encoding: position 1 is special, RoPE is the positional encoding, therefore RoPE. Nobody had cleanly separated the three things that make position 1 special until this paper.

Method: decompose position 1 into three suspects

Position 1 is special for exactly three reasons: it usually holds the BOS token, it holds the first RoPE rotation, and the causal mask forces it to attend only to itself (α₁,₁ = 1.0, what the authors call self-concentration). The paper intervenes on each independently, measuring sink strength with the Sink^ε metric — the fraction of layer-head pairs whose average attention to position j exceeds ε = 0.3 — across five models (Llama-2-7B, Llama-3.2-3B, Mistral-7B-v0.3, Qwen2-7B, pythia-1b) on WikiText. The key move is the third: self-concentration can be manufactured at any position by clamping attention weights, and the resulting mechanism — the attention output collapsing to a single unmixed Value vector, "Value-non-mixing" — can be reproduced without any intervention at all using repeated-token sequences.

mindmap
  root((Position 1 is special))
    Suspect A: BOS token
      Swap BOS to position 16
      Sink follows the token (0.76 at pos 16, Llama-2)
      BUT empty position 1 still sinks (0.92)
    Suspect B: RoPE rotation
      Randomize token 1's key rotation
      Sink barely moves (0.9305 → 0.9273)
      Sometimes RISES (Mistral 0.03 → 0.15)
    Suspect C: Causal mask
      Force self-attention at position 16
      Sink appears from nothing (0.00 → 0.74)
      Mechanism: Value-non-mixing

Results: the receipts

Three tables, three verdicts. First, the RoPE intervention (Table 3 of the paper, BOS-free sequences): Llama-2-7B's sink goes from 0.9305 to 0.9273 when token 1's key rotation index is randomized — a 0.3% relative drop. Llama-3.2-3B: 0.9218 → 0.9133. And in two models the intervention increases sink strength: Mistral-7B from 0.0343 to 0.1525, pythia-1b from 0.7337 to 0.7547. A causal driver should collapse when you ablate it; a passenger doesn't move when you rearrange the car. An intervention that raises the effect is the fingerprint of correlation, not causation.

Second, the BOS relocation (Table 1): move the BOS token to position 16 and the sink follows it — Llama-2 jumps from 0.0001 to 0.7597 at position 16. But the now-BOS-less position 1 still sinks at 0.9178 (vs 0.9321 vanilla). Two independent drivers, both real: token identity and structural position. Prior work argued for each separately; this is the first quantification of both in one metric.

Third, the causal-mask clamp (Table 4): force α₍₁₆,₁₆₎ = 1.0 at position 16 in natural text and sink strength explodes from effectively zero — Llama-2: 0.0073 → 0.8397, Llama-3.2: 0.0012 → 0.7424, Qwen2-7B: 0.0000 → 0.5085, pythia-1b: 0.0000 → 0.4703. Self-concentration manufactures a sink and its massive activations wherever you inject it. The repeated-token experiments then close the loop on mechanism: when all 64 tokens are identical, massive activations appear at every position while attention stays distributed — and when one distinct token is inserted, sinks and massive activations appear at exactly the positions whose output is still the unmixed repeated-token Value vector. The sink is not about where attention looks. It's about what the output is made of.

Limitations the authors flagged (and a few they underweighted)

The honest limitations: five models, all RoPE-based, English WikiText only. Mistral-7B-v0.3 and pythia-1b respond weakly and inconsistently — the Value-non-mixing hypothesis is clean in 3 of 5 models. Two they underweight: the RoPE intervention randomizes only the key-side index of token 1, sampled from {2…T}. Query-side rotations and query–key interaction are untouched, so "not RoPE" really means "not token-1's key-side RoPE index" — a narrower exoneration than the title implies. And the repeated-token inputs that support the mechanism are unnatural; whether naturally occurring mid-sequence sinks (Sun et al. observed them at delimiters like newlines) fit the same Value-non-mixing frame is explicitly open. That mid-sequence question is where the commercial value sits, and it's unanswered.

Why builders should care

If your quantization pipeline spends effort fighting outliers by suppressing BOS or fiddling with positional settings, this paper says the lever is elsewhere: the mechanism is an early-layer output-norm phenomenon driven by the mask's structure, so per-channel scaling or clipping that targets early-layer Value/output norms should beat token-level tricks. It also vindicates the mitigation direction Qwen3-Next and gpt-oss already took — architectures that reduce sink phenomena are treating a structural property, not a token-embedding artifact. And for streaming/KV-eviction people: since the sink regenerates from mask structure alone, your design constraint isn't "preserve the BOS embedding," it's "give position 1's unmixed output somewhere sane to go" — which is a different problem with different failure modes. Notably, this connects to our earlier coverage of quantization damage being global rather than local: massive activations are one of the mechanisms that makes bit allocation global in the first place. Fix the structural driver and the allocation math changes.

Related Posts