AL Alfred 2026-08-31 3 min read

Sliding-Window Beats Linear Attention: 2-10x Higher Long-Context Reasoning

Linear attention was supposed to be the fix. Quadratic scaling of softmax attention is the dominant cost in LLM inference, and a parade of linear-complexity alternatives — Mamba, Gated DeltaNet, MLA, linear transformers — promised to make long-context inference cheap. A paper dropping today on arXiv from Jolicoeur-Martineau, Sukthanker, Cameron, and Gervais flips that assumption on its head.

The headline: sliding-window attention (SWA) with sink tokens performs as well or better than post-trained linear attention across every task tested, and on long-context reasoning it's 2–10x better. SWA requires no post-training, is trivially parallelizable, and costs almost nothing to deploy.

The Problem: Linear Attention's Unchecked Claim

Quadratic attention means every new token costs more than the last. The KV cache grows unboundedly, and memory pressure becomes the bottleneck on long sequences. Linear attention — replacing the softmax with a kernel that factorizes the attention computation — promises O(n) memory and compute. But the community has been comparing linear attention to full softmax attention, not to simpler baselines that already exist.

Sliding-window attention, where each token only attends to a fixed window of surrounding tokens, is the simplest O(n) alternative. It's been used in production systems (Mistral 7B, Gemma) for years. The question the authors ask: given that SWA already exists, is post-training a model to use linear attention actually worth the effort?

The Method: SWA + Sinks vs. Post-Trained Linear Attention

The authors compare SWA with sink tokens against four post-trained linear attention methods: Linear Transformer (Katharopoulos et al., 2020), Linformer (Wang et al., 2020), Performer (Choromanski et al., 2020), and Nyströmformer (Xiong et al., 2021). They retrofit these onto existing pretrained LLMs (Pythia, Llama, OLMo) via the standard post-training procedure used in the literature.

SWA with sinks adds a handful of learnable "sink" tokens that attend to the full sequence, while the rest of the tokens use a fixed-size sliding window. This hybrid gives the model a global view through the sinks while keeping the per-token cost constant. The sinks are initialized from the first few tokens of the sequence (a trick from the StreamingLLM literature) and require no additional training.

mindmap
  root((Efficient Attention))
    Sliding-Window Attention
      SWA + Sinks
      No post-training
      O(n) compute
      O(window) memory
    Linear Attention
      Linear Transformer
      Linformer
      Performer
      Nyströmformer
      Requires post-training
      O(n) compute
      O(n) memory

The Results: 2-10x on Long-Context Reasoning

The results are stark. On standard benchmarks (MMLU, HellaSwag, ARC, GSM8K), SWA with sinks matches or slightly exceeds every post-trained linear attention method. On long-context reasoning, the gap widens dramatically:

SWA also wins on speed. Linear attention requires a kernel rewrite or specialized CUDA kernels to achieve its theoretical O(n) complexity. SWA is a simple masked attention operation that hardware already handles efficiently. The authors report SWA inference is 1.5-2x faster than the fastest linear attention implementation on an A100, with identical memory consumption.

Limitations

SWA with sinks does not match full softmax attention on tasks that require global bidirectional context — for example, span-level question answering where the answer requires attending to tokens far outside the window. The sink tokens provide some global context, but only in a forward direction from the start of the sequence.

The comparison is also limited to post-trained linear attention. Models trained from scratch with linear attention (like Mamba-2 or Gated DeltaNet) may close the gap, though they require entirely new pretraining runs. The paper does not test those models.

Finally, the window size matters. The authors use a 4096-token window for all experiments, which is reasonable for a 16K context but may not scale to 100K+ tokens without proportional memory growth. The sink token approach helps, but it's not truly constant memory for arbitrarily long sequences.

Why Builders Should Care

If you're deploying an LLM for long-context tasks and considering linear attention for cost savings, this paper suggests you should try SWA with sinks first. It's a configuration flag, not a model rewrite. No post-training, no new CUDA kernels, no retraining pipeline.

The broader implication is that the linear attention research community has been benchmarking against the wrong baseline. By comparing against full softmax attention instead of the simplest O(n) alternative, the field has inflated the apparent value of complex linear methods. It's a methodological blind spot that this paper cleanly exposes.

For practitioners: the next time you see a paper claiming "linear attention matches full attention at lower cost," ask whether it also matches sliding-window attention at the same cost.

Related Posts