Sliding-window beats linear attention

TL;DR

The authors argue that the recent wave of “retrofit a pretrained LLM into a linear-attention model via post-training” work has been evaluated against the wrong baseline. When compared against a Sliding Window Attention (SWA) mask with attention sinks — applied to the same pretrained model with no post-training, no extra parameters, and no kernel changes — SWA matches or beats post-trained linear attention across multiple LLMs and downstream tasks. The gap widens dramatically on long-context reasoning (Needle-in-a-Haystack, BABILong), where SWA is 2–10× stronger than linear-attention conversions. The paper’s practical prescription: to reduce inference memory cost of a pretrained LLM, use SWA + sinks; linear-attention retrofits are not worth their cost, and linear-attention models likely need to be trained from scratch to compete.

Key claims

• Post-training a pretrained LLM to use linear attention adds cost and complexity while degrading performance on long context “massively” [Abstract].
• Sliding Window Attention (SWA) with attention sinks matches or exceeds post-trained linear-attention baselines across multiple LLMs and downstream tasks [Abstract].
• On long-context reasoning benchmarks (Needle-in-a-Haystack, BABILong), SWA achieves 2–10× higher performance than post-trained linear attention [Abstract].
• SWA requires no post-training, is extremely fast, and low-memory — making it the recommended cheap-and-reliable inference-memory reduction for pretrained LLMs [Abstract].
• The core theoretical framing: linear attention has a fixed-size state and must therefore choose what to remember and what to forget; expecting a short post-training run to learn that policy well is misguided [author thread, Aug 31 2026].

Method

The paper contrasts two ways to cut the KV-cache cost of a pretrained decoder-only LLM without retraining from scratch:

1. Post-trained linear attention. Replace softmax attention with a linear-attention primitive (recurrent constant-state form) and run a moderate post-training pass to recover quality. This is the recipe underlying much of the 2025–2026 hybrid-linear line (e.g. Kimi Linear, Jet-Nemotron’s PostNAS-style conversions).
2. SWA + attention sinks. Keep the original softmax attention weights unchanged, but apply a sliding-window mask at inference: each query attends to the most recent W tokens plus a small set of “sink” tokens at the sequence start (the StreamingLLM recipe from Xiao et al., 2023). No parameter updates, no kernel changes beyond the mask.

The authors evaluate both across multiple pretrained LLMs and standard downstream tasks, with particular attention to long-context reasoning where the two mechanisms diverge sharply.

Results

• Across multiple base LLMs and downstream tasks, SWA + sinks matches or beats post-trained linear-attention conversions [Abstract].
• On Needle-in-a-Haystack and BABILong, SWA is 2–10× stronger than post-trained linear-attention baselines [Abstract].
• SWA delivers these numbers with zero post-training FLOPs versus the (non-trivial) post-training cost of the linear-attention retrofit [Abstract].

Why it’s interesting

This is a direct challenge to the retrofit branch of the hybrid-linear-attention program tracked in Hybrid Linear Attention. The concept page’s current stance — informed by Kimi Linear, MiniMax-M1, Jet-Nemotron — is that post-trained linear/sparse layers combined with a minority of softmax anchors can approach full-softmax quality at much better throughput. This paper says the retrofit half of that thesis has been evaluated against a stale baseline: if the goal is just to fit a pretrained model into a smaller inference budget, StreamingLLM (StreamingLLM and the Discovery of Attention Sinks) already does that at zero cost and holds up better on retrieval-style long-context tasks. It leaves the from-scratch hybrid-linear line (Kimi Linear, Qwen3-Next, Mamba-3) untouched — the authors are careful to say linear-attention models “likely require to be trained from scratch or extensive post-training in order to even match SWA” — but it narrows the practical case for the cheap-conversion recipes specifically. Complements the OlmPool long-context-extensibility findings (Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension) which showed SWA is one of the four “minor” architectural choices whose long-context effects don’t show up in short-context loss.

See also

• Hybrid Linear Attention — concept page whose “which linear:softmax ratio wins” and “does post-training close the recall gap” questions this paper takes a strong negative stance on for the retrofit case.
• StreamingLLM and the Discovery of Attention Sinks — the original StreamingLLM / attention-sinks recipe the paper’s SWA baseline uses directly.
• LLM Inference Efficiency — this paper’s practical recommendation (use SWA + sinks over linear-attention retrofits for inference-memory reduction) is a direct claim in this concept’s decision space.
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training — the “sinks are load-bearing outlier-driven rescaling” line that explains why permanently-retained sink tokens survive the SWA mask.
• Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension — OlmPool’s long-context-extensibility ablation, which identifies SWA as one of the “minor” architectural choices with outsized long-context effects.
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention — MiniMax-M1’s 7:1 lightning-attention hybrid is one of the retrofit-style recipes this paper’s critique targets.
• Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Linear’s KDA-based hybrid is the strongest current counter-argument on the from-scratch side, which this paper explicitly does not challenge.