Infinite Context Length with Global but Constant Attention Memory

TL;DR

A Han Lab (MIT) explainer from the SANA team that frames Linear Attention as the mathematical core enabling infinite-context LLMs and minute-scale video generation. The argument reduces to one identity: softmax attention’s O(N²) cost can be collapsed to O(N) by reordering Q(K^T V) into Q(K^T V) and accumulating a fixed-size “state” (K^T V matrix) over the sequence. The post positions this state-accumulation view as the load-bearing primitive behind the team’s own SANA-WM minute-scale world model and the broader hybrid-linear-attention wave in 2025–2026 frontier models.

Key claims

Linear attention reduces self-attention complexity from O(N²) to O(N) by reordering the matrix multiplications so that K^T V is accumulated into a constant-size state before being multiplied by Q [§TL;DR].
The constant-size accumulated state is what enables “infinite context” in the inference-time / memory-bandwidth sense: per-token decoding cost stops growing with sequence length [§TL;DR].
The framing positions linear attention as applicable to both LLMs (long-text generation) and long video generation as a shared problem with the same mechanical fix [§TL;DR].

Method

The post is an explainer, not a paper. The mechanical content is the standard linear-attention derivation: softmax(QK^T)V has O(N²) cost because the N×N similarity matrix is materialized; replacing softmax with a kernelizable feature map φ(·) lets you rewrite the attention as φ(Q) · (φ(K)^T V), and then accumulate S_t = Σ_{i≤t} φ(k_i) v_i^T as a constant-size state updated one token at a time. Per-token cost during decoding is O(d²) regardless of N. The blog presents this “state accumulation” view as the mental model worth holding in mind when reading the cluster of hybrid linear models (Mamba-2, Gated DeltaNet, RWKV, KDA) and the SANA team’s video-diffusion deployment.

Results

No new empirical results — this is a conceptual primer. The accompanying empirical artifact from the same authors is SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer (SANA-WM), which deploys the principle in a 2.6B hybrid-linear DiT.

Why it’s interesting

This blog is the author-side framing for SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer — Junsong Chen, Yuyang Zhao, Jincheng Yu, and Enze Xie are co-authors on both, and the SANA-WM design choice (frame-wise Gated DeltaNet interleaved with softmax) is the video-diffusion instantiation of the “state accumulation” mental model the post defends. It is a useful one-page reference for the Hybrid Linear Attention cluster on the LLM side (Kimi Linear: An Expressive, Efficient Attention Architecture, Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search, Mamba-3: Improved Sequence Modeling using State Space Principles) and an explicit bridge from that cluster to the video-context-length workarounds filed under Context Length / Quality Trade-off in Video Generation (MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation, Context Forcing: Consistent Autoregressive Video Generation with Long Context, PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference) — those compress or partition softmax KV caches; this post argues for changing the attention primitive instead.