Hybrid Linear Attention

TL;DR

A 2025–2026 convergence pattern in open-weight LLM architecture: most layers run a fixed-state-size linear-attention primitive (Mamba-2, Gated DeltaNet, Lightning Attention, KDA, JetBlock) and a minority of layers run softmax attention (full / MLA / GQA / SWA) as a “global anchor”. The motivation is that long-generation decoding is memory-bandwidth-bound, so collapsing the per-token KV cost to a constant via linear attention dominates the latency win — but pure linear models still trail Transformers on retrieval / in-context recall, which the softmax anchors restore. The current open question is what ratio of linear-to-softmax is right (filed claims range from 3:1 KDA:MLA in Kimi Linear to 7:1 linear:full in MiniMax-M1) and which linear primitive wins on a given workload.

Key claims

• Pure linear-attention LLMs trail softmax Transformers on retrieval and in-context recall; the recommended deployment is a hybrid with a small number of global self-attention layers, which beats both pure-Transformer and pure-linear configurations on retrieval at matched compute (Mamba-3: Improved Sequence Modeling using State Space Principles §Retrieval Tasks).
• Among six SOTA linear-attention blocks under a fixed PostNAS protocol — RWKV7, RetNet, Mamba2, GLA, DeltaNet, Gated DeltaNet — Gated DeltaNet ranks first on average, attributed to combining data-dependent gating with the delta rule (Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search §2.3, Table 1).
• KDA refines Gated DeltaNet with a finer-grained gating mechanism and a specialized Diagonal-Plus-Low-Rank chunkwise algorithm, deployed as a 3:1 KDA-to-MLA hybrid; the team claims wins over full MLA on short-context, long-context, and RL regimes under identical recipes (Kimi Linear: An Expressive, Efficient Attention Architecture Abstract, README “Key Features”).
• The right linear:softmax ratio is currently unsettled across the cluster: Kimi Linear uses 3:1 KDA:MLA (Kimi Linear: An Expressive, Efficient Attention Architecture); MiniMax-M1 uses 7:1 lightning:softmax (MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention §1); Qwen3-Next interleaves Gated DeltaNet with Gated Attention at an unspecified ratio (Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement)); Nemotron-3-Super uses periodic Mamba-2 + LatentMoE blocks with a small number of global-attention anchors (Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning §2.1).
• KV-cache size, not parameter count, is the right efficiency target — for fixed cache size, more parameters can be allocated to improve accuracy without hurting generation throughput because decoding is memory-bandwidth-bound (Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search §2.5, Table 2, Finding 4).
• Hybrid linear-attention architectures expose RL-specific failure modes that don’t appear in dense softmax models: precision mismatch between training and inference kernels driven by high-magnitude LM-head activations stalls reward growth, fixed by running the LM head in FP32 (MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention §3.2, Fig. 3).
• Optimizer hyperparameters need re-tuning for hybrid models — MiniMax-M1 reports gradient magnitudes spanning 1e-18 to 1e-5 with most below 1e-14, motivating AdamW eps=1e-15 vs the VeRL default eps=1e-8 which causes non-convergence (MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention §3.2).
• Context-extension in hybrid linear stacks is fragile: direct extension induces gradient explosion attributed to differing decay rates across early vs late linear layers, requiring staged extension (e.g. 32K → 1M in MiniMax-M1) (MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention §2.1).
• Mamba-3 explicitly rebuilds the SSM block for the inference-bound regime: an exponential-trapezoidal discretization subsumes the short causal convolution every prior Mamba/GDN block stacked, a complex-valued state is realized via RoPE on B/C without kernel rewrites, and MIMO raises per-step FLOPs essentially free at decode time because decode is memory-bound (Mamba-3: Improved Sequence Modeling using State Space Principles §3.1, Architecture).
• Throughput advantage of hybrid linear models grows with context length: Jet-Nemotron reports 21× at 4K → 47× at 64K → 53.6× at 256K over Qwen3-1.7B-Base (Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search §1, Fig. 6); Kimi Linear reports up to 6× TPOT at 1M (Kimi Linear: An Expressive, Efficient Attention Architecture README Fig. (b)); KV-cache reductions reach 47× (Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search §3.2, Table 3) and 75% (Kimi Linear: An Expressive, Efficient Attention Architecture Abstract) relative to softmax baselines.
• The mechanical core unifying every primitive in this cluster — Mamba-2, Gated DeltaNet, KDA, Lightning Attention, RWKV — is “state accumulation”: reordering Q(K^T V) so that K^T V is built up as a constant-size state, collapsing per-token decoding cost from O(N) materialized similarity to O(d²) state-vector update regardless of sequence length (Infinite Context Length with Global but Constant Attention Memory §TL;DR).
• Retrofit-style conversions of a pretrained LLM to linear attention via post-training are pitched as a cheap route to hybrid-linear inference gains (Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search PostNAS, Kimi Linear: An Expressive, Efficient Attention Architecture up-training variants).
  • Conflict: Sliding-window beats linear attention argues the retrofit branch has been evaluated against a stale baseline — Sliding Window Attention with attention sinks (StreamingLLM and the Discovery of Attention Sinks) applied to the same pretrained LLM with no post-training matches or beats post-trained linear-attention conversions across multiple models and downstream tasks, and is 2–10× stronger on long-context reasoning (NIAH, BABILong). Prescribes SWA + sinks over linear-attention retrofits for inference-memory reduction; explicitly does not challenge from-scratch hybrid-linear designs (Kimi Linear as originally trained, Qwen3-Next, Mamba-3), only the “convert an existing softmax model” recipes.

Recent contributions

• [2026-08-31] Sliding-window beats linear attention: Direct challenge to the retrofit branch of this cluster — Jolicoeur-Martineau, Sukthanker, Cameron & Gervais compare Sliding Window Attention + attention sinks (StreamingLLM, zero post-training cost) against post-trained linear-attention conversions across multiple pretrained LLMs. SWA + sinks matches or beats the retrofit on general downstream tasks and wins by 2–10× on Needle-in-a-Haystack and BABILong. Recasts the open “does hybrid linear close the recall gap” question: for pretrained models, the answer may be “the softmax you already have — masked, with sinks retained — closes it better than any post-training-based linear conversion will”. Leaves the from-scratch hybrid-linear line (Kimi Linear, Qwen3-Next, Mamba-3) explicitly out of scope; authors say linear-attention models “likely require to be trained from scratch or extensive post-training in order to even match SWA”.
• [2026-08-26] Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding: Qwen3.8-Flash-Next (125B / 6B-active + 51B n-gram embedding, Qwen4 preview) keeps the 3:1 linear:softmax ratio from Qwen3-Next but swaps Gated Attention for Qwen Sparse Attention (QSA) — block-level selection (512-block / 2048-token budget) via an MQA-style indexer, architecturally close to DeepSeek Sparse Attention. First filed production hybrid combining Gated DeltaNet with block-sparse softmax anchors instead of full/MLA/lightning — adds a new configuration to the cluster’s “right primitive” question.
• [2026-07-31] Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers: Chimera (Adobe) is the first filed video-DiT single-stream deployment of the KDA + interleaved MLA hybrid — text/image/video tokens flow through one raster-ordered stream with no positional embeddings, and modality-aware short convolutions add local spatiotemporal context on top of the linear+softmax attention mix. Reports 7.3× compute-efficiency over a Wan-2.1 2B full-attention baseline at 11B/2B-active. Second independent video-DiT convergence on the LLM-side hybrid recipe (after SANA-Video 2.0’s 3:1 pick), on a shared parameter path rather than a dual-stream one.
• [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — scales KDA to 2.8T-parameter frontier MoE with a lower-bounded log-decay reparameterization (scaled sigmoid, gmin=−5) that lets diagonal chunk tiles run as dense Tensor Core matmuls; also documents NoPE on the 24 Gated MLA layers with position-sensitivity carried by the 69 KDA layers.
• [2026-07-24] SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation: First filed video-DiT datapoint validating the 3:1 linear:softmax ratio on the diffusion side — SANA-Video 2.0 (5B/14B, trained from scratch) combines gated linear attention with 25% gated-softmax anchors, and reports 25% softmax as the compute-optimal quality-efficiency trade-off from reduced-resolution proxy sweeps. Matches the ratio Kimi Linear picked on the LLM side. Also the first filed external adoption of Block AttnRes (Attention Residuals (AttnRes / Block AttnRes) — Technical Report) outside Moonshot’s own stack — reports the same PreNorm-dilution fix (+12% deep-layer effective rank) in a diffusion transformer.
• [2026-07-22] SANA — Efficient High-Resolution Image and Video Generation family (NVlabs/Sana, sol-engine branch): The NVlabs/Sana umbrella repo consolidates the SANA linear-attention line — ReLU-based / Block Causal Linear Attention + Causal Mix-FFN — spanning image T2I, video (SANA-Video, LongSANA), world modeling (SANA-WM), and streaming V2V editing (SANA-Streaming). One code base, seven downstream applications, all built on the same linear-attention DiT architecture. Concrete evidence that linear attention has moved from “one paper claim” to “productionized model family”.
• [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): First trillion-scale deployment of Kimi Delta Attention (KDA): K3 (2.8T MoE) uses KDA as its attention backbone and ships a KDA-compatible prefix-cache implementation to vLLM to preserve serving economics.
• [2026-07-14] 1-bit Bonsai 27B — GGUF: 27B-class Reasoning at 1.125 Bits Per Weight: First filed evidence that a hybrid-linear backbone survives extreme weight quantization: 1-bit Bonsai 27B inherits Qwen3.6-27B’s ~75% linear / ~25% full-attention layout (full-attention KV cache grown on only 16 of 64 layers) and retains the full 262K context on-device — 100K peak fits in 11.6 GB without KV compression, and full 262K in ~9.4 GB peak with 4-bit KV. The architectural bet — that most layers can be linear-attention if a minority stay full — is what makes 27B-scale binary-weight edge deployment fit on mainstream laptops in the first place.
• [2026-07-09] Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity: SDM extends Gated DeltaNet by swapping the dense KV outer-product for sparse reads/writes into a large explicit memory bank — a state-capacity axis orthogonal to the layer-wise linear:softmax ratio question. Under iso-FLOP + iso-parameter, more state capacity via sparse addressing lifts in-context learning and long-context retrieval, arguing the “how much softmax” question is partly downstream of “how big is your linear state”.
• [2026-06-20] HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization: First filed argument that the head, not the layer, is the right hybridization granularity — interpretability analysis shows within-layer head specialization despite shared inputs, and the resulting head-axis FA/LA mix (with scale-normalized fusion + retrieval-critical head selector + 15B-token distillation transfer) matches a 3:1 layer-wise hybrid’s long-context quality at a 7:1 LA-to-FA ratio. Orthogonal axis to every filed layer-wise recipe (Kimi Linear, MiniMax-M1, Jet-Nemotron, Qwen3-Next, Nemotron-3-Super); resets the cluster’s open question on “right linear:softmax ratio” — layer-wise recipes may be over-allocating FA to heads that don’t need it.
• [2026-05-26] Laguna M.1/XS.2 Technical Report: Laguna XS.2 adopts 3:1 SWA:GA interleaving (SWA window 512, θ=10K, 64 Q-heads; GA θ=500K with partial RoPE on first 50% of head dim, 48 Q-heads) with softplus per-head gating — another datapoint that the 3:1 hybrid-SWA recipe is now standard in 30–300B-active coding MoEs.
• [2026-05-26] LT2: Linear-Time Looped Transformers: LT2 reaches the cluster’s prescription (“mostly linear, a few full-attention anchors”) from the looped-Transformer side. LT2-hybrid (Full + Gated DeltaNet) interleaves GDN with a small fraction of full-attention layers and exceeds the standard looped baseline; LT2-hybrid (GDN + DeepSeek Sparse Attention) matches it at fully linear-time cost. Adds a structural argument the cluster hasn’t made: looping itself acts as iterative state refinement for linear attention and progressive receptive-field growth for sparse attention, so the loop is not just an efficiency multiplier but a quality enhancer for these primitives.
• [2026-05-25] Infinite Context Length with Global but Constant Attention Memory: Author-side explainer from the SANA team (Junsong Chen, Yuyang Zhao, Jincheng Yu, Enze Xie) framing Linear Attention via the “state accumulation” mental model — Q(K^T V) reordering yields a fixed-size matrix state and O(N) total / O(d²) per-token decoding cost. Presented as the shared mechanical primitive behind both the LLM-side hybrid wave and the SANA team’s own minute-scale video deployment (SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer), making the cross-domain bridge explicit.
• [2026-05-25] Kimi Linear: An Expressive, Efficient Attention Architecture: Kimi Linear — KDA (refined Gated DeltaNet with finer gating + specialized DPLR chunkwise kernel) deployed as a 3:1 KDA-to-MLA hybrid at 48B-A3B / 1M context; first claim of identical-recipe wins over full MLA on short-context + long-context + RL regimes, with up to 6× TPOT and 75% KV-cache cut at 1M.
• [2026-05-25] Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search (retro-tag): PostNAS retrofits a pretrained full-attention checkpoint into a hybrid by freezing MLPs and searching only the attention stack; bakes off six linear primitives (Gated DeltaNet wins); JetBlock adds input-conditioned dynamic V-side causal conv on top.
• [2026-05-24] Mamba-3: Improved Sequence Modeling using State Space Principles (retro-tag): SSM-branch redesign for the inference-bound regime; explicitly recommends hybrid Mamba-3 + sparse global self-attention as the default deployment pattern.
• [2026-05-25] Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) (retro-tag): Qwen3-Next-80B-A3B production datapoint — Gated DeltaNet interleaved with Gated Attention, paired with a 512-expert MoE and native MTP; ~10× cheaper training and ~10× faster inference than Qwen3-32B per the team’s claim.
• [2026-05-23] Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning (retro-tag): hybrid Mamba-2 + Latent-MoE + global-attention anchors at 120B-12A, end-to-end NVFP4 pretraining at 25T tokens; the production-scale Mamba-Transformer hybrid datapoint in the open-MoE wave.
• [2026-05-24] MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (retro-tag): first frontier RL run on a hybrid linear stack (Lightning Attention 7:1 with softmax) at 456B-A45.9B; ships CISPO RL algorithm motivated by hybrid-attention RL pathologies, and documents the FP32 LM-head + retuned-AdamW recipe for stable training.

Open questions

• What is the right linear:softmax ratio? Filed claims disagree by a factor of two (3:1 KDA:MLA in Kimi Linear vs 7:1 linear:softmax in MiniMax-M1) and the gap maps directly to “how much retrieval / in-context recall does the workload need”. No paper in the cluster has run a controlled sweep yet.
• Which linear primitive wins at scale? Jet-Nemotron’s PostNAS bake-off picks Gated DeltaNet; Kimi Linear refines it into KDA; Mamba-3 argues the SSM branch is competitive after the inference-first redesign; MiniMax-M1 stays with Lightning Attention. No head-to-head at matched compute exists.
• Does hybrid linear close the recall gap for in-context tasks (RAG, long-form coding agents) where exact retrieval of distant tokens matters, or only for retrieval-style benchmarks like RULER? The strongest evidence so far is RULER-at-128K wins (Kimi Linear: An Expressive, Efficient Attention Architecture, Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement)) but real agent workloads are still mostly run on softmax. Sliding-window beats linear attention sharpens this question by showing that for retrofits the recall gap actively widens — SWA + sinks beats post-trained linear attention 2–10× on NIAH/BABILong; from-scratch hybrid designs remain the untested case.
• How do hybrid linear stacks interact with RL? MiniMax-M1 documents two failure modes (FP32 LM head; AdamW eps) and one algorithm change (CISPO clips IS weight not token update). Whether these are MiniMax-specific or general to the hybrid class is open.
• What’s the right context-extension protocol? Direct extension explodes gradients in hybrid stacks; staged extension works but is expensive. Mamba-3 sidesteps this via exponential-trapezoidal discretization that subsumes the short causal conv — whether that generalizes to delta-rule branches (KDA, GDN) is unclear.
• Is post-training conversion of a pretrained LLM to linear attention worth doing at all? Sliding-window beats linear attention argues no — SWA + sinks is a strictly better lever for cheap inference-memory reduction on an existing checkpoint. The from-scratch case is untouched by this critique, but the retrofit case (Jet-Nemotron PostNAS, Kimi Linear up-training variants, hypothetical “cheap linear conversion of Qwen/Llama”) now has a strong negative baseline to clear.

Papers

2026-08

• Sliding-window beats linear attention — Sliding-window beats linear attention (published 2026-08-28)

2026-05

• SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer — SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer (published 2026-05-14)

2026-04

• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning — Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning (published 2026-04-03)

2026-03

• Mamba-3: Improved Sequence Modeling using State Space Principles — Mamba-3: Improved Sequence Modeling using State Space Principles (published 2026-03-17)

2025-11

• Infinite Context Length with Global but Constant Attention Memory — Infinite Context Length with Global but Constant Attention Memory (published 2025-11-24)

2025-10

• Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Linear: An Expressive, Efficient Attention Architecture (published 2025-10-30)

2025-09

• DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention (published 2025-09-29)
• Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) — Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) (published 2025-09-11)

2025-08

• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search — Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search (published 2025-08-21)

2025-06

• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention — MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (published 2025-06-16)