Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

TL;DR

The original Qwen-team paper proposing Gated Attention — a head-specific sigmoid gate applied immediately after Scaled Dot-Product Attention (SDPA) — as a drop-in modification to softmax attention. A controlled sweep over 30 variants of 15B-MoE and 1.7B-dense models trained on 3.5T tokens establishes that this single change (a) consistently improves perplexity and downstream benchmarks, (b) enhances training stability and tolerates larger learning rates, and (c) eliminates the “attention sink” phenomenon by enabling input-dependent sparsity over the SDPA output. Awarded the NeurIPS 2025 Best Paper Award.

Key claims

A simple head-specific sigmoid gate applied after SDPA — modulating the attention output $Y$ by $\sigma(X W_\theta)$ — consistently improves performance over standard softmax attention across 30 architectural variants and two scales (1.7B dense, 15B MoE) on a 3.5T-token training budget [Abstract, §1].
The gain decomposes into two factors: (1) introducing non-linearity onto the otherwise low-rank value-output mapping, and (2) applying query-dependent sparse gating that modulates the SDPA output [Abstract, §3].
Gated Attention reduces first-token attention mass from ~46.7% (baseline) to ~4.8% — i.e. it eliminates the attention-sink behaviour identified by StreamingLLM rather than relying on it [Fig. 2].
The sigmoid gate frequently saturates near zero, creating an input-dependent sparse mask on value vectors that lets the model “reject” attention output when no relevant information is present in the context, removing the pressure to dump probability mass onto sink tokens [§3, §4].
The modification also eliminates the “massive activation” phenomenon — outlier residual-stream channels — that prior work tied to attention sinks [§4, Fig. 2].
Gated Attention improves training stability and tolerates larger learning rates than the baseline configuration at matched scale [Abstract, §1].
Eliminating attention sinks via sparse gating improves long-context extrapolation performance, suggesting sinks were not load-bearing for length generalization in the way prior heuristic fixes (StreamingLLM) assumed [Abstract, §4].
Across the ablation grid of gating positions (G1 post-SDPA, G2/G3/etc. at other locations), the post-SDPA head-specific sigmoid gate (G1) wins — gating elsewhere captures non-linearity but not the sparse modulation effect [§3].

Method

The setup is a controlled architectural sweep on Qwen3-style backbones at two scales: a 1.7B dense model and a 15B Mixture-of-Experts model, both trained on up to 3.5T tokens. The authors enumerate 30 variants that differ in (a) where a gate is inserted (positions G1 through several alternatives around the attention block), (b) the gate’s parameterization (per-head vs. shared, sigmoid vs. other activations), and (c) the gate’s granularity (head-specific scalar vs. value-vector-wise).

The winning variant (G1) is mechanically tiny: after computing the standard SDPA output $Y = \mathrm{softmax}(QK^\top / \sqrt{d})V$ for each head, multiply $Y$ elementwise by a per-head sigmoid gate $\sigma(X W_\theta)$ that takes the layer input $X$ as its only argument. The gate introduces a few million extra parameters and adds one elementwise op per head per token. There is no change to the softmax, no change to the residual stream, no change to the optimizer.

To diagnose why G1 works, the authors run two analyses. First, they probe attention maps in the trained models and quantify first-token attention mass and the prevalence of massive activations in the residual stream — both drop sharply under G1. Second, they argue analytically that without gating the value-output transform is a low-rank linear map ( $V \to Y \to W_O Y$ ); inserting a query-dependent sigmoid sandwich between $V$ and $W_O$ converts this into a non-linear map whose effective rank depends on input, which is the source of (1). The sparsity effect (2) is downstream of sigmoid saturation: when the model has no relevant information to retrieve, the gate saturates near zero and the attention block effectively contributes nothing to the residual stream, removing the pressure to manufacture a sink token.

Code and trained checkpoints (built on the Qwen3 model architecture) are released at https://github.com/qiuzh20/gated_attention and https://huggingface.co/QwQZh/gated_attention.

Results

On 1.7B-dense and 15B-MoE pretraining over 3.5T tokens, Gated Attention (G1) outperforms the un-gated baseline across the evaluation suite, with the gain holding at both scales [Abstract, §3].
First-token attention mass drops from ~46.7% (baseline) to ~4.8% under G1 — a >9× reduction that operationally eliminates the attention sink [Fig. 2].
Massive activations (extreme outlier channels in the residual stream) likewise disappear under G1, even on the 3.5T-token 15B MoE — a regime where outliers normally grow throughout training [§4].
G1 tolerates larger learning rates than the baseline at matched scale without instability [§3].
Long-context extrapolation improves under G1 relative to baseline at matched training-context length, indicating attention sinks were not providing length-generalization benefit [Abstract, §4].
Awarded the NeurIPS 2025 Best Paper Award (announced 2025-11-26) for the breadth of the sweep (30 variants × 2 scales × 3.5T tokens) and the mechanistic clarity of the non-linearity + sparsity decomposition.

Why it’s interesting

This is the empirical foundation the wiki has been missing — the wiki already files the unified-view follow-up A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training (same authors, January 2026) and the mechanistic dissection The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks (Sun/LeCun/Canziani, March 2026), but neither was the original paper that established Gated Attention as a thing. This is. Three concrete consequences. (1) The unified-view paper’s GatedNorm extension only makes sense in the context of this paper’s finding that the sigmoid-after-SDPA gate eliminates sinks — without that empirical anchor, the GatedNorm proposal in 2601.22966 reads as speculation rather than as a generalisation of a 30-variant sweep. (2) The Sun/LeCun dissection in The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks explicitly disagrees with the load-bearing-outliers framing the Qwen authors went on to develop; reading both alongside this paper makes clear that Sun et al. are arguing against Qiu et al.’s subsequent theoretical claim, not against the underlying G1 result (which they replicate qualitatively). (3) For Luma’s Nano Diffusion / DiT pipelines, this is the version of the gate to actually port — it’s mechanically tiny (one sigmoid per head per token), and the open-source Qwen3-architecture implementation is the natural starting point. The fact that it landed Best Paper at NeurIPS 2025 is mostly relevant as a signal that the result has been independently validated at review depth; the Luma-side question is whether the same gate transfers cleanly to AdaLN-modulated DiT attention blocks, which neither this paper nor its two follow-ups have tested.