A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training

TL;DR

A Qwen-team paper that argues attention sinks (a few tokens that accumulate huge attention logits) and residual sinks (a few residual-stream dimensions with huge activations) are two faces of the same mechanism: outliers that, together with their downstream normalization (softmax / RMSNorm), implicitly rescale the rest of the activations during training. The paper calls this outlier-driven rescaling, validates that removing it breaks training (clipping or normalization-removal both degrade), and proposes architectural replacements — Gated Attention and GatedNorm — that provide the same rescaling explicitly via learned gates, so the model no longer needs to manufacture huge outliers. Result: ~2-point average gain in pretraining benchmarks and a 1.2-point drop in W4A4 post-training-quantization degradation, i.e. models that are intrinsically friendlier to low-precision training and inference.

Key claims

Attention sinks and residual sinks emerge from the same functional pressure: outliers paired with softmax / RMSNorm act as rescale factors for the non-outlier components, not as information carriers [Abstract; §1].
Removing the corresponding normalization eliminates the outliers but degrades training stability and final performance — i.e. outliers are not an artifact to be wished away [Abstract conclusion (1); §3].
Directly clipping outliers while retaining normalization also degrades performance, confirming that the rescaling effect (not the outliers themselves) is what training needs [Abstract conclusion (1)].
The final additive contribution of attention and residual sinks to the residual stream is much smaller than that of non-outlier components — they act as scale, not signal [Abstract conclusion (2); §3].
Outlier-driven rescaling can be replaced by explicit learned gating: absorbing the rescale into learnable parameters or adding gated rescaling improves training (≈ +2 points average) and shrinks W4A4 quantization degradation by ≈ 1.2 points [Abstract conclusion (3); Table 1].
The Gated Attention (GA) variant and a GatedNorm replacement for RMSNorm together remove the need for either sink type, producing visibly smoother activation distributions across tokens and dimensions [§3, Table 1].

Method

The setup is a controlled pretraining sweep where, for each “rescaling location” (softmax attention and RMSNorm), the authors instrument what’s doing the rescaling: outlier-driven (the default), clipped (rescaling restricted), gating (an explicit learned scalar gate), or absorbed (rescaling baked into adjacent learnable parameters). Each configuration is trained from scratch and compared on a fixed pretraining schedule.

The architectural substitutions are small. Gated Attention multiplies the attention output by a learned per-head gate that takes over the rescaling role previously played by attention-logit outliers, so the softmax no longer needs a sink token to dump probability mass into. GatedNorm replaces RMSNorm with a variant that produces the equivalent rescaling without requiring a few residual-stream dimensions to grow enormous; the operative intuition is that GatedNorm explicitly factors out the activation’s magnitude and exposes only direction to the next layer (this is the part Kyle’s note frames as analogous to weight-norm — magnitude and direction get decoupled in the activation space, just as weight-norm decouples them in the weight space). Outlier statistics, per-row/column activation magnitudes, and the final additive contribution of sink positions are all tracked across configurations to validate the rescaling-vs-contribution hypothesis.

Results

The reported average pretraining gain from the gated-rescaling variants over the standard outlier-driven baseline is roughly +2 points across the evaluation suite [Abstract conclusion (3); Table 1].
Under W4A4 post-training quantization the gated-rescaling model loses ~1.2 points less performance than the baseline — i.e. the resulting model is materially more quantization-robust [Abstract conclusion (3); Table 1].
Ablating normalization without replacing the rescaling mechanism removes the outliers but loses training stability and final-loss [§3 (1)].
Ablating outliers (via clipping) with normalization intact also loses performance, isolating the rescaling effect as the load-bearing function [§3 (1)].
Sink positions contribute substantially less to the residual stream than non-outlier positions in the final model, supporting the “rescale factor, not contributor” framing [§3 (2)].

Why it’s interesting

This paper closes a circle that’s been open in the literature for a while. Streaming-LLM-era work identified attention sinks as load-bearing for inference; quantization papers (Outlier Suppression, AWQ, SmoothQuant) keep tripping over the same residual-stream outliers; and the µP-vs-modern-normalization debate (How to Set the Learning Rate for Large-Scale Pre-training?) keeps locating training stability in normalization choices. Qiu et al. tie them together: the outliers are how normalization-based architectures implement an implicit rescaling, and you can swap that mechanism out for an explicit gate that does the same thing without the side effects. For Luma the immediate hooks are (a) low-precision diffusion training and distillation — Gated Attention / GatedNorm makes a DiT’s activations intrinsically less hostile to FP8/INT4, which is the same direction mHC: Manifold-Constrained Hyper-Connections argues for from a different angle; and (b) Nano Diffusion specifically, where Kyle flagged the swap as architecturally cheap to drop in. The framing — “outliers are rescale factors, replace them with explicit gates” — is the kind of mechanistic-causal claim that should generalize to DiT attention blocks and AdaLN-style modulations, not just to LLM decoders.