The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

TL;DR

A mechanistic dissection of why massive activations (a few tokens with extreme outlier hidden values) and attention sinks (a few tokens that hoover up attention mass) reliably co-occur in pre-norm decoder Transformers (Llama, Qwen). The paper traces the full causal chain: one or two early SwiGLU blocks act as directional quadratic amplifiers that inject spikes into a shared trigger direction; RMSNorm then maps spike tokens to a sparse, near-constant low-dimensional vector; that geometric collapse is what lets specific attention heads create a stable “default” position to dump attention onto. The two phenomena are shown to be architecturally coupled rather than functionally bound — ablations that change the normalization configuration decouple them, and either can be independently suppressed without hurting LM performance.

Key claims

Massive activations follow a “rise–plateau–fall” lifecycle: 1–2 early step-up blocks inject extreme values in spike channels, the residual stream propagates them additively across intermediate layers, and 1–2 late step-down blocks cancel them with opposite-sign outputs before the LM head [§3.1.1, Fig. 1, Table 1].
The SwiGLU feed-forward block is the primary source of massive activations and operates as a directional quadratic amplifier: for high-gain output channels, the relevant $M^{(c)}$ matrices are rank-one-dominated and share a common principal eigenvector $\mathbf{v}^*$ across channels, so any input aligned with $\mathbf{v}^*$ simultaneously triggers all spike channels with fixed inter-channel ratios [§3.1.2, Figs. 3–4, Eqs. 16–21].
The first token is overwhelmingly a spike token (≥98.4% of vocabulary entries when placed at position 1, across Llama 2/3 and Qwen 2.5/3) because the position-1 attention block degenerates to a static linear map (the token can only attend to itself), reliably steering its representation toward $\mathbf{v}^*$ [§3.1.3, Table 2, Eq. 22].
Delimiter tokens (periods, newlines) become spike tokens via self-sinking: their embeddings are near-collinear with learned RMSNorm scale parameters, which inflates their early post-norm magnitudes, induces multiple early heads to attend almost only to themselves, and reproduces the position-1 linear-map regime — once aligned with $\mathbf{v}^*$ the quadratic amplifier fires [§3.1.3].
After RMSNorm, spike-token representations become (i) bounded, (ii) sparse with mass concentrated on the spike channels, and (iii) nearly token-invariant — cosine similarity between distinct spike tokens after a step-up block approaches 1 [§3.2.1, Fig. 5, Eqs. 23–25].
Attention sinks emerge from relative geometric alignment of query and key subspaces, not from absolute subspace volume: in sink heads, $\mathrm{span}(Q)$ sits closer to the fixed $\mathrm{span}(K_{\text{sink}})$ (typically 1–2 dimensions) than to $\mathrm{span}(K_{\text{non-sink}})$ , producing large persistent logit gaps favoring sink tokens [§3.2.2, Fig. 6, Eq. 26].
Across architectural ablations, changing the normalization configuration suppresses massive activations while preserving attention sinks — i.e. the two phenomena can be decoupled, demonstrating that their co-occurrence is an architectural artifact rather than a functional necessity [§4.2, Abstract].
Sink ratio is a robust proxy for optimization health: extreme LRs, disabling weight decay, or mis-specified AdamW $\beta_2$ all reduce sink ratio, while longer training or removing LR decay raises it; the magnitudes of massive activations vary largely independently of perplexity and sink ratio [§4.1, Table 3].
Attention sinks are primarily driven by the dimensionality of the attention head space and by the training context-length distribution, and they bias certain heads toward short-range / local-syntactic dependencies [§3.3, §4, Abstract].
Either phenomenon can be suppressed independently without degrading language-modeling perplexity, suggesting both are incidental architectural side-effects rather than necessary computational primitives [§4, Abstract].

Method

The paper combines mechanistic analysis of pretrained open-weight models (Llama 2 7B/13B, Llama 3 8B, Qwen 2.5 7B/14B, Qwen 3 8B/14B) with controlled from-scratch ablations of a Llama-style 7B trained on 100B DCLM tokens.

The mechanistic side does three things. (1) Trace top-3 channel magnitudes block-by-block to identify step-up and step-down blocks (Table 1 reports their indices). (2) Decompose the SwiGLU feed-forward block analytically: under the empirically-verified near-identity SiLU regime for spike tokens, the block reduces to a quadratic form $y_c = \mathbf{x}^\top M^{(c)} \mathbf{x}$ ; the authors examine the Frobenius norms and eigenvalue spectra of $M^{(c)}$ across channels and confirm that spike channels are exactly those with anomalously large $M^{(c)}$ that are also rank-one-dominated by a shared principal eigenvector $\mathbf{v}^*$ . (3) Probe why specific tokens align with $\mathbf{v}^*$ : a vocabulary-wide test inserting each token at position 1 (>98% become spike tokens) and an analysis of the degenerate position-1 attention map, plus a self-sinking analysis for delimiter tokens.

The geometric account of attention sinks shows that after RMSNorm, spike-token keys collapse into a 1–2-dimensional subspace and are nearly token-invariant; sink-vs-non-sink head classification is then explained by whether that head’s query subspace aligns more strongly with this fixed sink-key subspace or with the broader non-sink key manifold, visualized via t-SNE.

The ablation side trains the same Llama-style 7B from scratch while varying (a) optimization hyperparameters (LR, weight decay, AdamW $\beta_2$ , total tokens), (b) feed-forward design (GeLU two-layer, single linear, attention-only), and (c) normalization configuration. For each configuration the authors report perplexity, sink ratio, and max activation magnitude, which lets them separate optimization-driven effects (sink ratio tracks optimization health) from architecture-driven effects (normalization governs whether spikes and sinks co-occur).

Results

All seven open-weight models examined exhibit the rise–plateau–fall pattern with step-up blocks in the first ~14 layers and step-down blocks in the last 1–2 layers [Table 1].
Position-1 vocabulary probes find ≥98.40% of vocabulary entries become spike tokens at position 1, and the few exceptions are mostly rare low-resource-script characters whose embeddings remain near initialization [§3.1.3, Table 2].
Empirically, the key vectors of spike tokens collapse to a 1–2-dimensional subspace of head dimension $d_k$ , against full head dimensions of 64–128 in the studied models [§3.2.2].
The baseline 100B-DCLM-trained 7B reproduces perplexity ≈10.1, sink ratio 46.0%, max spike magnitude 3818 [Table 3].
Optimization-hyperparameter sweep: best perplexity (9.5) is at 1T tokens with sink ratio 63.3% (highest); worst (11.8) is at the smallest LR with sink ratio 18.6% (lowest), confirming the sink-ratio / optimization-health correlation [Table 3].
Disabling weight decay drives the max spike magnitude to 12275 without improving perplexity or sink ratio, dissociating spike magnitude from sink-formation: once spikes are large enough to dominate post-RMSNorm representations, further growth gives no additional sink ratio [§4.1, Table 3].
Architectural ablations in §4.2 (full quantitative details continue into the truncated portion) show that the normalization configuration is the single architectural lever that decouples spikes from sinks — changing it suppresses spikes while sinks remain, with no perplexity penalty [Abstract, §4.2 intro].

Why it’s interesting

This is the mechanistic complement to A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training. Where the Qwen paper says “attention sinks and residual sinks are the same outlier-driven-rescaling phenomenon, and you can replace it with explicit Gated Attention / GatedNorm,” Sun et al. answer the prior question: exactly how does the co-occurrence get set up, layer by layer, in a standard pre-norm Llama/Qwen stack? The answer — SwiGLU step-up blocks act as a quadratic amplifier along a shared $\mathbf{v}^*$ direction, position-1 plus self-sinking deliver tokens onto that direction, RMSNorm then sparsifies them to a fixed low-dim subspace that specific attention heads learn to target — is a complete causal chain that the unified-view paper does not provide. It also disagrees on a load-bearing point: the Qwen paper argues outliers are load-bearing (clipping them hurts training), while Sun et al. argue both phenomena are architectural artifacts that can be independently suppressed without hurting perplexity. That tension belongs in Training stability at scale as a real open question — is the rescaling functional or incidental? — and the ablations from this paper are the cleanest experimental wedge anyone has produced on it. For Luma, the same hooks Kyle flagged for Gated Attention / GatedNorm apply here in reverse: this paper tells you which architectural choice (pre-norm + SwiGLU, specifically) is the source of the FP8/INT4 outlier problem, and that a different normalization configuration can decouple the issue without an architecture rewrite — directly relevant to Nano Diffusion’s training-quantization pipeline and to any DiT/MMDiT stack that inherits the Llama-style pre-norm + SwiGLU pattern.