StreamingLLM and the Discovery of Attention Sinks

TL;DR

A retrospective blog post by Guangxuan Xiao (MIT HAN Lab) on the 2023 StreamingLLM work, written in August 2025 after OpenAI’s GPT-OSS release shipped an attention-sink mechanism that explicitly cites StreamingLLM. The narrative traces how a naive sliding-window KV cache catastrophically fails when the first few tokens are evicted, attributes this to a softmax-induced “pressure valve” — attention sinks, a few initial tokens that absorb otherwise-unused attention mass — and shows that permanently retaining the first ~4 tokens lets pretrained Llama-style models generate stably across 4M+ tokens. A pretraining ablation with a dedicated [SINK] token at position 0 reduces the number of required sink tokens from 4 to 1, foreshadowing OpenAI’s per-head learned-scalar sink. The post also frames attention sinks as a unifying lens connecting softmax-normalization geometry, “over-mixing” in deep stacks, and the activation-outliers that hurt low-precision inference.

Key claims

Naive sliding-window KV-cache management — drop the oldest tokens when the cache fills — fails immediately on Llama-2-style models: evicting the first few tokens causes perplexity to skyrocket and generation to collapse into nonsense, even though those tokens often carry minimal semantic content (just a BOS or a stopword). [§“The Catastrophic Failure”]
Attention visualization on Llama-2 across most layers shows large attention mass directed at the very first positions regardless of what the current token is predicting — these positions behave as sinks in the graph-theoretic sense, absorbing attention without redistributing it. [§“The Attention Sink Discovery”]
The mechanism is forced by softmax: attention weights must sum to 1, so when a head has no informative context to attend to, the mandatory budget gets allocated to a learned default position. Initial tokens consistently appear across training examples, so a small early bias gets amplified into a stable “pressure valve” role. [§“The Mathematical Origin of Attention Sinks”]
StreamingLLM is the minimal fix: permanently retain the K/V states of the first sink_size tokens (≈4) and run a sliding window over the rest. No retraining required; ~10 lines of cache-management code. [§“StreamingLLM: A Simple Solution”]
The retained-sink fix lets Llama-style models generate stably across 4 million tokens — orders of magnitude beyond their training context — while remaining computationally bounded. [§“StreamingLLM: A Simple Solution”]
A pretraining ablation on two 160M-parameter models (vanilla vs. prepended [SINK] token in every training sample) shows the dedicated-sink model needs only one sink token at inference, vs. four for the vanilla model, and converges slightly better during training. [§“The Pre-Training Insight”]
OpenAI’s GPT-OSS implementation chooses an alternative form of the same idea: rather than a learned key-value token, it adds a single learnable scalar to each attention head’s softmax denominator — parameter-efficient and retrofittable, but less expressive per-token than a dedicated sink embedding. [§“OpenAI’s Implementation: A Universal Approach”]
Follow-up work has converged on three mechanistic interpretations: sinks act as “pressure valves” preventing over-mixing in deep stacks (Barbero et al.), sinks emerge specifically from the softmax sum-to-one constraint (Gu et al., where non-softmax attentions don’t form them), and learned KV biases or sink prefixes can substitute for the implicit sinks while improving low-precision quantization (Sun et al., Son et al. “CushionCache”). [§“Understanding Why Attention Sinks Exist”]
Attention sinks appear in 80% of Llama 3.1 405B’s attention heads, and the phenomenon is more pronounced at larger scale. [§“Understanding Why Attention Sinks Exist”, citing Barbero et al.]
The connection between attention-sink mechanisms and quantization stability suggests that GPT-OSS’s built-in sink scalars may be part of what enables the model’s aggressive 4-bit weight quantization. [§“OpenAI’s Implementation: A Universal Approach”]

Method

The post is a retrospective, not a methods paper, but it lays out the original StreamingLLM recipe and the dedicated-sink ablation:

Inference-time fix. Standard KV-cache management keeps a sliding window over the most recent window_size tokens. StreamingLLM changes this to: keep the first sink_size (e.g. 4) tokens permanently, plus a sliding window of window_size - sink_size recent tokens. Total cache size and per-step compute remain bounded; the only structural change is which tokens get evicted. Applied to pretrained Llama-2 / Llama-3 without retraining.

Pretraining-time variant. Two 160M-parameter models trained from scratch on the same data, differing only in whether every training sample is prepended with a single dedicated [SINK] token. At inference, both can use the StreamingLLM cache management; the dedicated-sink model needs sink_size=1 (just the trained [SINK] token) where the vanilla model needs sink_size≈4. Training loss is reported as slightly better with the dedicated sink.

OpenAI GPT-OSS contrast. Per the GPT-OSS model card, each attention head’s softmax includes one learnable scalar added to the denominator — a universal-escape-valve form that doesn’t require modifying the training data and adds only one parameter per head. The post frames this as a complementary point on the design axis vs. the StreamingLLM full-token embedding.

Results

StreamingLLM enables stable perplexity on Llama-style models across 4M+ tokens vs. collapse beyond a few thousand under naive sliding-window. [§“StreamingLLM: A Simple Solution”]
The dedicated [SINK] token variant at 160M scale needs 1 sink vs 4 for the vanilla model, with a small training-loss improvement. [§“The Pre-Training Insight”]
Adoption timeline reported in the post: Intel Extension for Transformers (Oct 2023), HuggingFace Transformers main branch + endless-LLM-generation-on-iPhone demo (Dec 2023), NVIDIA TensorRT-LLM (Jan 2024), OpenAI GPT-OSS-20B / GPT-OSS-120B (Aug 2025). [§“From Research to Reality”]
Llama 3.1 405B exhibits attention sinks in 80% of attention heads — sink behavior intensifies at scale rather than diminishing. [§“Understanding Why Attention Sinks Exist”, citing Barbero et al.]

Why it’s interesting

This is the originating artifact for the attention-sinks thread that the wiki has been tracking through mechanistic and architectural follow-ups, and it’s worth having on the wiki as the canonical reference for that thread’s framing. The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks is the mechanistic dissection — what SwiGLU + RMSNorm + position-1 actually do to set sinks up — and A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training is the architectural-replacement story (Gated Attention / GatedNorm replace sinks while keeping the implicit-rescaling function). The blog post adds two things those papers don’t: (a) the production-adoption history (GPT-OSS shipping per-head sink scalars in Aug 2025 is the strongest signal yet that this is a settled design, not a research curiosity), and (b) the specific design-axis comparison between full-embedding sinks (StreamingLLM) and learned-scalar sinks (GPT-OSS), which is the immediately-actionable lever for any internal model that wants to retrofit sink behavior without retraining the embedding table. For Luma the relevance is direct: any decoder shipped at Luma that needs long-context streaming inference (>training-context) inherits this design question, and the GPT-OSS / StreamingLLM contrast is the cleanest way to frame the build-vs-retrofit choice. The quantization angle — that built-in sinks may be load-bearing for GPT-OSS’s 4-bit weights — is a concrete bridge between this concept and LLM Inference Efficiency‘s 1-bit / 4-bit weight-quantization sub-thread.