Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

TL;DR

Latent Sparse Attention (LSA) is a learned-indexer sparse-attention scheme designed for ultra-long context (>1M tokens), where the dominant cost in prior sparse-attention methods isn’t the attention itself but the indexer that picks which keys to attend to — reportedly ~98% of attention-layer time at 10M tokens. LSA clusters keys hierarchically in a learned latent space so each query scores a small tree of clusters instead of every key. Two variants: D-LSA retrofits a pretrained model via short distillation (only the indexer is trained); N-LSA is trained from scratch on next-token loss. On GLM-5.3-Flash (Qwen-3.8-27B) at 10M tokens, D-LSA reports 22.7× (72.4×) prefill speedup and 4.6× (42.5×) decode speedup over the baseline sparse (full) attention at comparable accuracy; N-LSA matches full-attention validation loss with 1.6× per- layer training speedup at 128K.

Key claims

At ultra-long context, prior sparse attention is bottlenecked by the key-selection indexer, which consumes about 98% of attention-layer runtime at 10M tokens [§1, Abstract].
LSA organizes keys in a learned latent space and clusters them hierarchically, so each query scores a tree of clusters rather than every key [§Abstract, §1 Introduction].
Two training recipes are proposed: Distilled-LSA (D-LSA) retrofits a pretrained model by training only the new indexer under a short distillation loss, and Native-LSA (N-LSA) trains the indexer natively against the next-token loss [§Abstract].
D-LSA on GLM-5.3-Flash at 10M tokens reports 22.7× prefill speedup and 4.6× decode speedup for one attention layer, versus the model’s existing sparse attention baseline, with similar downstream accuracy [§Abstract].
D-LSA on Qwen-3.8-27B at 10M tokens reports 72.4× prefill and 42.5× decode speedup versus full attention, again at comparable accuracy [§Abstract].
N-LSA matches full-attention validation loss while per-layer attention training is 1.6× faster at 128K tokens [§Abstract].
The subquadratic variant D-LSA (Tree) scales as approximately O(T log T) in prefill indexing cost, versus quadratic for naive full-key indexing [§1 — hierarchical-clustering motivation; corroborated by alphasignal summary].
Across downstream knowledge, reasoning and long-context tasks, D-LSA stays close to baseline Full Attention on Qwen-3.8-27B and to baseline sparse attention on GLM-5.3-Flash [Fig. 5, §Results].

Method

LSA replaces the “score every key” step of standard sparse attention with a hierarchical scoring over cluster centroids in a learned latent space. Keys are grouped into clusters at multiple levels; a query first picks the most promising top-level clusters, then descends into their children, recursively, until it has a small shortlist of keys to run exact softmax attention against. Because the clustering is in a learned latent space, nearby cluster centroids correspond to keys with similar query-relevance regardless of position in the sequence.

Two deployment paths are specified. D-LSA keeps the base model’s weights frozen and trains only the new indexer under a short distillation loss against the original attention — compatible with both full-attention backbones (Qwen-3.8-27B) and backbones that already use sparse attention (GLM-5.3-Flash). N-LSA trains the indexer end-to-end against the next-token loss and serves as the from-scratch design point. A Distilled training-free variant (D-LSA (Training-Free), a PCA map) is also released, in addition to Flat and Tree index-search variants of D-LSA (released code at src/lsa/core, served via a SGLang fork).

Baselines in the released code span Full Attention, SWA, NSA, PISA, and the two block-sparse-attention selection modes (Flat and Tree).

Results

Prefill: at 10M tokens on H200, one D-LSA attention layer runs 22.7× faster than GLM-5.3-Flash’s built-in sparse attention and 72.4× faster than Qwen-3.8-27B’s full attention [§Abstract, §Results].
Decode: at the same scale, 4.6× over the sparse baseline and 42.5× over full attention [§Abstract].
Training: N-LSA tracks Full Attention’s validation loss at 128K and runs 1.6× faster per attention layer; translated end-to-end, this is reported as 1.24× faster wall-clock on a from-scratch hybrid model (per the companion tweet).
Accuracy: D-LSA stays close to Full Attention on Qwen-3.8-27B and to baseline sparse attention on GLM-5.3-Flash across downstream knowledge, reasoning and long-context tasks [Fig. 5].
All reported speedups are “one attention layer” numbers, not end-to-end wall clock; the paper argues this is justified because at ultra-long context the indexing cost dominates everything else.

Why it’s interesting

LSA attacks the specific failure mode that recent long-context sparse- attention work has implicitly accepted: the indexer scales badly. The value proposition rhymes directly with the diagnosis LLM Inference Efficiency has been accumulating — “find the sub- component of attention whose cost dominates at the regime you care about, and attack that kernel” — but LSA pushes the regime out to 10M tokens, where the dominant cost is key selection rather than the softmax or the KV read. The D-LSA + N-LSA split is also a direct test of whether learned-indexer sparse attention is a retrofit or an architecture: the retrofit path (short distillation, indexer-only training) is what makes this a plausible drop-in for existing open models like GLM-5.3-Flash and Qwen-3.8-27B. Finally, the fact that the compression / selection branch of NSA was already identified as the wall-clock bottleneck for learnable sparse attention (VideoNSA: Native Sparse Attention Scales Video Understanding §4 Finding 5) makes LSA’s “cluster-the-keys-in-latent-space” idea a natural next move on that same thread — this is the first filed design where the index space itself is learned and hierarchical, rather than the selection pattern being learned over a fixed block grid.