VideoNSA: Native Sparse Attention Scales Video Understanding

TL;DR

VideoNSA adapts DeepSeek’s Native Sparse Attention (NSA) to a video MLLM (Qwen2.5-VL-7B) by running NSA’s three-branch sparse attention (compression / selection / sliding-window with a learned per-head gate) on the vision tokens while keeping standard GQA dense attention on the text tokens. End-to-end fine-tuned on 216K LLaVA-Video instruction pairs for 4600 H100-hours, it scales reliably to a 128K vision-text context while using only 3.6% of the dense attention budget, and beats both token-compression baselines (FastV, VScan, VisionZip) and training-free sparse-attention baselines (MInference, FlexPrefill, XAttention, Tri-Shape) on long-video, temporal, and spatial benchmarks. Mechanistic analyses report task-dependent global/local budget allocation, gate-collapse across layers, and — directly relevant to the sink literature — branch-specific attention-sink behavior with an overall 0.3% sink ratio.

Key claims

A hybrid sparse-attention scheme — NSA on vision tokens, GQA on text tokens, with token classification by position ID at every layer — preserves instruction following while exploiting video’s temporal redundancy [§2.2].
The three NSA branches are necessary together: any single-branch or two-branch ablation underperforms by large margins (single-branch LongVideoBench ≤49.1 vs. full 60.0; best two-branch 49.4 vs. 60.0) [Table 2].
VideoNSA matches or beats every filed compression/sparse baseline on Qwen2.5-VL-7B: 60.0 LongVideoBench, 51.8 MLVU-test, 83.7 TimeScope, 44.4 LongTimeScope, 26.5 Tomato, 36.1 VSIBench [Table 1].
Extrapolation beyond training length is reliable: trained at 36K context, performance continues to improve out to 128K tokens, with token-per-frame vs. number-of-frames being a task-dependent knob (LongVideoBench prefers more tokens/frame; Tomato and TimeScope prefer more frames) [§4 Finding 2, Fig. 2].
Performance is highly sensitive to the global/local attention budget allocation; configurations near the training allocation generally dominate, and enlarging global attention (more selected blocks) outperforms enlarging local attention (wider sliding window) under a fixed budget [§4 Finding 3, Fig. 3].
The compression branch maintains roughly stable mean gate weight across most layers (information condensation), while selection and sliding-window branches lose importance in deep layers (L22–L26) — the last layer anomalously reactivates all three [§4 Finding 4, Fig. 5].
Inter-head gate similarity is high in middle layers for selection and sliding-window (heads converge on block-selection / local-integration policies) but consistently low for the compression branch (head-independent operation) [§4 Finding 4].
Wall-clock latency is dominated by the compression branch; it grows roughly linearly with context while selection and sliding-window remain cheap — the kernel-design target is the compression path [§4 Finding 5].
Sparse-trained QKV weights transfer back to dense inference: a “Dense-NSA” model (VideoNSA weights run under full attention) recovers and surpasses the Qwen2.5-VL-7B baseline on several benchmarks (e.g. VSIBench +10.7%, Tomato +3.5%), suggesting sparsity acts as an inductive-bias regulariser on attention distributions, not just a compute reduction [§4 Finding 1, Table 3].
Learnable sparse attention induces branch-specific attention sinks: the compression branch produces the most sinks with banded value-norm structure from token merging; the selection branch produces almost none (top-k filtering smooths value norms); the sliding-window branch shows clean sink/non-sink separation along the value-norm axis [§4 Finding 6, Fig. 7].
Dynamic gating mitigates the compression branch’s sink production — overall VideoNSA sink ratio is 0.3%, and stays low across layers; vanilla FlashAttention’s sink ratio grows with depth and spreads uniformly across the sequence [§4 Finding 6, Fig. 9].
Token-compression baselines (FastV, VisionZip) lag sparse-attention baselines across the board, especially on order-sensitive temporal reasoning (Tomato) and ultra-long video (LongTimeScope, 10-hour videos) [§3.1, Table 1].

Method

The backbone is Qwen2.5-VL-7B (28 query heads, 4 KV heads, GQA). At every LLM-decoder layer, input tokens are partitioned into vision and text sets using position IDs. Vision tokens are fed into the three NSA branches: (a) Compression (CMP) averages all tokens within blocks of size 64 (matched to tokens-per-frame) into block-level keys/values via a learnable MLP; (b) Selection (SLC) keeps the top-32 most salient key-value blocks by importance score; (c) Sliding Window (SWA) retains the most recent 256 tokens. A two-layer MLP + sigmoid gate per head learns a soft combination of the three branches’ outputs. Text tokens use standard GQA. The per-layer output concatenates the vision and text outputs at their original positions.

Training data is a 216K-pair filter of LLaVA-Video-178K: 4 fps, videos with 350–550 frames, max 50,176 pixels per frame, max 36K context length per training instance. The NSA implementation is adapted from FLA on top of the SWIFT framework; total compute is 4,600 H100-hours. Inference is evaluated on LMMs-Eval and VLMEvalKit, comparing dense FlashAttention, AWQ quantization, three training-free token-compression methods (FastV, VScan, VisionZip), and four training-free sparse-attention methods (Tri-Shape, MInference, FlexPrefill, XAttention) — all on the same Qwen2.5-VL-7B base.

The attention sink analysis follows the Pai et al. definition (low value norm + high average key-attention score, applied per-branch) and is reported per layer and per token position to localize where sinks accumulate.

Results

On Qwen2.5-VL-7B, VideoNSA delivers the strongest sparse-attention method results: LongVideoBench 60.0 (vs. dense 58.7), MLVU-test 51.8 (vs. 51.2), TimeScope 83.7 (vs. 81.0), LongTimeScope 44.4 (vs. 40.7), Tomato 26.5 (vs. 22.6), VSIBench 36.1 (vs. 29.7) [Table 1]. Token-compression methods regress on most benchmarks compared to dense — FastV LongVideoBench 57.3 / Tomato 21.6, VisionZip LongVideoBench 52.4 / MLVU-test 33.1 — while sparse-attention methods generally improve or match dense [Table 1].

VideoNSA’s largest wins over baseline Qwen2.5-VL-7B are on the spatial and fine-grained temporal benchmarks: VSIBench +20.3%, Tomato +15.9%, LongTimeScope +9.1% [Table 3]. The Dense-NSA ablation (same weights, dense inference) keeps most of the VSIBench (+10.7%) and Tomato (+3.5%) gain — most of the spatial uplift transfers without runtime sparsity, but the temporal-reasoning uplift requires dynamic gating to fully materialize.

Three-branch ablation: CMP-only LongVideoBench 48.1, SLC-only 48.4, SWA-only 49.1; best two-branch CMP+SLC 49.4. Adding the third branch jumps to 60.0 — the dynamic gate is the load-bearing element [Table 2].

Headline efficiency: at 128K vision-text context, VideoNSA uses only 3.6% of the dense attention budget while leading on the table.

Why it’s interesting

VideoNSA is the first filed work that runs NSA on video MLLMs end-to-end at production scale, and the timing matters for the Luma video stack. The wiki already tracks two adjacent threads on sparse attention: Sparsity is Cool: Reverse-Engineering MoBA and NSA reverse-engineered NSA/MoBA on text LMs and reported that NSA’s three branches collapse to one or two active branches per layer, with sliding-window dominating late layers and compression dying off after early layers; VideoNSA reports the same late-layer gate-collapse pattern on video MLLMs (selection and sliding-window dim in L22–L26, with a strange last-layer reactivation), and the opposite qualitative story for the compression branch — compression here stays the dominant gate across most layers, presumably because video tokens are far more redundant than text and condensation is the load-bearing operation. That’s a clean candidate experiment: does Tilde’s “prune unused branches in late layers” recovery hold on video MLLMs too?

The branch-specific attention-sink finding is the second non-obvious connection. The wiki’s two prior sink papers — The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks (dense pre-norm + SwiGLU causal chain) and A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training (Gated Attention / GatedNorm as a replacement) — frame sinks as a single dense-attention phenomenon. Tilde’s MoBA result added the “sinks survive when you change the attention mechanism” point. VideoNSA goes further: under learnable sparse attention with three branches, sinks become a per-branch property — compression manufactures them via token merging, selection eliminates them via top-k filtering, sliding-window cleanly separates them along value norm — and dynamic gating between branches drives the overall sink ratio down to 0.3%, an order of magnitude below dense FlashAttention’s depth-rising baseline. That’s an experimentally separable handle on sinks that the dense-attention literature can’t produce.

Finally, the long-context video angle is directly relevant to Luma’s video understanding stack. Where Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation applies a static energy-decay sparse mask to a video DiT for generation, VideoNSA applies a learnable sparse pattern to the same kind of long video token sequence for understanding — the two together sketch the shape of a unified “sparse attention is the right primitive for long video, but the right sparsity pattern is learned not hand-coded” thesis that no single paper makes yet.