Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing

TL;DR

VSA2 is a trainable sparse attention for video DiTs applied across the whole development cycle — pretraining, RL, and inference — producing a model with comparable or better quality than a full-attention counterpart. Two architectural changes vs. the previous VSA: a fine-grained router that improves critical-token identification and allows each query to attend to a variable number of KV pairs (dynamic compute); and a Hard-to-Easy training curriculum where models trained under high sparsity but evaluated at lower inference sparsity outperform full-attention baselines in motion quality. VSA2 also supports sparse rebasing: full-attention early-stage checkpoints from progressive low-to-high resolution pretraining can be swapped onto sparse attention mid-training. On 720p, VSA2 reports 8.9× attention speedup and 4.62× end-to-end generation speedup over FlashAttention-3, at half the attention compute of the previous VSA and lower loss.

Key claims

A fine-grained router improves precision of critical-token identification over the previous VSA block-level router, and permits per-query variable KV-count attention (dynamic compute) rather than a fixed per-query budget [Abstract].
Hard-to-Easy curriculum: training at higher sparsity than used at inference generalizes effectively to lower inference sparsity, and the resulting models outperform full-attention counterparts in motion quality [Abstract].
Sparse rebasing: VSA2 can replace full attention in the middle of progressive low-to-high resolution pretraining, consuming full-attention early-stage checkpoints without discarding them [Abstract].
Halves attention compute relative to the previous VSA with lower loss [Abstract].
On 720p videos: 8.9× attention speedup and 4.62× end-to-end generation speedup over the FlashAttention-3 baseline, at comparable or better video quality [Abstract].

Method

VSA2 is a trainable sparse attention module for video Diffusion Transformers (DiTs), designed to be inserted across the entire DiT development cycle — pretraining, RL, and inference — rather than retrofit at the end. The architectural core is a fine-grained router that scores critical tokens at a finer granularity than the previous VSA block router, and that permits each query to attend to a variable number of key-value pairs (per-query dynamic compute, not fixed-K top-K). The training-side contribution is a Hard-to-Easy curriculum: train at high sparsity, evaluate at lower sparsity. The authors report that this curriculum generalizes down to the inference sparsity and produces better motion quality than a full-attention baseline — the training discipline, not the sparsity pattern alone, is what unlocks the quality parity.

The deployment-side contribution is sparse rebasing: in progressive low-to-high resolution pretraining, early-stage full-attention checkpoints can be rebased onto sparse attention mid-training, so the VSA2 model doesn’t require sparse pretraining from scratch and can consume whatever full-attention compute has already been invested.

Results

Attention compute: halved vs. the previous VSA, with lower loss [Abstract].
720p attention speedup: 8.9× over FlashAttention-3 [Abstract].
720p end-to-end generation speedup: 4.62× over FlashAttention-3 [Abstract].
Video quality: comparable or better than the full-attention counterpart on the same DiT backbone [Abstract].

(The abstract is the only text available at filing time; per-benchmark tables, VBench numbers, and the quality-vs-sparsity curve are expected in the body but are not accessible via the arxiv HTML landing page.)

Why it’s interesting

VSA2 is the direct successor to the VSA kernel underlying the FastVideo line (FastWan: Generating a 5-Second Video in 5 Seconds via Sparse Distillation, FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup) and shares an author lineage (Peiyuan Zhang, Will Lin, Hao Zhang — Hao AI Lab). It pushes the sparse-attention frontier in two directions the previous generation didn’t touch: (1) per-query variable KV count, which breaks the fixed-K/fixed-p routing assumption shared by SVG2 (Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation) and Sol-Attn (Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification) in different ways; and (2) training-time integration with a specific curriculum (Hard-to-Easy) and a specific stage-boundary mechanism (sparse rebasing), rather than inference-only retrofit. The curriculum claim — training at higher sparsity than inference improves motion quality beyond full attention — is the most unusual: it echoes the over-regularization-helps-generalization story more than the standard “sparsity costs quality” tradeoff this cluster has been tracking.

The sparse-rebasing mechanism complements progressive low-to-high resolution pretraining recipes already filed, making this the first wiki entry where sparse attention and resolution-progressive pretraining are explicitly co-designed.