Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation

TL;DR

SVG2 (Sparse VideoGen 2) is a training-free sparse-attention recipe for video DiTs (HunyuanVideo, Wan 2.1) whose core trick is to cluster Q/K tokens by k-means semantic similarity and physically permute them into contiguous blocks before running block-sparse attention. This fixes the two failure modes of prior dynamic sparse attention (SpargeAttention, XAttention, SVG1): (a) position-based block aggregation under-represents tokens whose semantics diverge from their neighbors, hurting critical-token identification; (b) scattered critical tokens force tensor-core kernels to pad with non-critical tokens, wasting compute. Add top-p budget allocation, a custom variable-block-size FlashInfer-derived kernel, and a centroid cache that reuses k-means initializations across denoising steps. End-to-end: 2.30× speedup on HunyuanVideo-T2V-13B at PSNR 30.4, 1.58× on Wan2.1-I2V-14B at PSNR 26.5 — outperforming SpargeAttention, XAttention, and SVG1 at every density on the Pareto frontier. NeurIPS 2025 Spotlight.

Key claims

Self-attention in video DiTs is naturally sparse: ~7% of computation is enough to recall the dominant attention scores and preserve near-lossless PSNR, with up to 14× theoretical end-to-end speedup as the upper bound for any sparse-attention recipe in this family [§3.1, Fig. 3].
Prior dynamic sparse-attention methods (SpargeAttention, XAttention) fall significantly below the oracle Pareto frontier at every computational budget; the two causes are position-based clustering (block representations don’t match in-block token semantics) and computation waste (scattered critical tokens force padded tensor-core tiles) [§3.2, Fig. 3, Fig. 4].
Semantic-aware permutation: apply k-means independently to Q and K tokens per attention head and layer, then physically permute the QKV tensors so semantically similar tokens live in contiguous memory blocks. This is mathematically equivalent to the original attention (permutation matrices applied symmetrically to Q and K cancel in the output) [§4.1].
Critical-cluster identification uses centroid-based attention-score estimation: dot product of cluster centroids approximates pre-softmax attention; scores are weighted by cluster size; clusters are selected top-p (sorted descending, accumulate until reaching predefined target). Overhead < 5% of full-attention compute because the number of clusters is small [§4.2, Eq. 1, Eq. 2].
k-means convergence is the latency bottleneck — k-means++ takes ~5 iterations and can rival attention compute time. SVG2 caches centroids across consecutive denoising steps (DiT timesteps are known to be similar) to warm-start k-means; reduces k-means runtime by 3.5× without quality loss [§4.3, Fig. 7(a)].
Custom attention kernel handles dynamic block sizes (cluster sizes vary naturally), built on FlashInfer; existing block-sparse kernels (FlashAttention, FlexAttention, FlashInfer) only support fixed block sizes (typically 128) and would pad away ~52% of compute [§4.3, Fig. 7(b)].
On HunyuanVideo-T2V-13B 720p: density 25.45%, FLOPs 335 PFLOPs (vs 612 dense), 2.30× speedup, PSNR 30.45 — best across SpargeAttn (1.53× / PSNR 27.9), XAttention (1.56× / 28.9), SVG1 (1.91× / 29.2) [Table 1].
On Wan 2.1-I2V-14B 720p: SVG2 reaches 1.58× speedup at PSNR 26.56 / SSIM 0.861 / LPIPS 0.138; SVG2-Turbo trades quality for 1.84× speedup at PSNR 24.51 [Table 1].
On Wan 2.1-T2V-14B 720p: SVG2 at 29.51% density delivers PSNR 25.81 / SSIM 0.854, vs SVG1 22.99 / 0.785 at comparable density 30.25%; SVG2-Turbo at 12.87% density reaches PSNR 23.68 — within 2 PSNR of SVG1 but using ~⅓ the density [Table 1].
Pareto-frontier claim: across a wide range of density budgets on Wan2.1-I2V-14B, SVG2 dominates SpargeAttn, XAttention, and SVG1 at every operating point; equivalently, SVG2 reaches the same PSNR with up to 2.4× lower density [§5.4, Fig. 2].
Ablation: with semantic-aware permutation disabled, identification accuracy and computation efficiency both drop; the dynamic-block-size kernel alone (without k-means clustering) achieves average 52% computation reduction over fixed-block-size FlashInfer [§5.3, §5.5, Fig. 7(b)].

Method

SVG2 runs three coordinated stages around each attention layer’s QKV projection. (1) k-means clustering. Independently per head and layer, run k-means++ on the Q and K activations to form n_Q query clusters and n_K key clusters (the paper uses values that empirically generalize across HunyuanVideo and Wan 2.1). To keep this affordable, a centroid cache persists cluster centroids across denoising timesteps and warm-starts k-means in the next step — DiT timesteps are sufficiently similar that this converges in 1–2 iterations instead of 5. (2) Semantic-aware permutation. Build permutation matrices P_Q, P_K from the cluster assignments and physically rewrite the QKV tensors so that tokens in the same cluster occupy contiguous memory. Critically, the same permutation is applied to K and V so the attention output is mathematically equivalent up to a final inverse-permutation of the rows. (3) Centroid-based top-p selection. Approximate attention scores between Q-clusters and K-clusters as a (centroid-Q × centroid-K) dot product weighted by cluster sizes; sort descending; pick clusters until the cumulative weighted score exceeds a target p (the user-facing knob). (4) Variable-block-size sparse attention kernel. Cluster sizes vary, so the resulting block-sparse attention has non-uniform tile shapes. SVG2 ships a customized FlashInfer-derived CUDA kernel that gathers variable-sized blocks from global memory into SMEM and dispatches to tensor cores, eliminating the 52%-tile-padding waste that fixed-block-size kernels incur.

The first ~10% of denoising steps run dense attention (sparse attention is known to be particularly damaging early in the diffusion trajectory; this skip-rate matches the rule used by SVG1, FasterCache, and DistriFusion).

Results

On NVIDIA H100 with CUDA 12.8: SVG2 achieves a 2.30× end-to-end speedup on HunyuanVideo-T2V-13B at PSNR 30.45 (density 25.45%) and 1.58× on Wan 2.1-I2V-14B at PSNR 26.56 (density 31.28%), both best-in-class across all reported sparse-attention baselines [Table 1]. The Pareto-frontier plot [Fig. 2] shows SVG2 strictly dominating SpargeAttention, XAttention, and SVG1 at every density between roughly 10% and 50%. SVG2-Turbo (more aggressive top-p) hits 1.84–1.89× speedup on Wan 2.1 at PSNR ~24, beating SVG1 at comparable PSNR while using less than half the density. Attention-map visualizations [Fig. 6] confirm that semantic-aware permutation re-organizes scattered critical tokens into dense contiguous blocks, and un-permuting recovers a map that closely resembles the original dense attention. Quantitative VBench metrics (background consistency, motion smoothness) are preserved within 0.1–0.5 points of dense across all settings [Table 1].

Why it’s interesting

SVG2 sits exactly in the sparse-attention space the wiki has been tracking for video DiTs, but pivots the design from static sparsity patterns (SVG1’s spatial/temporal head modes, Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation‘s energy-decay radial mask) to dynamic semantic-aware sparsity that explicitly co-designs the kernel for non-uniform block sizes. This complements Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation from the same lab: Radial Attention chooses one static block-sparse mask up-front per the empirical Spatiotemporal Energy Decay law, whereas SVG2 chooses the mask per attention head per timestep from k-means clusters — Radial trades adaptivity for kernel simplicity, SVG2 trades kernel simplicity for adaptivity. Both win different operating points: Radial dominates the long-video / LoRA-extension regime, SVG2 dominates the per-step Pareto curve at default video length. It also contrasts with MonarchRT: Efficient Attention for Real-Time Video Generation‘s structural critique that video attention is “separable + sparse + dense, not reliably sparse” — SVG2’s response is operational rather than theoretical: dynamic semantic clustering recovers the missing structure rather than imposing a Monarch factorization on top of it. The IO-aware angle is the variable-block-size kernel, which adds a third design point to the IO-Aware Kernel Design template — alongside FlashAttention’s softmax-over-blocks and FlashSampling’s argmax-over-partitions, SVG2 contributes “tile-shape variability is a first-class kernel concern when sparsity is data-dependent.”