Veda: Scalable Video Diffusion via Distilled Sparse Attention

TL;DR

Veda is a tile-based sparse-attention method for video DiTs that reaches 90% sparsity at perceptual parity with full attention on WaverBench1.0 and scales to 95% sparsity for 720P / 241-frame generation (Waver-T2V-12B: 19.4 min → 3.8 min, 5.1× end-to-end). The core move is KL-distilling tile-score estimators against a max-pooled full-attention teacher before top-K thresholding, so the sparsity mask is chosen to preserve the dense softmax geometry rather than cherry-picking individual query-key pairs. Supporting pieces: {avg, max, min} tile summaries (not just mean), per-layer per-head learned tile shapes (pt, ph, pw), and a ThunderKittens sparse-attention kernel that fetches only the selected K/V tiles via TMA with WGMMA-overlapped compute.

Key claims

• Max-pooled full attention defines the oracle tile-level target distribution; retaining the high-response tiles per query tile preserves high-fidelity structure at fixed 90% sparsity, whereas average pooling collapses — Veda’s estimator recovers 66.4% of oracle-selected tiles vs 34.2% for average pooling at the same budget [§“To separate sparsity from mask quality”].
• Prior sparse-attention methods fail at high sparsity with a distinct non-semantic failure mode — water-ripple warping, local geometric distortion, and temporal flicker between adjacent frames — indicating broken spatiotemporal attention connectivity rather than ordinary generator hallucination [§“When sparsity exceeds the moderate regime”].
• Tile-score estimation from compressed queries and keys needs more than the centroid: the dot product decomposes into a mean-product term plus a covariance term, so two tiles with similar means can have very different attention scores; {avg, max, min} is a compact estimator of both centroid and spread, justified by the Bhatia-Davis inequality [§“A tile score must be estimated”].
• Video DiT heads induce different anisotropic metrics on the spatiotemporal manifold under 3D RoPE — a fixed raster (pt, ph, pw) tile layout scatters high-affinity temporal neighbors for one head while merging unrelated spatial tokens for another. Head-Aware Tiling searches per-head tile shapes to minimize sparse-output error, controlling the intra-tile-variance term that bounds score estimation error [§“Video DiT heads are not exchangeable”].
• Top-K selection is discrete, but the attention row inducing it is a continuous entropy-regularized distribution over keys. Veda therefore supervises the estimator with row-wise KL divergence against the max-pooled full-attention teacher before thresholding, aligning Fisher-weighted information geometry on the simplex rather than raw Euclidean score space [§“Top-k selection is discrete”].
• A ThunderKittens kernel fetches only selected K/V tiles via TMA and overlaps sparse gathers with WGMMA compute, turning the structural sparsity mask into measured wall-clock savings — 95%-sparse Waver-T2V-12B 720P / 241-frame generation drops from 19.4 min to 3.8 min (5.1× E2E) [§“A ThunderKittens kernel”].
• Dropping a dominant key does not just locally perturb attention — it changes the softmax normalization constant and renormalizes the remaining keys as P̃j = (Z/Ẑ)·Pj > Pj, motivating oracle-based mask selection rather than per-key score thresholding [§“For one query, attention is the gradient of a Log-Sum-Exp potential”].

Method

Veda factors sparse attention into four load-bearing choices: (1) preserve the oracle attention structure derived from the dense softmax rather than approximate per-key scores; (2) estimate tile scores from {avg, max, min} sufficient statistics instead of centroids alone, controlling the variance term the Bhatia-Davis bound exposes; (3) group tokens with head-aware low-variance tilings — a per-layer per-head search over (pt, ph, pw) chosen to minimize sparse-output error on a validation batch; (4) distill the dense tile-level distribution via row-wise KL before top-K is applied, so the discrete mask selection operates on an already-aligned score geometry.

At inference the materialized binary top-K mask is executed by a dedicated ThunderKittens kernel: TMA-driven sparse K/V tile loads, WGMMA compute overlapped with the gather, no proxy-score map materialized. Reported budgets: 90% sparsity for Waver-T2V-1B at 480P / 81 frames, 95% sparsity for Waver-T2V-12B at 720P / 241 frames.

Results

• 95%-sparse Waver-T2V-12B at 720P / 241 frames: 19.4 min → 3.8 min, 5.1× E2E acceleration.
• 90% sparsity reported at perceptual parity with full attention on WaverBench1.0 (304-prompt human side-by-side benchmark covering object motion, camera motion, and temporal consistency).
• Oracle-tile recall: 66.4% (Veda {avg, max, min} + KL) vs 34.2% (average pooling) at the same 90% sparsity budget — isolating mask-quality from sparsity-budget as the actual lever.
• Per-layer latency reported against FlashAttention-3 at 720P and 480P across Waver and Wan2.1, accounting for MLP, mask preparation, and sparse-kernel execution (measured at the layer level, not inferred from FLOPs).

Why it’s interesting

Veda sits at the junction of three threads already active on the wiki. Against the serving-optimization frontier (Diffusion serving optimization), it is a training-free sparse-attention retrofit in the family of Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation (SVG2) and Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification (Sol-Attn), but inverts their approach: SVG2 clusters Q/K semantically and Sol-Attn thresholds proxy scores on-the-fly, while Veda commits to a fixed, distilled oracle-aligned mask and relies on the kernel to make the structural sparsity cheap. Against the diffusion distillation thread (Diffusion Distillation), it extends what distillation targets — not the step count (DMD / TMD / RMD) and not the decoder (PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion), but the tile-level attention distribution itself, before the discrete mask is realized. The structural sparsity + per-head tile search complements Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing (VSA2), which also pushes trainable sparse attention past inference-only retrofit — Veda’s contribution is on the mask-selection geometry, VSA2’s on the training curriculum. Finally, it’s a concrete new datapoint for the IO-Aware Kernel Design template: the ThunderKittens kernel is the “exploit algebraic structure, don’t materialize the intermediate” instance of the pattern, applied to sparse-attention gathers rather than fused softmax.

See also

• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation — SVG2’s semantic-permutation sparse attention; Veda replaces k-means clustering with KL-distilled oracle-aligned masks under the same sparse-attention budget regime.
• Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification — Sol-Attn folds routing + approximate correction into one online-softmax pass; Veda instead precomputes a static distilled mask offloaded to a dedicated kernel.
• Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing — VSA2 pushes trainable sparse attention into pretraining with a hard-to-easy curriculum; shares the “sparsity as first-class training signal” axis with Veda’s KL distillation.
• Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation — Radial Attention is a static-pattern sparse baseline; Veda learns the mask per-head via tile-shape search instead of hard-coding spatial locality.
• Diffusion serving optimization — Veda contributes a training-free sparse-attention design point to the serving frontier.
• Diffusion Distillation — the KL-distilled tile-score estimator extends “what distillation targets” past step count and decode.
• IO-Aware Kernel Design — the ThunderKittens sparse-attention kernel is a new instance of the exploit-algebraic-structure template.