Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation

TL;DR

Radial Attention is a static sparse-attention mask for video diffusion transformers that translates an empirically observed exponential energy-decay law — post-softmax attention scores drop off as spatial and temporal distance grow — into a corresponding compute-density decay, yielding O(n log n) attention. Each token attends densely within its own frame (and the immediate temporal neighborhood), with the per-frame attention window shrinking by powers of two as the temporal gap doubles, plus a first-frame attention sink. Because the mask only prunes low-energy relations and leaves softmax untouched, the recipe slots into pretrained DiTs (HunyuanVideo, Wan2.1-14B, Mochi 1) with LoRA fine-tuning: up to 1.9× speedup at default length, and 4× longer generation at up to 4.4× lower training cost and 3.7× faster inference vs. dense.

Key claims

• Post-softmax attention scores in video DiTs decay exponentially with spatial and temporal distance between tokens — formalized as a “Spatiotemporal Energy Decay” law $A_{q,k} \le C \exp(-\alpha|f_q-f_k| - \beta\|x_q-x_k\|)$ where $\alpha, \beta$ control temporal and spatial decay [§4.1, Fig. 4].
• The mask is static per generation: token at $(f, x)$ attends to $(f', x')$ iff $|x-x'| \le w/2^{|f-f'|}$ — a diagonal band of width $w$ in the same frame, halved per doubling of temporal distance, with diagonal frequency halved instead once width drops below 1 [§4.2, Eq. 4, Fig. 5].
• Total computation cost scales as $O(n \log n)$ in the number of frames at fixed spatial resolution, derived from the geometric series of band widths [§4.2, Eq. 5–6].
• A first-frame attention sink (every token attends to all of frame 0) is added on top of the radial pattern to preserve quality [§4.2].
• The approximation error is exponentially bounded in the decay rates $\alpha, \beta$, and is provably smaller than SVG’s error bound under the same energy-decay model [§4.2, Eq. 7; §5.3].
• Radial Attention unifies SVG’s separate spatial and temporal head modes into one static mask — the central diagonal band subsumes SVG’s spatial attention while the radially shrinking off-diagonal bands reallocate budget from far frames to near frames [§4.2].
• Block-sparse implementation uses a 128×128 block size [§4.2, “Hardware-friendly block sparsity”]; inference uses FlashInfer, training uses Block-Sparse Attention.
• Because the mask preserves softmax structure, length-extension to 2×/4× video length can be done with LoRA fine-tuning on the Q/K/V/O projections rather than full-parameter training [§4.3].
• At default length on HunyuanVideo 117 frames / 768p: 1.88× speedup, PSNR 27.3, SSIM 0.886, LPIPS 0.114, Vision Reward 0.139 — matches dense (0.141) and beats STA, PowerAttention, SVG on quality metrics at comparable TFLOPs [Table 1].
• For 4× length extension on HunyuanVideo (125 → 509 frames): training time 21.4h vs. 93.6h for full-parameter fine-tuning (4.37× speedup); inference 781s vs. 2895s dense (3.71×); Vision Reward 0.134 beats RIFLEx (0.037) and Long LoRA (0.130) [Table — longer-video generation].
• On Wan2.1-14B 161-frame (2×): 73.6% sparsity, training 14.5h vs. 28.0h full-FT (1.93×), inference 2847s vs. 5735s (2.01×) with Vision Reward 0.145 vs. 0.150 [Table — longer-video generation].
• Linear-attention alternative (SANA) collapses quality on long video — Vision Reward −0.205 / −0.201 vs. Radial 0.126 / 0.110 on HunyuanVideo / Mochi at 2× length — confirming that linear attention is too lossy and the O(n log n) middle ground matters [Table — longer-video generation].
• Length-extension LoRA is compatible with existing style LoRAs (e.g. Studio Ghibli, MJ-style) [§5.2].

Method

The mechanism is a single static block-sparse mask. After flattening (frame, spatial) indices, a token at frame $f$, position $x$ attends to a token at frame $f'$, position $x'$ if and only if $|x - x'|$ lies within a band of width $w \cdot 2^{-|f-f'|}$ (clamped from below to “every $2^{|f-f'|/w}$-th diagonal” once the band would shrink below a single token). Conceptually the attention map is partitioned into diagonal bands indexed by temporal offset: band 0 is dense within-frame, band ±1 is half-density, band ±2 quarter-density, and so on — a “radial” pattern whose total density forms a geometric series. The first frame is left fully connected as an attention sink, following the Sparsity is Cool: Reverse-Engineering MoBA and NSA mechanistic story that block-skipping sparse attention needs explicit sink anchoring. The mask is materialized at 128×128 block granularity for FlashAttention-2-style execution; training uses Block-Sparse Attention kernels and inference uses FlashInfer.

For length extension the pretrained DiT is fine-tuned with LoRA on Q/K/V/O projections (full attention layers, not just a subset), using 2k top-aesthetic/motion-scored OpenVid-1M clips per target length. The same LoRA is shown to compose with downstream style LoRAs in [§5.2].

Results

At native lengths Radial Attention matches dense attention on perceptual metrics (HunyuanVideo: PSNR 27.3, SSIM 0.886, LPIPS 0.114 vs. SVG 27.2 / 0.895 / 0.114) and slightly trails SVG on Wan2.1-14B PSNR (23.9 vs. 23.2 — SVG higher) but wins on SSIM (0.842 vs. 0.825) and LPIPS (0.163 vs. 0.202), at ~1.8–1.9× speedup [Table 1]. For length extension the gap over baselines opens up: at 4× length on HunyuanVideo, Radial reaches Vision Reward 0.134 — best of all methods — while reducing training time 4.37× vs. full-parameter fine-tuning and inference 3.71× vs. dense. Linear-attention substitutes (SANA) catastrophically degrade quality at long video (Vision Reward ≈ −0.2) [Table — longer-video generation]. VBench-Long Subject Consistency, Aesthetic Quality, and Imaging Quality at 4×/HunyuanVideo are 0.973 / 0.623 / 0.672, all best-or-tied across baselines.

Why it’s interesting

Radial Attention is the first published sparse-attention recipe in this space to leverage a continuous energy-decay law rather than a discrete sparsity pattern, and is the direct prior that MonarchRT: Efficient Attention for Real-Time Video Generation explicitly identifies as plateauing at ~50% FLOPs in the few-step autoregressive regime — MonarchRT’s structural critique (“video attention is not reliably sparse, it’s separable + sparse + dense”) is essentially aimed at Radial Attention, SVG, and SLA. So this paper anchors one of the two main axes of the current “structured attention for video DiTs” debate (energy-decay-driven static sparsity vs. Monarch-style structured factorization). It complements MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens (memory-side sparse attention scaling to 100M tokens) on a different scaling axis: Radial scales spatiotemporal attention, MSA scales memory attention. The LoRA-based length-extension also lines up cleanly with the Parameter-Efficient Finetuning story — “only the boundary capacity needs retraining when you change the structural inductive bias” — making this a useful datapoint for thinking about how to extend pretrained Luma video models to longer durations without re-training the full stack.

See also

• MonarchRT: Efficient Attention for Real-Time Video Generation — direct competitor; argues Radial-style static sparsity plateaus and replaces it with structured Monarch factorization for the few-step AR-video regime.
• Sparsity is Cool: Reverse-Engineering MoBA and NSA — mechanistic study of MoBA/NSA sparse attention; provides the attention-sink rationale Radial Attention uses for its first-frame sink.
• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens — sparse attention scaled to 100M-token memory; orthogonal scaling axis to Radial’s per-clip O(n log n).
• Spectral Progressive Diffusion for Efficient Image and Video Generation — different efficiency lever (frequency-progressive denoising) for the same long-sequence video bottleneck.
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — dense-attention kernel co-design; Radial’s wins come from skipping HBM traffic that FA4 still pays for.
• IO-Aware Kernel Design — concept page; Radial is the “static block-sparse mask” instance of the IO-aware family.
• Context Length / Quality Trade-off in Video Generation — concept page; Radial’s energy-decay model is a clean upstream theoretical explanation for why naïve sliding-window or full-context approaches hit walls.
• Parameter-Efficient Finetuning — concept page; LoRA-based length extension is a clean instance of “only fine-tune the boundary capacity when changing structural inductive bias.”
• Project page · GitHub