SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer

TL;DR

SANA-Video is a 2B small video diffusion model that replaces softmax attention with ReLU-based linear attention everywhere in the DiT, giving O(N) complexity over the very long token sequences video generation requires. A reformulation of causal linear attention reduces the KV cache for autoregressive long-video generation to a constant-size state (the cumulative key/state sum), enabling minute-length rollouts at fixed GPU memory rather than the linearly growing cache of softmax variants. Continue-pretrained from the SANA-1.6B T2I model in 12 days on 64 H100s (~1% of MovieGen’s training cost), it generates a 5s 720p video in 36s on H100 and 29s on RTX 5090 with NVFP4 quantization, matching VBench Total of Wan2.1-14B at 13–53× lower latency.

Key claims

Linear attention with ReLU(·) feature map plus 3D RoPE placed after the ReLU and applied only in the numerator of the linear-attention formula gives a numerically stable variant that focuses on local regions while keeping O(N) complexity [§3.2, Fig. 3, Eq. 2].
A 1D temporal convolution added as a shortcut to the Mix-FFN at the end of each block recovers the locality that linear attention drops, yielding lower training loss and better motion consistency [§3.2, Fig. 6(a)–(b)].
Causal linear attention output for token N can be re-expressed as a function of two cumulative sums (state $S_N$ and key sum $Z_N$ ); storing these gives a constant-memory KV cache of $O(d^2 + d)$ vs. $O(Nd)$ for causal full attention and $O(Wd)$ for causal local attention [§3.3.1, Eq. 3, Table 1].
Because the temporal Mix-FFN convolution must also be causal at inference, the last frame of the previous block is cached alongside the cumulative sums — the full long-video state is (S, Z, last_frame) of fixed size regardless of total video length [§3.3.1, Fig. 4(b)].
Autoregressive block training uses a monotonically increasing SNR sampler: one block is sampled with the SNR sampler, all later blocks get larger timesteps via propagated probability — smaller sampling space → faster convergence, and the random pivot still sees every block trained [§3.3.2].
Improved Self-Forcing exploits the constant-memory cache to roll out 1-minute self-generated training clips (vs. Self-Forcing’s 5s) and select training windows from them, narrowing the train/inference exposure-bias gap that capped prior streaming-tuning methods [§3.3.2].
DCAE-V — a fine-tuned DCAE with spatial down-sampling 32×, temporal 4×, 32 channels (overall 128× compression) — matches Wan2.1/2.2/LTX VAE reconstruction (PSNR 33.25, SSIM 0.94, LPIPS 0.03) while having the same channel count as the pretrained SANA T2I, enabling fast image→video adaptation [§3.4, Table 3].
At 720p 5s, SANA-Video latency is 36s on H100 vs. 1897s for Wan2.1-14B (52.7× faster) and 116s for Wan2.2-5B (3.2× faster) at comparable VBench Total (84.05 vs. 83.73 / 83.28) [§3.4, Table].
On VBench at 480p 81-frame, SANA-Video-2B achieves Total 83.71 / Semantic 81.35 — the best semantic score in the comparison set, beating Wan2.1-14B (76.11), Wan2.1-1.3B (75.65), Open-Sora-2.0 14B (80.12), and SkyReels-V2 1.3B (74.53) [Table 4].
On I2V at 480p, SANA-Video reaches Total 88.02 / I2V 96.40 — first place on I2V score across all compared models including Wan2.1-14B (92.90), HunyuanVideo-I2V 13B (95.10), MAGI-1 4.5B (96.12) [Table 4].
NVFP4 quantization of QKV/output projections in self-attention, Q/output projections in cross-attention, and 1×1 FFN convolutions (norms / temporal convs / cross-attn KV projections kept higher precision) takes 5s 720p generation from 71s → 29s on RTX 5090 (2.4× speedup) [§5, Fig. 7].
On long-video VBench, SANA-Video reaches Total 83.70 / Semantic 80.78, comparable to Self-Forcing (84.31 / 81.28) and beating SkyReels-V2 (82.67) and CausVid (81.20) [Table 5].

Method

The backbone is a Linear DiT inherited from the SANA T2I architecture, with every attention module replaced by ReLU linear attention. The numerator of the linear-attention formula carries RoPE on both query and key — RoPE(ReLU(Q)) · RoPE(ReLU(K))ᵀ V — but the denominator drops RoPE from one side so the normalizing constant stays positive (ReLU outputs can become negative after RoPE rotation, which destabilizes the softmax-free variant). A 1D temporal convolution is added as a residual branch at the end of the Mix-FFN to inject the locality that softmax attention provides for free but linear attention loses.

The long-video extension rests on a reformulation of causal linear attention. Writing the linear-attention output at token N as a sum of $K_i V_i^T$ over $i \leq N$ , the per-token state $S_N = \sum_{i\leq N} K_i V_i^T$ and key-sum $Z_N = \sum_{i\leq N} K_i$ are sufficient statistics — at the next token only $S$ , $Z$ , and the new $Q$ are needed. This collapses the KV cache to a fixed-size tensor pair plus the last frame of the previous block (needed for the causal temporal convolution). Training is three-stage: VAE adaptation on T2I, continue pre-training from SANA-1.6B with the new temporal modules zero-initialized and trained coarse-to-fine from 192p 2.5s to 720p 5s, then autoregressive block training with the monotonically increasing SNR sampler followed by an improved Self-Forcing variant that uses the constant-memory cache to roll out 1-minute clips at training time. NVFP4 SVDQuant deployment targets only the linear projections; temporal convolutions and cross-attention KV are kept at higher precision to preserve semantics.

Results

The headline efficiency numbers are dramatic: 36s for 5s 720p on H100 (52.7× faster than Wan2.1-14B at 1897s, 3.2× faster than Wan2.2-5B at 116s), 29s for the same video on RTX 5090 with NVFP4 (a single consumer GPU). On VBench Total at matched quality, SANA-Video-2B reaches 84.05 (720p) and 83.71 (480p), outperforming Wan2.1-14B (83.69) and matching Open-Sora-2.0 14B (84.34). The semantic score is where SANA-Video is structurally ahead — 81.35 at 480p vs. 75–80 for the comparison set — which the authors attribute to the SANA-1.6B T2I backbone’s text–vision alignment carrying over through continued pretraining. On I2V the I2V-score reaches 96.40 (first place in the comparison) at 60s latency vs. 493s for Wan2.1-14B. Long-video VBench at the autoregressive setting puts SANA-Video at Total 83.70 vs. Self-Forcing 84.31, SkyReels-V2 82.67, CausVid 81.20.

Why it’s interesting

Two ways to read SANA-Video. First, in the Autoregressive Video Generation / Context Length / Quality Trade-off in Video Generation cluster it adds a sixth operating point alongside MALT’s recurrent memory vector, PFP’s compression encoder, Context Forcing’s partitioned cache, PackForcing’s compressed-mid cache, and LongVie 2’s curriculum: constant-memory cache as an algebraic consequence of using linear attention everywhere, rather than as a designed cache structure on top of softmax attention. This is conceptually upstream of the partitioning vs. compression debate — if linear attention works at this quality bar (and the VBench numbers say it does), the partitioned-softmax-cache lineage is solving a problem the architecture choice can dissolve. Complements MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation‘s fully-recurrent approach (same constant-memory property reached via a different mechanism) and contrasts with Context Forcing: Consistent Autoregressive Video Generation with Long Context / PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference which keep softmax attention and pay the partitioning cost. Second, the improved Self-Forcing recipe — roll out 1-minute self-generated clips at training time because the cache is constant-memory — is a structural reason linear attention helps closing the train/inference gap that Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation identifies as the central problem in the CausVid / Self-Forcing distillation lineage.