SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

TL;DR

SANA-Video 2.0 is NVIDIA’s 5B/14B hybrid video diffusion transformer, trained from scratch, that matches full-softmax video DiTs in quality while retaining linear-attention’s long-sequence scaling. Two architectural moves carry the paper: Hybrid Linear-Softmax Attention combines gated linear attention (O(N) mixing) with periodic gated-softmax anchors at a 3:1 linear:softmax ratio, and Block Attention Residuals (AttnRes) route completed block summaries into later linear layers to boost deep-layer effective rank by ~12%. On one H100, the 5B pipeline generates a 720p / 5s clip in 13.06s — 3.2× faster than a matched softmax baseline at 720p/60s, and 120× faster than Wan 2.2-A14B — while reaching VBench 84.30 at 480p/13.2s with 40 sampling steps.

Key claims

Hybrid Linear-Softmax Attention interleaves gated linear attention with periodic gated-softmax anchor layers at a 3:1 ratio; reduced-resolution proxy studies establish 25% softmax as the optimal quality-efficiency trade-off [§Abstract].
Block Attention Residuals route completed block summaries into later linear layers, restoring the full-rank token interactions pure linear attention lacks and boosting deep-layer effective rank by ~12% [§Abstract].
Training is from scratch rather than by linearizing a pretrained softmax checkpoint — the hybrid is learned as a unit at both 5B and 14B under a unified architecture [§Abstract].
With 40-step sampling, SANA-Video 2.0 achieves VBench 84.30 in 13.2s at 480p on a single H100 [§Abstract].
The compiled DiT forward pass is 3.2× faster than a matched full-softmax baseline at 720p/60s, and the gap expands with video duration [§Abstract].
Full-stack Sol-Engine optimization (kernel fusion, caching, sparse attention) delivers an additional 3.58× speedup on the hardware-friendly backbone, bringing the 5B pipeline to 13.06s at 720p/5s [§Abstract].
End-to-end, the 5B pipeline is 120× faster than Wan 2.2-A14B on one H100 at 720p/5s [§Abstract].

Method

The backbone is a video DiT with two structural deviations from a full-softmax stack. First, most layers run gated linear attention for O(N)-dominated token mixing across the very long video-token sequence, with gated-softmax anchor layers inserted every fourth block (25% softmax overall). The 3:1 ratio was picked from reduced-resolution proxy sweeps as the quality-efficiency optimum. Second, Block Attention Residuals (AttnRes) — the depth-wise mechanism from Moonshot’s Kimi Team Attention Residuals (AttnRes / Block AttnRes) — Technical Report — route completed block summaries into subsequent linear layers, so the refreshed representations produced at each softmax anchor are reused downstream instead of decaying under PreNorm dilution. The paper reports a ~12% lift in deep-layer effective rank as a direct consequence.

The model is trained from scratch rather than by linearizing a pretrained softmax DiT, instantiated at both 5B and 14B under a unified architecture. Inference is further accelerated by the Sol-Engine full-stack optimization layer — kernel fusion, activation caching, and sparse attention — the same infrastructure filed under Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation and shipped in the SANA — Efficient High-Resolution Image and Video Generation family (NVlabs/Sana, sol-engine branch) sol-engine branch.

Results

Headline numbers (all on one H100 unless noted): VBench Total 84.30 at 480p in 13.2s (40 steps); compiled DiT forward pass 3.2× faster than a matched full-softmax baseline at 720p / 60s, with the gap growing at longer durations. Sol-Engine adds a further 3.58× on top of the backbone speedup, bringing the 5B pipeline to 13.06s at 720p/5s — reported as 120× faster than Wan 2.2-A14B on the same hardware [§Abstract, project page latency chart].

Why it’s interesting

This is the load-bearing datapoint the Hybrid Linear Attention cluster was waiting for on the video-diffusion side. Every prior filed hybrid — Kimi Linear, MiniMax-M1, Jet-Nemotron, Qwen3-Next, Nemotron-3-Super — is an autoregressive LLM; the SANA line has been building the video-DiT analogue in stages (SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer with pure ReLU linear attention, then SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer with frame-wise Gated DeltaNet interleaved with softmax for one-minute world modeling), and SANA-Video 2.0 lands both the 3:1 linear:softmax ratio and the depth-side AttnRes fix in one 5B/14B model trained from scratch — the same 3:1 ratio Kimi Linear picked on the LLM side (Kimi Linear: An Expressive, Efficient Attention Architecture), now validated as compute-optimal for video. Second, this is the first filed external adoption of Moonshot’s Block AttnRes Attention Residuals (AttnRes / Block AttnRes) — Technical Report: the mechanism was released as a Kimi-team tech report a month prior with scaling laws claiming a 1.25× compute-equivalent improvement; SANA-Video 2.0 uses it outside Moonshot’s own Kimi Linear stack, and reports the same PreNorm-dilution fix (deep-layer effective rank +12%) in a diffusion transformer rather than an LLM. Third, the Swayam framing (“attention residuals … related to mHC in some ways, better than mHC”) tracks the intellectual conflict recorded on Attention Residuals (AttnRes / Block AttnRes) — Technical Report and mHC: Manifold-Constrained Hyper-Connections — mHC keeps the linear depth-wise recurrence but constrains it to the Birkhoff polytope; AttnRes replaces it with softmax depth-attention entirely — and SANA-Video 2.0 is the first downstream user to pick a side.