VideoDeltaNet on MiniMax H3 — Near-Lossless Hybrid Attention for Video Models

TL;DR

VideoDeltaNet (VDN) retrofits MiniMax H3 (33B dense joint-video-audio DiT) into a hybrid attention architecture that reaches near-lossless quality at frontier-linear speed. Each block splits video-video attention into (a) a bidirectional sliding-window softmax branch with 4-way boundary-frame anchors for local detail + global consistency (+3.57% attention density) and (b) a bidirectional linear branch — Video Delta Attention (VDA) — that solves the frame-wise delta rule as a single joint normal equation (I+A)⁻¹ instead of SANA-WM’s frozen-state batch gradient step. VDA is provably non-expansive without SANA-WM’s 1/√U frame-size scaling and adapts each eigendirection to the observed key Gram spectrum, so repeated evidence accumulates without saturating and independent directions aren’t penalized. A three-stage training pipeline (layer-wise alignment → E2E branch adaptation → E2E LoRA co-adaptation) preserves the pretrained softmax path. Result on 8×B200: 14.3s video generated in 11.23s (74.5× over dense single-GPU baseline, 10.7× over dense 8-GPU baseline), with quality claimed nearly indistinguishable from dense H3 and higher than FastVideo’s FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup on the same base model.

Key claims

On MiniMax H3, softmax attention over long token sequences accounts for more than 85% of total runtime, motivating hybrid attention as the primary lever for video-DiT acceleration [§Intro].
Video DeltaNet splits video-video attention into a bidirectional 5-frame sliding-window softmax branch (retaining the original H3 chunking) plus 4-way boundary-frame anchors (first and last frames attend everyone, everyone attends them) — adding only 3.57% attention density while restoring long-range stability [§Local Softmax and Long-Range Linear Memory].
The linear branch uses bidirectional linear attention with forward state Sₜ→ summarizing pre-window frames and reverse state Sₜ← summarizing post-window frames; temporal regions are disjoint from the softmax window so summing readouts double-counts no frame [§Long-Range Linear Memory].
Text conditioning is written into both forward and reverse linear states via the delta rule scaled by 1/2 each at initialization, so summing the readouts counts the text contribution exactly once [§Local Softmax and Long-Range Linear Memory].
Softmax and linear branch outputs are combined via content-dependent sigmoid gates + separate output projections rather than a straight sum, because the softmax normalizer sums over a small local window and can over-weight local context; the linear branch additionally uses RMSNorm + element-wise sigmoid gating following Gated DeltaNet and KDA [§Gating the Softmax and Linear Branch Outputs].
Video Delta Attention (VDA) reformulates the delta rule as a frame-wise joint solve: instead of SANA-WM’s per-token frozen-state batch gradient step, VDA differentiates the joint frame-level fitting objective ∑ β_u ‖S k_u − v_u‖² + ‖S − S̄‖_F² w.r.t. S and solves the resulting normal equation S = (I + A)⁻¹ (S̄ + B) with A = Kᵀ diag(β) K, B = Vᵀ diag(β) K [§Video Delta Attention: A Frame-Wise Solution].
VDA is provably non-expansive without frame-size scaling: for any A ⪰ 0, (I+A)⁻¹ has operator norm ≤ 1, so a single frame update cannot amplify the inherited-state component regardless of key alignment [§Stability of the Inherited State].
This contrasts with GDN/KDA (need per-token unit-key scaling to keep |1 − β_u ‖k_u‖²| ≤ 1) and SANA-WM (needs an additional 1/√U frame-size scale because parallel-token I − A can have λ_max(A) > 2 when many keys align) — VDA needs no frame-size scaling at all because the required contraction comes from the observed Gram spectrum rather than a worst-case rescale [§Why VDA needs no frame-size scaling].
VDA’s (I+A)⁻¹ is correlation-aware in a way SANA-WM’s frame-size rescale cannot be: with U repeated keys, VDA’s effective data weight is Uβ/(1+Uβ) (grows with evidence, capped at 1) vs SANA-WM’s constant β; with U orthonormal keys, VDA writes each direction with full β/(1+β) vs SANA-WM’s β/U [§Correlation-Aware Frame Updates].
Three-stage training pipeline is required because a randomly initialized linear branch perturbs the well-calibrated softmax path: Stage A1 calibrates each linear branch layer-by-layer with Taylor-Calibrate initialization (softmax branch + QKVO + FFN frozen); Stage A2 trains all hybrid blocks end-to-end with the softmax path still frozen; Stage B adds LoRA to QKVO projections and co-adapts hybrid attention + LoRA [§Adapting the Pretrained H3 backbone].
The softmax gate must remain frozen through Stages A1 and A2 — a trainable gate before the linear branch is calibrated gives the optimizer an easier path by over-suppressing the softmax branch instead of learning the linear branch [§Adapting the Pretrained H3 backbone].
Separate learning rate groups per parameter family: higher LR for smaller-fan-in linear-attention parameters (following Chimera’s principle Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers), lower LR for others; LoRA adapters also get higher LR than standard fine-tuning recipes [§Adapting the Pretrained H3 backbone].
Per-block latency on B200: dense H3 332.5 ms → VDA + Sliding Window 192.1 ms → +optimized kernels + FP8 linears 125.3 ms (2.65× per-layer speedup) [§Hybrid Attention Architecture].
On 8×B200, standard Ulysses parallelism reduces latency from 6.46 s/NFE to 1.62 s/NFE (3.99× per-step); placing the softmax branch and VDA branch on separate GPUs (5 GPUs softmax, 3 GPUs VDA, chosen by profiling) via uneven all-to-all further drops latency to 1.405 s/NFE (13.3% beyond Ulysses) [§Parallelization & Few-Step Generation].
Few-step distillation via DMD2 from community MiniMax-H3 turbo LoRA gives 8-step generation (explicitly not 4-step, prioritizing quality); total DiT denoising = 11.23s for a 14.3s video → 74.5× over dense single-GPU baseline, 10.7× over dense 8-GPU baseline [§Parallelization & Few-Step Generation].
Full pipeline: 50-step Dense Attention DiT takes ~13.95 min for a 15s / 768p video on single B200; VDN-H3 with FP8 linears and optimized kernels takes ~5.35 min on single B200 [§Hybrid Attention Architecture].

Method

The base model is MiniMax H3 (MiniMax H3 — Open-Weights Omni-Reference Video-Audio Generation Model Card), a 33B dense single-stream Omni-Transformer over packed text + video + audio tokens. VDN retrofits each attention block into a two-branch hybrid.

Softmax branch. Five consecutive latent frames form a chunk (matching H3’s original architecture). Each chunk attends to itself, the previous chunk, and the following chunk (bidirectional sliding window). Every frame additionally attends to all tokens in the first and last frames (4-way boundary anchors), and both boundary frames attend to every other frame. Text tokens remain globally visible to every frame throughout.

Linear branch (Video Delta Attention). Following KDA, the channel-wise decay gate α produces S̄ = S_{t-1} diag(α). The frame-level fitting problem — one shared state S must fit all U spatial tokens in the new frame while staying close to S̄ — has closed-form solution S = (I + A)⁻¹ (S̄ + B) with A = Kᵀ diag(β) K and B = Vᵀ diag(β) K on unit-normalized keys. Forward and reverse states operate on disjoint pre- and post-window frames; text tokens are written into both states at initialization, each scaled by 1/2. The linear-branch output passes through RMSNorm + sigmoid gate + separate output projection.

Combination. Content-dependent sigmoid gates calibrate each branch, then separate output projections add both results into the residual stream.

Three-stage training. (A1) Freeze softmax branch + QKVO + FFN + softmax gate; Taylor-Calibrate each new linear branch one block at a time. (A2) With softmax path still frozen, train all hybrid blocks jointly end-to-end. (B) Add LoRA adapters to QKV and O projections, unfreeze softmax gate, jointly co-adapt hybrid attention and QKVO LoRA across the full model.

Systems. Ulysses sequence parallelism for the baseline; a 5+3 uneven-all-to-all split places softmax and VDA branches on separate GPUs. FP8 linears + optimized kernels. Few-step LoRA distilled via DMD2 from a community turbo LoRA to 8-step.

Results

Per-block: 332.5 ms → 125.3 ms on single B200 (2.65× per-layer speedup from architecture + kernels + FP8).

Ulysses 8×B200: 6.46 s/NFE → 1.62 s/NFE (3.99× standard); the 5+3 branch-parallel split gives another 13.3% → 1.405 s/NFE.

End-to-end: 14.3s video in 11.23s on 8×B200 → 74.5× over dense H3 single-GPU baseline, 10.7× over dense H3 8-GPU baseline. Single-B200 full pipeline drops from ~13.95 min to ~5.35 min for 15s / 768p.

No numerical quality metrics reported. Comparison is qualitative: side-by-side videos claimed nearly indistinguishable from dense MiniMax H3, and higher quality + better instruction-following than FastVideo’s FastH3 preview (FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup) on the same base model.

Why it’s interesting

First filed VDA-style joint-solve linear attention for video DiTs. The Hybrid Linear Attention cluster has been tracking the SANA-WM (SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer) recipe — parallel-token delta rule with 1/√U frame-size scaling — as the frontier linear-attention primitive for video. VDN replaces the frame-size scaling with a joint normal-equation solve that is provably non-expansive on any A ⪰ 0 and adapts to the observed key Gram spectrum. This is a qualitatively different linear-attention primitive that treats correlated tokens as evidence-accumulation rather than double-counting — the cluster’s standing “which linear primitive wins” open question gets a new datapoint that specifically targets the video regime where spatial tokens within one frame naturally share direction.
Second retrofit-side entry challenging the Sliding-window beats linear attention critique. That paper argued SWA+sinks beats post-trained linear-attention retrofits by 2-10× on NIAH/BABILong for LLMs, casting doubt on the whole retrofit branch. VDN is a retrofit but combines SWA (locally) with a video-specific linear branch, and the ablation-style latency waterfall (332.5→192.1→125.3 ms) shows the linear branch is doing real work beyond what pure SWA would give. Suggests the retrofit critique may not transfer cleanly from LLMs to video DiTs where the recall gap matters less and per-token softmax cost dominates.
Concrete counterpoint to FastVideo’s FastH3 on the same base model. FastVideo FastH3 V1 — 4-Step Sparse Distilled MiniMax H3 for 14x Blackwell Speedup takes the opposite retrofit path — 4-step sparse distillation (VSA + rCM-style) without architectural change — and reaches similar 14× headline speedups on single-B200. VDN commits to architecture change (hybrid attention) + more conservative 8-step distillation and claims better quality preservation. First filed head-to-head on frontier open joint-AV where two accelerator teams have taken opposite bets: distillation-first (haoailab) vs architecture-first (openvdn) on the same base checkpoint.
Systems contribution: uneven branch-parallel all-to-all. The 5+3 GPU split placing softmax and VDA on separate GPUs is a new pattern for the Diffusion serving optimization page — sequence parallelism plus branch parallelism, chosen by per-branch latency profiling. Distinct from the FastVideo / SGLang Diffusion / Sol-Engine framework-level frontier because it exploits the fact that hybrid attention has two branches with different compute characteristics.