LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation

TL;DR

LongLive-2.0 is an NVFP4-based parallel training + inference infrastructure built around a long, multi-shot, interactive autoregressive video diffusion model. Rather than another rollout-drift algorithm, it co-designs Balanced Sequence Parallelism (a teacher-forcing-aware SP layout that pairs clean-history and noisy-target temporal chunks on each rank) with NVFP4 GEMMs in training and W4A4 NVFP4 inference on Blackwell, including NVFP4 KV cache and asynchronous streaming VAE decoding. It also breaks from the Self-Forcing/CausVid lineage by directly fine-tuning a diffusion model into an AR diffusion model — no ODE initialization, no Distribution Matching Distillation — and reaches real-time (4→2 step) generation via standalone LoRA. Reported wins: up to 2.15× training and 1.84× inference speedup, 45.7 FPS on a 5B variant.

Key claims

• The system is positioned as the first NVFP4 training-and-inference system for long video generation — end-to-end NVFP4 spans training GEMMs, W4A4 forward passes at inference, and NVFP4 KV cache [Abstract].
• Balanced Sequence Parallelism co-designs the teacher-forcing layout with SP execution by pairing clean-history and noisy-target temporal chunks on each rank, enabling a natural teacher-forcing mask under SP-aware chunked VAE encoding [Abstract].
• The proportion of GEMM compute that NVFP4 accelerates grows with video length, making the precision win largest exactly where the training bottleneck is hardest [Abstract].
• The pipeline departs from Self-Forcing-series methods by training the AR diffusion model directly, without ODE initialization or subsequent Distribution Matching Distillation (DMD) [Abstract].
• Real-time generation is enabled through standalone LoRA weights that convert a 4-denoising-step model to 2 steps, decoupling the real-time variant from the base training run [Abstract].
• On Blackwell, inference uses W4A4 NVFP4 on the forward pass plus NVFP4 KV-cache quantization; end-to-end throughput is further boosted by asynchronous streaming VAE decoding [Abstract].
• On non-Blackwell GPUs the design falls back to sequence-parallel inference to match Blackwell wall-clock; the quantized KV cache doubles as a way to reduce inter-GPU SP communication volume [Abstract].
• Headline numbers: up to 2.15× training and 1.84× inference speedup; LongLive-2.0-5B reaches 45.7 FPS while remaining competitive on benchmarks [Abstract].

Method

LongLive-2.0 treats algorithm and infrastructure as a single co-design surface. The training side combines two ideas: (1) Balanced Sequence Parallelism — temporal chunks of the long video are split across ranks so that each rank holds both a clean-history chunk and a noisy-target chunk, which lets the standard AR teacher-forcing mask emerge naturally from the SP layout without the cross-rank communication a generic SP scheme would force; and (2) NVFP4 GEMMs throughout — memory cost drops and matmul throughput rises, with the benefit growing with sequence length (because GEMM dominates more of the budget at longer context). VAE encoding is itself chunked to be SP-aware. Crucially, the AR student is not initialized from an ODE-distilled bidirectional teacher (as in CausVid / Self-Forcing / Causal Forcing) and is not aligned with DMD; the diffusion base is fine-tuned directly into a multi-shot, interactive AR diffusion model. A separate, post-hoc LoRA converts the 4-step AR model into a 2-step real-time generator.

The inference side, on Blackwell, runs W4A4 NVFP4 forward passes with the KV cache also stored in NVFP4. Asynchronous streaming VAE decoding overlaps latent generation with pixel decoding to push end-to-end throughput. On non-Blackwell silicon, the system substitutes SP inference for the FP4 wins and uses the quantized KV cache to lower inter-GPU communication.

Results

• Training: up to 2.15× speedup vs. the relevant non-NVFP4 / non-Balanced-SP baseline [Abstract].
• Inference: up to 1.84× speedup [Abstract].
• Throughput: LongLive-2.0-5B at 45.7 FPS while attaining “strong performance on benchmarks” — the paper does not name the FPS-paired benchmark numbers in the abstract [Abstract].
• The training wins scale with video length, since the NVFP4-accelerated GEMM share grows with sequence length [Abstract].

Why it’s interesting

LongLive-2.0 sits in a different operating regime from the rollout-drift work the wiki has been accumulating in Autoregressive Video Generation. Where Context Forcing: Consistent Autoregressive Video Generation with Long Context, Pretraining Frame Preservation in Autoregressive Video Memory Compression, MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation, PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, and FlowAct-R1: Towards Interactive Humanoid Video Generation all attack drift via memory / cache / curriculum design, LongLive-2.0 attacks the training-cost axis of long-AR — making it tractable to fine-tune at long context in the first place — and skips the ODE-init + DMD stack that Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation argue is the right place to fix few-step AR initialization. The implicit bet is that with enough training compute (NVFP4 + SP-aware teacher-forcing), the Self-Forcing → Causal-Forcing line of theoretical correction is no longer needed.

It also extends the NVFP4-training story now developing alongside Attn-QAT: 4-Bit Attention With Quantization-Aware Training (FA4-FP4 forward attention with QAT precision-consistency tricks) and Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan (MXFP8 + DeepEP for DeepSeek-V3 on B200). LongLive-2.0 is the first filed paper that pushes NVFP4 all the way through video AR training plus W4A4 inference plus NVFP4 KV cache as one system — a useful datapoint for IO-Aware Kernel Design when the wiki next gets a Blackwell-quantization concept page.

See also

• Autoregressive Video Generation — LongLive-2.0 is an AR video diffusion model but explicitly skips the ODE-init + DMD stage other entries fix.
• Context Length / Quality Trade-off in Video Generation — adds a sixth operating axis: SP-aware teacher forcing + NVFP4 to make training at long context affordable.
• Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation — Causal Forcing fixes Self-Forcing’s ODE-init theory; LongLive-2.0 sidesteps the whole ODE-init pipeline by direct AR fine-tuning.
• Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation — Causal Forcing++ replaces ODE init with causal consistency distillation; LongLive-2.0’s “direct AR + 2-step LoRA” is a different choice at the same decision point.
• Attn-QAT: 4-Bit Attention With Quantization-Aware Training — sibling NVFP4 training-side co-design, but at the attention-kernel granularity; LongLive-2.0 is the system-wide version.
• Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan — sibling Blackwell-quantization training infra for LLM pre-training (MXFP8 + DeepEP); LongLive-2.0 is the video-AR sibling.
• IO-Aware Kernel Design — NVFP4 GEMM share growing with video length is the same “kernel-bound work that grows with sequence” pattern FA4 targets at the attention level.
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — the underlying NVFP4 attention kernel LongLive-2.0 sits on top of for Blackwell inference.