LongLive: Real-time Interactive Long Video Generation

TL;DR

LongLive is a frame-level autoregressive video generation framework that fine-tunes a 1.3B Wan2.1-style short-clip model into a real-time minute-long interactive generator in 32 GPU-days. It threads three ingredients together — a KV-recache mechanism that refreshes cached states on new prompts for smooth on-the-fly prompt switching, streaming long tuning (train-long-test-long) to close the train/inference horizon gap, and short-window attention paired with a frame sink (frame-level attention sink, a video analog of StreamingLLM’s text sinks) to preserve long-range consistency while cutting compute. At inference it sustains 20.7 FPS on a single H100, supports up to 240-second videos, and degrades gracefully under INT8/FP8 post-training quantization. It is the system the wiki’s long-AR-video lineage repeatedly cites as a 3.0s-effective-context baseline.

Key claims

The long-AR-video problem has two coupled bottlenecks: bidirectional diffusion / diffusion-forcing models give quality but are inefficient (no KV cache); causal AR models cache cheaply but degrade in quality on long videos due to memory challenges during long-video training [Abstract].
Beyond static prompts, interactive streaming prompt input is the dynamic-content regime; the hard sub-problem is preserving visual consistency and semantic coherence across prompt transitions [Abstract].
A causal frame-level AR design with three new mechanisms — KV recache on new prompts, streaming long tuning, short-window attention + frame sink — addresses both bottlenecks together [Abstract].
KV recache trades off two failure modes that bracket the prompt-switch problem: no KV cache gives new-prompt adherence but abrupt transitions; plain KV cache gives smooth visuals but new-prompt non-adherence; KV recache (refresh cached states with the new prompt) recovers both [§A, Fig. D].
Streaming long tuning aligns training with the inference rollout horizon — “train-long, test-long” — and is necessary specifically to keep quality from degrading at long context, even with the cache mechanisms above [Abstract].
The frame sink — a small set of frame-level attention sink tokens at the start of the rollout — preserves long-range consistency under short-window attention, by analogy to StreamingLLM’s BOS/early-position sinks but constructed in vision token space where natural global anchors don’t exist [Abstract; cf. author’s follow-up commentary on inference-time vs. training-time sink injection in vision].
Fine-tunes a 1.3B-parameter short-clip model to minute-long generation in just 32 GPU-days [Abstract].
Sustains 20.7 FPS on a single NVIDIA H100; supports up to 240-second videos on a single H100 GPU [Abstract].
Supports INT8 post-training quantization with only marginal quality loss, measured on VBench; FPS reported on a 5090 with INT8 [Table B].
FP8 quantization boosts inference to 24.8 FPS with marginal quality loss [project page].
The interactive setting is evaluated on 60-second videos composed from six sequential prompts; ablations explicitly compare no-cache / cache / recache on the prompt-transition regime [Figs. A, B, D].

Method

LongLive starts from a short-clip diffusion-forcing-style AR backbone (1.3B params, Wan2.1-family) and converts it into a streaming long-video AR generator through a single fine-tuning run. The three core ingredients:

KV-recache. On a new user prompt, the past KV cache is refreshed — keys/values for prior frames are re-encoded conditional on the new prompt before the next frame is sampled. This recovers prompt adherence without losing visual continuity, which a naive cache-flush or a plain cache cannot achieve.
Streaming long tuning (“train-long-test-long”). Rather than training only on short clips and hoping the AR rollout extrapolates, LongLive trains on the same long rollouts it will be evaluated on, using the full streaming inference path (KV cache + recache + frame sink) during training.
Short-window attention + frame sink. Attention is restricted to a short temporal window for compute, while a small number of frame-level sink tokens at the start of the rollout are kept globally attendable. Empirically this restores long-range consistency that a pure short-window attention loses.

At inference the system runs causal AR generation with KV caching; INT8 PTQ on weights/activations gives a roughly 1.9× model-size reduction (Table B) with marginal VBench drop, and FP8 pushes throughput to 24.8 FPS.

Results

Throughput: 20.7 FPS on a single H100 at BF16; 24.8 FPS at FP8 [Abstract, project page].
Long-horizon: up to 240-second videos on a single H100, with VBench scores reported for both short- and long-video settings [Abstract].
Training cost: 32 GPU-days to fine-tune from a 1.3B short-clip model to minute-long generation [Abstract].
Quantization: INT8 PTQ with marginal quality loss, measured on a 5090 GPU [Table B].
Effective context (cross-paper context): later work measuring effective context across streaming-tuning baselines places LongLive at ~3.0s effective context, vs. Infinity-RoPE 1.5s, Rolling Forcing 6.0s, FramePack-F1 9.2s, Context Forcing >20s (Context Forcing: Consistent Autoregressive Video Generation with Long Context Table 2 caption).

Why it’s interesting

LongLive is the paper the wiki’s autoregressive-video-generation cluster has been implicitly building around. It introduced the specific recipe — frame-level AR + streaming long tuning + short-window attention + frame sink + KV cache, on a 1.3B Wan2.1-class backbone — that Context Forcing: Consistent Autoregressive Video Generation with Long Context, PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference, MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation, Pretraining Frame Preservation in Autoregressive Video Memory Compression, and FlowAct-R1: Towards Interactive Humanoid Video Generation then critique or extend. Context Forcing in particular cites LongLive as the 3.0s-effective-context baseline its slow/fast cache + Bounded Positional Encoding pushes past 20s, and PFP positions itself as a memory-encoder alternative to LongLive’s short-window-plus-sink design.

It is also the first entry in the now-explicit “frame sink” line — the video-domain analog of the StreamingLLM attention-sink mechanism (StreamingLLM and the Discovery of Attention Sinks) — where authors observed that visual streams lack the natural BOS/title anchors text has, so sinks need to be trained in during fine-tuning rather than dropped in inference-only. This is the framing the wiki’s StreamingLLM blog write-up refines.

Finally, the paper is direct ancestor to LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation — same NVIDIA EAI / MIT-Han-Lab group — which keeps the frame-sink + streaming-long-tuning skeleton but rebuilds the training stack around NVFP4 and Balanced Sequence Parallelism and drops the ODE-init/DMD distillation pipeline entirely in favor of direct AR fine-tuning. Reading LongLive 1.0 is the cleanest way to see why the LongLive-2.0 abstract claims a system-level shift rather than an algorithmic one.