Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

TL;DR

A unified, training-free attention framework that drops into existing autoregressive video diffusion / world-model backbones and aims to keep dense-attention quality while making long rollouts fast and memory-stable. Three modules: TempCache merges near-duplicate KV-cache entries across frames via attention-derived temporal correspondence (top-1 ANN lookup per query) to bound cache growth; AnnCA prunes prompt tokens per frame by retaining only prompt keys with a nearby latent query in a shared ANN space; AnnSA sparsifies self-attention by ANN-bucketing tokens and restricting each query to keys in the same semantic bucket(s). Reported ×5–×10 end-to-end speedups vs. dense FlashAttention3 while preserving near-identical visual quality and maintaining stable throughput + near-constant peak GPU memory over a simulated 2-minute (2880 frames @ 24 FPS) rollout. Reinterprets attention itself as a fast approximate-nearest-neighbour retrieval problem.

Key claims

Autoregressive video diffusion has three persistent redundancy sources: near-duplicate cached keys across frames, slowly-evolving (largely semantic) queries/keys making many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame [project page §Observations].
TempCache bounds KV-cache growth at inference by retrieving each current-frame query’s top-1 nearest-neighbour key from previous frames via lightweight ANN (LSH / quantized similarity search), grouping keys that track the same content, and keeping a single representative (typically the most recent) [project page §TempCache].
AnnCA accelerates cross-attention by projecting current-frame latent queries and prompt keys into a shared ANN space and computing cross-attention only over prompt tokens with a nearby latent query — pruning is per-frame, not per-sequence [project page §AnnCA].
AnnSA sparsifies self-attention by ANN-bucketing tokens into semantic neighbourhoods and restricting each query to attend only to keys within the same bucket(s), yielding a high-recall sparse attention pattern [project page §AnnSA].
The framework is training-free, drop-in, and “plug-and-play” — no fine-tuning of the underlying AR video diffusion / world model is required [project page §Approach].
Headline simulated rollout: 2-minute / 2880-frame / 24 FPS generation with a KV cache that is not bounded — reported up to ×5–×10 end-to-end speedup vs. Dense FlashAttention3 at near-identical visual quality, with stable throughput and nearly constant peak GPU memory [project page §Abstract, §Side-by-side].
Comparison baselines reported in the side-by-side panels: Dense FA3 (quality reference), SVG1/2 (sparsification baselines — exhibit artifacts and drift in challenging scenes), RadialAttention and FlowCache + RadialAttention (more stable but degrades in challenging scenes) [project page §Side-by-side].
The unifying reinterpretation: attention is recast as a fast ANN-retrieval problem — first retrieve a small candidate key set likely to dominate the dot-products, then compute attention only over those candidates with sparse kernels [project page §Approach].

Method

The starting observation is that an autoregressive video DiT denoises one segment at a time, conditioned on a KV cache that grows monotonically with every generated frame. As context grows, dense attention dominates inference latency and peak memory rises, eventually forcing short context windows that hurt long-range consistency. Rather than retrain the backbone (the standard recipe in Context Forcing / PFP / FlowAct-R1) or distill to fewer steps (Causal Forcing / Causal Forcing++), this work intervenes at the attention layer only and replaces every dense softmax with an ANN-bounded approximation.

Mechanically: for each attention call, an ANN index over keys (locality-sensitive hashing or quantized similarity search) returns a small candidate set per query; sparse attention kernels compute the softmax over only those candidates. The three modules differ in what is indexed and what is pruned. TempCache indexes past-frame keys and uses top-1 retrieval as a temporal-correspondence signal: keys that match the same query content across frames are merged, with the most recent representative retained — turning cache growth from unbounded into bounded by a “distinct semantic tracks” budget. AnnCA projects prompt keys and current-frame latent queries into a shared ANN space; only prompt tokens with a nearby latent query are kept for cross-attention at that frame. AnnSA assigns self-attention tokens to ANN buckets (semantic neighbourhoods) and restricts each query to its bucket(s) — a structured sparsity pattern that exploits the empirical observation that video self-attention is mostly intra-semantic-region. All three are training-free and compatible with existing AR video diffusion backbones and world models.

Results

End-to-end speedup: up to ×5–×10 vs. Dense FlashAttention3 on the simulated 2-minute / 2880-frame / 24 FPS rollout [project page §Abstract].
Long-rollout stability: stable throughput and near-constant peak GPU memory across the 2-minute rollout, whereas prior methods “progressively slow down and suffer from increasing memory usage” [project page §Abstract].
Quality: “near-identical visual quality” relative to Dense FA3; SVG1/2 sparsification baselines exhibit artifacts and drift, RadialAttention is more stable but degrades on challenging scenes [project page §Side-by-side, §Abstract].
Caveat: project page only — no tables, no VBench / PhysicsIQ / MovieGenBench numbers, no per-module ablations published. Speedup figures are reported only against Dense FA3 + SVG1/2 + RadialAttention baselines.

Why it’s interesting

This is the first filed AR-video paper that intervenes purely at inference on the attention operator itself, with no training, distillation, or backbone modification. The three training-time long-AR-video recipes the wiki has filed — PFP’s learned memory encoder, Context Forcing’s KV partitioning + error-recycling teacher, LongVie 2’s multi-stage curriculum, FlowAct-R1’s slot-structured memory bank, plus the Causal Forcing / Causal Forcing++ distillation stack — all require either pretraining a new module or fine-tuning the base model. TempCache + AnnCA + AnnSA targets the same KV-growth and long-rollout drift symptoms by reformulating attention as ANN retrieval. The closest analogue on the discriminative side is Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (RLT), which detects near-duplicate patches in pixel space via L1 thresholding; this paper does the same job on KV-cache entries in key-space via ANN — a key-space generalization of the same temporal-redundancy primitive that should compose with training-time fixes rather than substitute for them. The plug-and-play, no-train claim is what makes it potentially the cheapest of the six axes on Inference-Time Scaling to drop into a video serving stack — but the absence of quantitative benchmark numbers on the project page is a real gap; this needs the paper / code drop before the ×5–×10 claim can be taken as a load-bearing number.