Towards Video World Models

TL;DR

Position essay by Xun Huang (Luma AI, co-author of Self-Forcing) that first disentangles “world model” into two paradigms — internal world understanding (LWM, JEPA line) vs. external world simulation (video/physics/neural-3D substrates) — then focuses on video world models and argues they must satisfy five properties: causal, interactive, persistent, real-time, physically accurate. Explains why frontier T2V models (Sora, Veo3) don’t yet qualify: bidirectional DiTs are non-causal, which alone blocks interactivity and real-time latency regardless of throughput. Advocates combining causal AR diffusion with few-step distillation (CausVid → Self-Forcing → APT2 lineage) as the path forward, and closes with a taxonomy of complementary substrates (video WFMs vs physics engines vs neural 3D/4D vs hybrids like WonderPlay).

Key claims

“World model” is overloaded: it can mean an internal cognitive model that predicts events at a semantic level (LLMs, LWM, JEPA line) or an external high-fidelity simulator; both can coexist in embodied AI where robots plan with internal WMs while learning inside external WMs [§What Are World Models?].
A true video world model must be causal: bidirectional-attention DiTs let future frames influence past ones, violating temporal asymmetry and predetermining the whole sequence after the first frame — precluding interactivity [§Causal].
LLM-style AR video generators (CogVideo, VideoPoet) are causal but quality-bottlenecked by lossy vector-quantized latents and token-by-token generation; the win is combining diffusion-quality/speed with AR causality [§Causal].
Interactive controllability — the ability to influence the future through actions injected on the fly — is the second hard constraint; the training-data bottleneck for unlabeled internet video is unsolved (Genie’s unsupervised action learning is the current best attempt) [§Interactive].
Persistence (long-enough, temporally consistent generation) is a soft constraint tightened by the real-time requirement: unlike LLMs, video WMs cannot let inference time grow with context; addressed by aggressive compression of distant frames (FAR, FramePack), integration with persistent 3D conditions (GEN3C, WVD, SPMem), or linear-RNN long-range recurrence (Po et al. 2025) [§Persistent].
Real-time means both throughput and latency; real-time throughput is necessary but not sufficient — latency must be sub-second (≤1s live streaming, ≤0.1s gaming, ≤0.01s VR). A non-causal diffusion model that generates t seconds at a time has minimum latency t, so no non-causal model can ever be truly real-time, regardless of throughput [§Real-Time].
The current recipe for real-time video WMs is causal frame-by-frame AR + few-step distilled diffusion — CausVid (AR-DiT + DMD), Self-Forcing (train-time rollout + KV caching, 17 FPS + sub-second latency on 1 GPU), APT2 (interactive finetuning of the real-time base) [§Real-Time].
Physical accuracy is the hardest and most brittle property: Kang et al. show current models achieve combinatorial generalization but fail on extrapolative physics (out-of-distribution velocity/mass). Mitigations include auxiliary objectives beyond pixel reconstruction (VideoJAM’s optical-flow denoising) and dataset curation for rare events (Ctrl-Crash for accidents, PISA for object drops) [§Physically Accurate].
The five properties trade off — real-time and physical accuracy are the two remaining tensions: larger models are more physically accurate but slower; different applications will prioritize differently (entertainment → real-time; robotic learning → physical accuracy, sim can be offline) [§Summary].
Video WMs are one of several substrates for world simulation; conventional physics-engine + renderer stacks (MuJoCo, Isaac, Genesis) and neural 3D/4D methods (Wonderland, WonderWorld, Bolt3D, World Labs, SpAItial) offer distinct strengths, and hybrid recipes (WonderPlay: physics + 3DGS + video diffusion) are a promising near-term direction [§Other Approaches to World Simulation].

Method

The essay is a position/synthesis piece, not an empirical study. It structures the video-WM design space along the five properties in dependency order (causality is prerequisite for interactivity; interactivity is prerequisite for real-time responsiveness; persistence and physical accuracy are soft/hard constraints on top) and slots representative work under each property, drawing the specific line between “video generator” and “world model” at causality.

Key structural argument: because a non-causal T2V model generating t-second segments has minimum latency t for any interactive action, latency-bound real-time interactivity is architecturally impossible without causal AR generation — no amount of engineering fixes it. This makes causal-AR-diffusion + few-step distillation the load-bearing recipe.

Results

No new results — it’s a survey/position piece. Concrete data points cited: Self-Forcing hits 17 FPS with sub-second latency on a single GPU while matching SOTA open-source video diffusion quality. Kang et al.’s PhyWorld shows combinatorial generalization is achievable but extrapolative physics (unseen velocities/masses) fails. Application-specific latency thresholds: ~1s live streaming, ~0.1s gaming, ~0.01s VR.

Why it’s interesting

This is the closest thing on the wiki to a specification for what a video world model must be, from someone (Xun Huang, Luma) who co-authored one of the load-bearing recipes (Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion) and has been directly shaping the space. The five-property decomposition (causal → interactive → persistent → real-time → physical) gives the wiki a vocabulary the existing concept pages have been reaching for piecemeal — Autoregressive Video Generation has been organizing entries around the forgetting/drifting axis (persistence), while World Foundation Models carries the internal-vs-external and generative-vs-latent-predictive splits. This essay collapses both into one dependency graph and stakes out the hard-vs-soft-constraint structure explicitly.

Two normative claims worth flagging: (a) the latency proof — “no non-causal diffusion model will ever be truly real-time regardless of throughput” — cuts against the SORA/Veo/HunyuanVideo/Wan line as a WM substrate; and (b) the hybrid taxonomy framing agrees in shape with Fei-Fei Li’s renderer/simulator/planner functional split (A Functional Taxonomy of World Models) but comes at it from the video-generation side rather than the World-Labs 3D-representation side, providing a useful cross-reference. Contrast with Sitzmann (The flavor of the bitter lesson for computer vision), who argues generative-rollout should replace explicit 3D — Huang’s essay treats hybrids like WonderPlay as first-class, more moderate on the 3D-dissolves position.