Interactive World Simulation with Cosmos for Physical AI

TL;DR

NVIDIA presents a shared post-training framework that converts the bidirectional Cosmos 3 world foundation model into interactive streaming simulators for Physical AI — exposing the WFM to actions, scene controls, or source video as they arrive and emitting observations incrementally. The framework combines temporal-causal attention over video chunks, teacher-forced causal training, causal consistency distillation, DMD with Self-Forcing, and streaming KV-cache runtime optimizations into one recipe, then instantiates it three ways: Cosmos3-Nano-Sim-Bimanual (action-conditioned robot manipulation), Cosmos3-Nano-Sim-Transfer (control-guided single-view video transfer), and Cosmos3-Nano-Sim-Auto (synchronized multi-view AV with LiDAR). All four open-weight checkpoints and the Cosmos-Framework code are released. For Luma, this is the sharpest filed instance of the bidirectional → causal post-training stack applied end-to-end to a frontier open WFM across three different simulator domains in one report. [§1, §3, §5–7]

Key claims

The shared recipe is a four-step pipeline on top of Cosmos 3: bidirectional adaptation (domain post-training with full attention) → causal training with teacher forcing (bidirectional → block-causal under temporal-causal attention) → causal consistency distillation with teacher forcing (student samples under causal attention, teacher provides single-step targets) → distribution matching distillation with self-forcing (student rolls itself out, two trainable score networks adapt as training proceeds) [§3, Fig. 7].
Temporal-causal attention is enforced on chunks of latent time steps rather than individual frames; a chunk bundles clean controls U_j and jointly-denoised visual targets V_j, so the control group and the generated frames it governs stay in the same prediction unit [§2.2].
The clean history retained before predicting chunk j is represented as the layerwise K/V cache; new chunks are appended to this cache after completion, and streaming inference commits the K/V in latent-time order within each completed chunk [§2.2, Eq. 2].
Multimodal 3D-RoPE (temporal, height, width) is inherited from Cosmos 3; action tokens use 1/4 the temporal rotary increment of vision tokens so that the last action in a chunk shares its temporal coordinate with the corresponding vision latent frame [§2.1, §2.2].
Cosmos3-Nano-Sim-Bimanual predicts future visual dynamics for camera navigation + robot manipulation from a streaming sequence of actions, with domain-specific action projections for camera 6-DoF and end-effector transforms [§5].
Cosmos3-Nano-Sim-Transfer supports single-view video transfer where state is maintained externally by a simulator; clean source-latent frames (or structured spatial controls like depth, segmentation, edges, blur) form the per-chunk control group and remain accessible as conditioning [§2.2 Fig. 2(b), §6].
Cosmos3-Nano-Sim-Auto jointly generates synchronized multi-view RGB videos and LiDAR for autonomous-driving simulation; the architecture extends Cosmos 3’s generator tower to multi-view + joint LiDAR output with configurable cross-view attention visibility rules [§2.1, §7].
Four open-weight checkpoints are released on Hugging Face (Cosmos3-Nano-Sim-Bimanual, Cosmos3-Nano-Sim-Transfer, Cosmos3-Nano-Sim-Auto, plus Cosmos3-Nano-Transfer-Auto), with code at github.com/nvidia/cosmos and github.com/nvidia/cosmos-framework [front matter].
Sequence packing from Cosmos 3 is preserved — tokens from different recordings remain isolated across device partitions through boolean attention masks over query/key indices, so causal conversion coexists with the shared-buffer training recipe inherited from the base model [§2.2].
The framework’s purpose is to make Cosmos 3’s broad visual knowledge usable throughout an ongoing interaction — supporting both prediction of visual consequences of actions and generation of observations guided by an external simulator’s physical state [§1].
Policy-driven simulator utility is scoped out of the report — quality, control fidelity, and runtime are evaluated under each example’s stated protocols, but policy-driven closed-loop validation is deferred as separate work [§1].

Method

The architectural backbone is Cosmos 3’s Mixture-of-Transformers: a reasoner tower (causal self-attention over text) + a generator tower (iterative denoising over visual, control, and denoising tokens), coupled via dual-stream joint attention where generator queries read reasoner keys/values but not vice versa. The simulation framework’s primary architectural change is replacing the generator tower’s bidirectional temporal attention with block-causal attention over chunks: each chunk’s visual targets are jointly denoised and can read clean history + aligned clean controls, but cannot read any future chunk. Within a chunk, clean action tokens or clean source-video latents remain visible to the generator’s noisy targets throughout denoising, and video-transfer layouts can further restrict token-role visibility (Fig. 3).

The four-step post-training recipe keeps Cosmos 3’s weights continuous through conversion: Stage 1 adapts the bidirectional base to the target domain, Stage 2 adds temporal-causal masking with teacher forcing (clean reference branch vs. noisy target branch separated), Stage 3 runs causal consistency distillation under the new attention mask so student samples match teacher single-step outputs, and Stage 4 applies DMD with self-forcing where the student rolls itself out and two trainable score networks learn to match its distribution to the data distribution as training proceeds. Streaming inference maintains the generator’s K/V cache across chunks, with history eviction/reselection policies configurable per example; action-conditioned paths refresh clean K/V in latent-time order within each completed chunk, while multi-view paths follow the clean-history visibility rules configured per-view in Sec. 7.1.

The three example post-trainings specialize controls, data, and losses: Sim-Bimanual uses relative camera + end-effector transforms as action tokens; Sim-Transfer encodes source video or structured spatial controls (depth, segmentation, edges, blur) at the target’s frame rate and temporal compression; Sim-Auto extends the layout to multi-view RGB + LiDAR with configurable cross-view attention. Full method bodies (data preparation, inference optimization, evaluation metrics, and per-example results) occupy §§5–7 of the report and were truncated at fetch; the shared architectural + distillation stack and its three endpoints are the load-bearing contribution.

Results

The report publishes four open-weight checkpoints on Hugging Face and two open-source code repositories (Cosmos core + Cosmos-Framework). Per-example quantitative results are reported in each of §§5.5, 6.5, 7.5 under example-specific protocols — Sim-Bimanual on action-conditioned prediction quality + control fidelity + runtime, Sim-Transfer on video-transfer quality under source-control conditioning, Sim-Auto on multi-view + LiDAR joint prediction with the full Cosmos3-Nano-Transfer-Auto combined checkpoint also released. Full result tables lie in sections beyond the ~30KB fetch window; the architectural + recipe disclosure plus the four open-weight checkpoints are the headline reproducibility contribution for Luma’s purposes.

The report explicitly scopes out closed-loop policy evaluation: quality, control fidelity, and runtime are evaluated under each example’s stated protocols, with policy-driven simulator utility requiring separate closed-loop validation [§1]. This is a deliberate design choice — the WFM is positioned as a reusable interactive-simulation substrate for Physical AI, with downstream policy-evaluation validity left to the training/evaluation harnesses that consume it.

Why it’s interesting

This is the sharpest filed instance to date of the entire bidirectional → causal → distillation pipeline that Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation (Causal Forcing) formalized and Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation (Causal Forcing++) sharpened, applied end-to-end to a frontier open WFM (Cosmos 3: Omnimodal World Models for Physical AI Cosmos 3) with three specialized endpoints released in one report. It’s a direct open-source alternative to the Odyssey-3 Pro architecture disclosure (Meet Odyssey-3: Our Most Powerful Foundation World Model) at the same stage of the recipe — Odyssey-3 ships closed weights with teacher-forced AR extension + DMD + adversarial distillation; Cosmos ships open weights with teacher-forced AR extension + causal CD + DMD-with-self-forcing. Together with SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models SolarWM (which runs the same three-stage recipe across four unrelated backbones) and Atlas: A World Model for Spatial Intelligence Atlas (which AR-conditions on spatial pose rather than temporal order), these form the strongest filed cluster of evidence that the causal-distillation + chunk-level teacher-forcing stack is the shared recipe for frontier interactive WFMs — regardless of backbone or output modality.

The three-way specialization (robot manipulation, controllable video transfer, multi-view AV + LiDAR) also lands a complementary deployment pattern vs. the WAM / WFM-as-policy line (Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2, EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data EgoWAM, Long-WAM: Scaling the Context of World-Action Models Long-WAM): Cosmos positions the simulator as the environment a downstream policy interacts with, rather than as the policy itself. This complements rather than competes with the WFM-as-policy framings tracked under VLA Models.