Cosmos 3: Omnimodal World Models for Physical AI

TL;DR

Cosmos 3 is NVIDIA’s omnimodal world-model family for Physical AI — a single Mixture-of-Transformers (MoT) architecture that jointly processes and generates language, image, video, audio, and action tokens, subsuming VLMs, T2I/T2V/I2V generators, world simulators, and VLA/world-action policies under one backbone. The model splits each layer into an autoregressive (AR) tower for understanding and a diffusion (DM) tower for generation, coupled via dual-stream joint attention, with modality-specific encoders (ViT for visual understanding, Wan2.2 VAE for visual generation, audio VAE, and a unified 9D-rotation + grasp action representation). Cosmos 3 is released open under OpenMDW-1.1 in Super and Nano variants alongside five large synthetic SDG datasets (PhyxSim, RobotSim, DriveSim, SynHuman, Warehouse) and the Cosmos-HUE evaluation benchmark. The post-trained Super variants are reported as the best open-source Text-to-Image and Image-to-Video models on Artificial Analysis and the best policy model on RoboArena at submission.

Key claims

Cosmos 3 unifies VLMs, image generators, audio-visual generators, policy / world-action models, and forward/inverse dynamics models in a single MoT backbone without architectural modifications across modes [§1, Fig. 1].
The architecture splits each transformer layer into a dual-tower AR + DM structure: AR tokens (language + ViT-encoded image/video for understanding) route to one parameter set, DM tokens (VAE-encoded image/video, audio, action) route to a separate parameter set, and the two interact via dual-stream joint attention each layer [§2.3].
Token arrangement is uniform across tasks: AR subsequence first, then DM subsequence, with clean conditioning tokens before noisy diffusion tokens and ordered by vision → audio → action within each block [§2.2.1].
Action is treated as a core modality, not a head: a unified 9D ego-pose + 9D effector-pose + grasp-state representation (1D gripper or 15D fingertip) covers AVs, cameras, single-arm, dual-arm, humanoid, and egocentric embodiments, mapped via domain-aware input/output projections sharing the MoT backbone [§2.1.3, Fig. 3].
Visual understanding uses a jointly trained ViT with 16×16 patches plus DeepStack aggregation and interleaved text timestamps; visual generation uses the frozen Wan2.2-TI2V-5B VAE (4× temporal, 32×32 spatial compression) projected by a linear layer [§2.1.1].
Cosmos3-Super outperforms specialized open-source baselines and is competitive with closed models across reasoning and generation: 73.7 general reasoning, T2I 91.36 (vs Qwen-Image-2512 84.25, Gemini 3 Pro Image 90.85), Image2Video 82.8 (vs Veo-3.1 82.6, Wan2.2-A14B 81.3), audio 7.31, robot policy 26.0 [Table 1].
The Reasoner can serve as a prompt upsampler for the Generator (agentic upsampling templates released), and ablations show the Reasoner improves Generator outputs [§6.3.2, §E.1, App. B.7].
Five open synthetic datasets cover physical-interaction scenes, embodied robot scenes, AV scenarios, digital-human scenes, and warehouse operations, with an ablation isolating the contribution of each SDG split to downstream generation/policy quality [§3, App. C].
The training stack uses vLLM-Omni as the generator inference framework — the same disaggregated multimodal serving stack filed separately as vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models — alongside vLLM/TensorRT-LLM for the reasoner [§5.3].

Method

Cosmos 3 places all modalities into a single token stream split into two subsequences. The AR subsequence carries language tokens plus ViT-encoded image/video tokens for understanding and reasoning; the DM subsequence carries Wan2.2-VAE image/video tokens, audio-VAE tokens, and action tokens, and is generated via iterative denoising. Each transformer decoder layer is a dual tower: AR tokens use one set of parameters, DM tokens use a separate set, but the two streams share a joint attention operation each layer so that diffusion outputs are conditioned on the entire AR context (and vice versa for understanding tasks that benefit from generation-side features).

Modality-specific encoders front the backbone: a jointly trained ViT for visual understanding (16×16 patches, DeepStack aggregation, interleaved text timestamps following Qwen3-VL), a frozen Wan2.2-TI2V-5B VAE for visual generation (4× temporal / 32×32 spatial, with a 2×2 patch merge then a linear projection), a frozen audio VAE at 25 tokens/sec stereo 48kHz, and the unified 9D-rotation + grasp action representation with domain-aware projection matrices per embodiment family. Multimodal position embedding combines a per-modality position-index allocation with an absolute temporal modulation for video/audio synchronization.

The model ships in Super and Nano sizes with separate post-trained checkpoints for Text-to-Image, Image-to-Video, and DROID robot policy. Generator training proceeds in four stages — pre-training, mid-training, T2I post-training, I2V post-training — with a dedicated robot-policy post-training pass on DROID for the world-action mode. Five large SDG datasets (PhyxSim / RobotSim / DriveSim / SynHuman / Warehouse) seed generator training; an ablation in App. C.7 isolates each split’s effect.

Results

General reasoning: Cosmos3-Super 73.7 vs Gemini 3.1 Pro 77.5, Qwen3-VL-32B 72.8, Gemma-4-31B 69.8 [Table 1].
Robotics reasoning: Super 57.8, Nano 55.1; smart-infra reasoning Super 62.6 / Nano 61.0; driving reasoning Super 79.3 / Nano 76.0 [Table 1].
Text-to-Image: Super post-trained 91.36, beats Qwen-Image-2512 (84.25) and closely behind Gemini 3 Pro Image (90.85) [Table 1] — claimed best open-source T2I on Artificial Analysis at submission.
Text-to-Video: Super 80.0, Nano 79.4, vs Veo-3.1 79.1, Wan2.2-A14B 78.0 [Table 1].
Image-to-Video: Super 82.8, Nano 82.7, vs Veo-3.1 82.6, Wan2.2-A14B 81.3 [Table 1] — claimed best open-source I2V on Artificial Analysis.
Audio generation: Super 7.31, Nano 7.34, vs Veo-3.1 7.45 [Table 1].
Forward-dynamics robot benchmark: Super 26.0 post-trained, Nano 25.5 post-trained, vs Ctrl-World 23.0 [Table 1].
DROID robot policy: Cosmos3-Nano-Policy 39.7, vs π0.5 28.1 [Table 1] — claimed best policy on RoboArena at submission.
Ablations in App. E demonstrate (i) the Reasoner improves the Generator, (ii) explicit FPS control matters, (iii) audio data improves pre-training, (iv) action modes synergize, (v) video-action consistency is non-trivial to maintain.

Why it’s interesting

This is the first filed open-weights world-foundation-model release that bundles understanding, generation, and action under one MoT backbone with a single tokenization and training recipe — the explicit antidote to the fragmented “VLM + VLA + WAM + video model” stack the wiki’s World Foundation Models cluster has been tracking. It directly extends NVIDIA’s CES 2026 NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry omnibus drop (Cosmos Reason 2 / Predict 2.5 / Transfer 2.5 as separate models) into a single unified successor, and it ships the full SDG dataset suite that supports the Synthetic Training Data thesis at physical-AI scale. Architecturally, the dual-tower AR+DM-with-joint-attention layer is the same family pattern as Dual-stream diffusion transformer (Waver, SkyReels-V4) but generalized beyond audio+video to include action and reasoning streams, and the MoT routing places it in the MoE Routing Design design space without expert-router complexity. The post-training-into-DROID-policy result also makes it the closed-loop counterpart to Evaluating Gemini Robotics Policies in a Veo World Simulator — where Google DeepMind specialized Veo into an action-conditioned simulator; here NVIDIA goes further by training the policy inside the same model.