How Cosmos 3 Helps Physical AI Think Before It Acts

TL;DR

NVIDIA Cosmos 3 is announced at GTC Taipei (COMPUTEX, June 1 2026) as the next generation of the Cosmos world foundation model platform: a single mixture-of-transformers model that combines vision reasoning and multimodal generation across text, images, video, ambient sound and action trajectories. It pairs a reasoning transformer (interprets the scene) with an expert generation transformer (emits physically grounded video, captions, scenario variations, or numerical action data such as joint angles and gripper positions). NVIDIA positions it as the world’s first fully open omnimodel for physical AI, with Cosmos 3 Super and Cosmos 3 Nano available now under the Linux Foundation OpenMDW 1.1 license and an Edge real-time variant coming. The launch is accompanied by the NVIDIA Cosmos Coalition — Agile Robots, Black Forest Labs, Generalist, LTX, Runway, Skild AI — explicitly framed as the open counterpart to closed-flagship interactive world models (Genie 3, Runway GWM-1).

Key claims

Cosmos 3 is a single mixture-of-transformers architecture that pairs a reasoning block (scene interpretation) with a generation block (physically grounded outputs), producing text, video, ambient sound and action trajectories from one model [§Architecture].
It is positioned as an “omnimodel with native action generation” — emitting numerical action data (joint angles, gripper positions, trajectory points) alongside synthetic video, not as separate task heads [§Cosmos 3 Architecture].
Cosmos 3 Nano post-trained policy is reported to lead RoboLab (policies in simulation, language-guided tasks) and RoboArena (policies on DROID robots in real-world environments) [§Policy benchmarks].
Cosmos 3 is the top-ranked open vision language model on VANTAGE-Bench (smart-infrastructure scene understanding) and TAR (traffic anomaly reasoning) [§Vision benchmarks].
Cosmos 3 variants are reported as ranking first on Artificial Analysis open-weights leaderboards and topping Physics-IQ, R-Bench and PAI-Bench for world generation [§World-generation benchmarks].
The release lineup is Cosmos 3 Super (post-training for robotics / AV), Cosmos 3 Nano (deployable policy), and Cosmos 3 Edge (real-time inference, coming) [§Availability].
The NVIDIA Cosmos Coalition is launched alongside the model — explicit named partners include Agile Robots, Black Forest Labs, Generalist, LTX, Runway, and Skild AI [§Coalition].
Cosmos 3 is shipped on build.nvidia.com, Hugging Face (nvidia/cosmos3 collection), GitHub (nvidia/Cosmos), and as NVIDIA NIM microservices, under OpenMDW 1.1 from the Linux Foundation — a single model-centric license covering weights, architecture, documentation, datasets, benchmarks, and code [§Availability, §License].

Method

Cosmos 3 uses a mixture-of-transformers (MoT) architecture: a reasoning transformer first interprets the scene — object interactions, motion, spatial-temporal relationships — and an expert generation transformer uses that interpretation to produce physically grounded outputs. The same model emits text (dense captions, scene-change predictions, root-cause analyses for traffic systems), images, video, ambient sound, and numerical action data (joint angles, gripper positions, trajectory points). Training is on what NVIDIA describes as “one of the largest multimodal physical AI datasets” — billions of samples across text, image, video, sound and action trajectories. Developers fine-tune Cosmos 3 to specialize the model for a particular robot embodiment, camera layout, workspace or task; the NVIDIA GEAR team uses Cosmos 3 as the base for video-action models that run across games, simulations, and real-world robotics, and Agile Robots fine-tunes it to generate action-conditioned policy data at scale.

The post does not disclose parameter counts per variant, training-token totals, or per-modality tokenizer details — those are deferred to model cards and code releases on Hugging Face / GitHub.

Results

Benchmark claims in the announcement (no numbers given, leaderboard positions only):

Cosmos 3 Nano policy: 1st on RoboLab (sim, language-guided tasks) and RoboArena (real-world DROID robots) [§Policy benchmarks].
Cosmos 3 VLM: top-ranked open model on VANTAGE-Bench (smart infrastructure) and TAR (traffic anomaly reasoning) [§Vision benchmarks].
Cosmos 3 world generation: 1st on Artificial Analysis open-weights leaderboards, plus 1st on Physics-IQ, R-Bench, and PAI-Bench [§World-generation benchmarks].

Production examples shown:

Linker Vision uses Cosmos as the VLM backend to analyze live camera streams across smart-city deployments — spatial-context extraction, anomaly surfacing, root-cause analysis across “thousands of feeds” [§Smart cities].
Agile Robots generates action-conditioned robot policy data for humanoids and arms (Thor 3, FR3) [§Robotics].
Image-to-video / audio-prompt demos: “Put all the bananas on the plate” produces a robot policy execution; dashcam image + “high-speed racing event” prompt produces a video continuation [§Demos].

The announcement also lists adopters: Agile Robots, Doosan Robotics, LG Electronics, Samsung, Skild AI (robotics); Li Auto (AVs); Centific, Fogsphere (vision agents).

Why it’s interesting

Cosmos 3 is the first filed datapoint of an open omnimodel claiming to unify vision reasoning, world generation, and native action emission in a single mixture-of-transformers — the architecture pattern shows up as Cosmos’s open answer to closed-flagship interactive world models like Project Genie: Experimenting with infinite, interactive worlds (Project Genie / Genie 3) and Runway GWM-1 (Runway GWM-1 launch broadcast — world model, avatars, audio, multi-shot editing), but with action trajectories as a first-class output rather than a downstream consumer concern. It also escalates NVIDIA’s prior Cosmos stack (NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry — Cosmos Reason 2 + Predict 2.5 + Transfer 2.5 as three separate models) into a single unified backbone — collapsing the “WFM as backbone + VLA as task-head” pattern that the wiki has been tracking. The NVIDIA Cosmos Coalition (with Black Forest Labs, Runway, LTX among the AI partners) is also the first filed datapoint of a closed-flagship video-model lab (Runway, LTX) publicly aligning with an open WFM release — worth watching against Introducing Runway Labs‘s portfolio-incubator framing.