Introducing Cosmos 3 Edge

TL;DR

Cosmos 3 Edge is NVIDIA’s on-device sibling of the Cosmos 3 world-model family: a 4B-parameter open-weights model (with a 2B Nemotron-based reasoner) that keeps the dual-tower autoregressive + diffusion architecture of the flagship Cosmos 3: Omnimodal World Models for Physical AI but is sized to run on NVIDIA Jetson (including newly-announced T2000 / T3000 modules), DGX Spark, RTX PRO, and GeForce RTX GPUs. It ships in two roles — a small VLM for real-time reasoning over live video, and a post-trained World-Action Model (WAM) that operates at 640×360 observations and produces 32 actions per inference at 15 Hz control on Jetson Thor. NVIDIA also releases Cosmos 3 Edge Policy (DROID) for pick-and-place manipulation together with post-training scripts, and a companion 4-step distillation of Cosmos 3 Super (T2I + I2V) that cuts diffusion from 35–50 steps to 4 for up to 25× faster inference.

Key claims

Cosmos 3 Edge is a 4B-parameter open world model targeted at edge deployment across NVIDIA Jetson (including T2000 / T3000), DGX Spark, RTX PRO, GeForce RTX, and DGX Station, aiming to deliver “data center–level performance on memory-constrained systems” [§ Introducing Cosmos 3 Edge].
Architecturally it retains Cosmos 3’s dual-tower structure — an autoregressive tower for vision + text understanding and reasoning, and a diffusion tower for vision, audio, and action tokens — with separate normalization + MLPs but shared multimodal attention layers, so reasoning tokens and denoised video/action tokens read the same context [§ Two Transformer Towers, One Shared Representation].
Attention pattern is modality-adapted: language uses causal attention while diffusion tokens attend broadly to available context for coherent prediction/generation [§ Two Transformer Towers, One Shared Representation].
Action is a first-class modality — encoded as compact geometric vectors capturing translation, rotation, and manipulation state — with domain-aware projections mapping different embodiments (vehicles’ ego-pose, camera motion, end-effector pose, grasp state) into a single common representation [§ A Common Representation for Action].
In policy mode, Cosmos 3 Edge is bidirectional: given current state it predicts an action and its expected visual consequence, and the same model can infer the action responsible for an observed effect [§ Policy Mode].
As a WAM post-trained on DROID (pick-and-place), it operates at 640×360 observations, generates 32 actions per inference on NVIDIA Jetson Thor, and achieves real-time control at 15 Hz [§ Introducing Cosmos 3 Edge].
Ranked #1 on VANTAGE-Bench for vision analytics among open models of similar (4B) size, and state-of-the-art for robot policy learning at its scale [§ Introducing Cosmos 3 Edge].
Along with Cosmos 3 Edge, NVIDIA ships Cosmos 3 Super 4-Step Distillation (T2I + I2V): a distribution-matching distilled checkpoint reducing diffusion from 35–50 denoising steps to 4, reported at up to 25× faster inference while preserving quality [§ Post-trained samples for improved performance].
Fine-tuning workflow: a small H100 cluster or DGX Station is sufficient to post-train Cosmos 3 Edge for a target workload before deploying to edge hardware; post-training scripts are released alongside the DROID policy [§ Policy Mode].
Roadmap explicitly names interactive world generation, driving-scenario simulation, and robotics policies as the next improvements, plus vLLM-based inference optimization [§ What’s ahead].

Method

Cosmos 3 Edge is a scaled-down instance of the Cosmos 3 Mixture-of-Transformers backbone from Cosmos 3: Omnimodal World Models for Physical AI. Each layer is split into two transformer towers: an autoregressive tower that processes language and ViT-encoded image/video tokens (understanding, reasoning), and a diffusion tower that processes video, audio, and action tokens (prediction, generation, neural simulation). Normalization layers and MLPs are per-tower; multimodal attention is shared, giving diffusion outputs access to the full AR context and vice versa. Attention masking is per-modality: causal for language, broad for diffusion tokens.

Actions are treated as one of the model’s core modalities, not a task head bolted on. Every embodiment — autonomous vehicles (ego pose), cameras (motion), robot arms (end-effector pose + grasp state, from 1-D grippers to multi-finger hands) — is mapped through domain-aware input/output projections into a shared representation that encodes translation, rotation, and manipulation state as compact geometric vectors. This gives the model a bidirectional interface: a scene + goal produces an action and a predicted next frame; a scene + observed next frame produces an inferred action.

The post-trained policy release (Cosmos 3 Edge Policy) is a WAM fine-tuned on DROID for pick-and-place at 640×360 observation resolution, emitting 32-step action chunks per inference. The companion Cosmos 3 Super 4-Step distillation is a separate release path — distribution-matching distillation of the flagship Cosmos 3: Omnimodal World Models for Physical AI Super generator down to 4 denoising steps for T2I and I2V, with post-training scripts released so developers can rerun the distillation on their own domains.

Results

Vision analytics: #1 on VANTAGE-Bench among ~4B open models (no absolute scores disclosed) [§ Introducing Cosmos 3 Edge].
Robot policy learning: reported “state-of-the-art for robot policy learning” at 4B scale, with 32 actions per inference at 15 Hz control on Jetson Thor at 640×360 observations [§ Introducing Cosmos 3 Edge, § Policy Mode].
Distillation: Cosmos 3 Super 4-Step delivers up to 25× faster inference by reducing 35–50 denoising steps to 4 while preserving image/video quality [§ Post-trained samples for improved performance].
Compared to the flagship Cosmos 3 Cosmos 3: Omnimodal World Models for Physical AI, no leaderboard-vs-leaderboard results are given for the 4B Edge model; only bench rankings within the 4B open-model set.

Why it’s interesting

Cosmos 3 Edge is the first filed on-device variant of an omnimodal WFM that keeps the full dual-tower AR+DM architecture — this matters for the World Foundation Models cluster because most existing edge robotics deployments strip a policy out of a larger backbone (see Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots on the runtime side, DimOS — The Agentive Operating System for Physical Space (v0.0.13) on the OS side). Cosmos 3 Edge instead ships the same reasoning + generation + action interface as the flagship Cosmos 3: Omnimodal World Models for Physical AI, just at 4B — sharpening the “WFM as unified backbone” position (Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models) with a concrete on-device datapoint against fragmented VLM+VLA+WAM stacks.

For VLA Models it adds a data point to the recipe board: rather than clean teleop scaling (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), UMI-hour scaling (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training), or advantage-conditioned flow-matching RL (π*0.6: a VLA That Learns From Experience (RECAP)), NVIDIA is betting that a shared WFM backbone post-trained into a specific embodiment (here DROID pick-and-place) produces competitive policies at edge-hardware scale. Complements the Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models pretrained-then-act framing with the actual small-model release. The 4-step distillation of Cosmos 3 Super is the largest reported diffusion-distillation compression ratio in this family, and lands in Diffusion Distillation alongside DMD variants like Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals and Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield.