Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models

TL;DR

NVIDIA Technical Blog post (authored by Moritz Reuss, drawing on his earlier “State of VLA” ICLR’26 essay) that lays out the modern World-Action Model (WAM) landscape as the second major recipe for robot foundation models alongside VLM-backbone VLAs. The post defines WAMs (policies built on pretrained video / world-model backbones that predict both future states and actions), argues the WAM revival is driven by VLAs hitting a language-to-action grounding wall, and proposes a three-axis design space (paradigm × action integration × architecture) under which it maps current systems including DreamZero, LingBot-VA, Cosmos Policy, mimic-video, Sereact Cortex 2.0, DVA, VPP, Unified Video Action, and Fast-WAM. The author’s bet: neither pure VLA nor pure WAM wins — the long-run robot foundation model is a hybrid, and which formulation of WAM wins is the open question.

Key claims

• The field has converged on two representation bets for generalist robot foundation models — VLM-backbone VLAs (Pi-0, Pi-0.5, GR00T N1, Xiaomi Robotics, Being-H0.5) vs WAMs (DreamZero, LingBot-VA, Cosmos Policy, DVA, Sereact Cortex 2.0, mimic-video) — and the backbone choice cascades through the entire training and evaluation pipeline, so most teams will prioritize one direction first [§“The two representation bets for generalist policies”].
• Modern VLAs hit a grounding wall because adapting VLM representations for action generation degrades pretrained capabilities — VLM2VLA frames it as catastrophic forgetting during VLM→VLA transition, and Knowledge Insulation isolates action-expert gradients from the VLM backbone to preserve language/vision priors [§“The grounding gap”].
• Three hypotheses for why video backbones are attractive policy priors (the author flags these as recurring claims, not proven results): (i) predicting future world changes correlates with generating the necessary actions because inverse dynamics is often easier than direct action prediction; (ii) video pretraining provides language↔physical-change grounding from web-scale data robot demos can’t match; (iii) video data regularizes robot policies — co-training with a video-prediction objective during fine-tuning helps in DreamZero and Fast-WAM [§“Core hypotheses for WAMs as policy representations”].
• Veo 3.1 zero-shot probe on a DROID-setup toaster task: given a single context frame and a text instruction (“push lever, then pick up orange”), Veo generates smooth, sequenced rollouts on a one-shot prompt with no robotics fine-tuning — but the pinch gripper morphs into a four-fingered hand and the fixed-base arm is reimagined as a different robot, so the prior is useful but not reliable enough for control [§“A quick experiment: how much does a frontier video model already ‘understand’ about robot manipulation?”].
• Three-axis WAM design space (axes are not fully independent): (a) paradigm = what the model predicts and how predicted video maps to actions — inverse dynamics (UniPi, LingBot-VA, DVA, VPP, mimic-video) vs joint prediction (GR-1, DreamZero) vs representation-only; (b) action integration = how actions enter the model — standalone tokens vs image-shaped targets vs compressed latent actions/plans; (c) architecture = how components compose — monolithic transformer (DreamZero) vs Mixture-of-Transformers with modality-specific experts coupled by shared attention (LingBot-VA) vs hierarchical video-then-action pipelines [§“Understanding modern WAMs: Core formulations”, Fig. 8].
• UniPi was the first modern WAM recipe in 2023 (CNN-based Imagen-Video-era video diffusion + separate inverse-dynamics CNN), but pretraining cost (~167 ZFLOPs estimate) put it outside reach of most labs — what changed is the availability of open DiT-based video backbones (Wan, Cosmos) that can be fine-tuned rather than pretrained from scratch [§“Inverse dynamics: Predict the future, then infer the action”].
• LingBot-VA is the modern scaled inverse-dynamics WAM: fine-tunes Wan 2.2-5B with 16k hours of cross-embodiment robot data, runs causal on long visual histories for closed-loop rollouts (not open-loop video gen), uses a Mixture-of-Transformers with two-way coupling — video conditions actions and generated actions condition video [§“Inverse dynamics: Predict the future, then infer the action”, Fig. 11, Table].
• DreamZero is the modern scaled joint-prediction WAM: initialized from Wan 2.1-I2V-14B-480P, denoises video and action tokens together in one monolithic DiT — no separate inverse-dynamics module, action is just another generated modality [§“Joint prediction: Learn video and actions together”].
• GR-1 (2023) made the key conceptual shift behind joint-prediction WAMs: used video to learn better policy representations, not just better visual encoders (vs R3M, Voltron) — on CALVIN ABC→D reached 3.06/5 average sequence length when prior methods stayed below 1.0 [§“Joint prediction: Learn video and actions together”, Fig. 14].
• Author’s bet: WAMs will become the second major recipe for robot foundation models alongside VLM-based VLAs, but the likely winner is neither pure VLA nor pure WAM — a hybrid combining elements of both [§“Introduction”].

Method

The post is a structured taxonomy + position essay, not a system paper. The core analytical move is the three-axis WAM design space (Fig. 8): (1) paradigm distinguishes inverse-dynamics (video first, then action decode) from joint-prediction (video and action denoised together) from representation-only (video model used purely as an encoder, no video generation at inference); (2) action integration distinguishes default action tokens from action-as-image (so the video backbone can natively denoise them) from latent actions / plans (LAPA-style compressed action codes); (3) architecture distinguishes monolithic transformers from Mixture-of-Transformers expert stacks coupled by shared attention from hierarchical video-then-action pipelines. Each axis is illustrated by pairing an early paper (UniPi 2023, GR-1 2023) with a modern scaled-up instantiation (LingBot-VA fine-tuning Wan 2.2-5B, DreamZero fine-tuning Wan 2.1-I2V-14B). The post also includes a hands-on probe with Veo 3.1 zero-shot on a DROID toaster task to make the “video prior already encodes useful robot-object interaction structure” hypothesis legible. The author maintains a substantial inline glossary covering VLA / WAM / VLM / video backbone / world model / grounding / inverse dynamics / joint prediction / action chunk / MoT / DiT / VAE / Wan / Cosmos / DROID / RoboArena / RoboLab / CALVIN / LIBERO / RoboTwin / FAST / BEAST / VPP / LAPA / OOD / FLOP / ZFLOP / H100 / BF16 / I2V — making this filed page usable as a vocabulary reference for the cluster.

Results

No new experiments beyond the Veo 3.1 zero-shot probe (qualitative only — no quantitative head-to-head). The post’s “results” content is its synthesis: a side-by-side UniPi-vs-LingBot-VA table (backbone, latent video VAE, action expert, action-video coupling, robot training scale), a CALVIN ABC→D chart re-drawn from the GR-1 paper and Xiaomi-Robotics-0 2026 numbers, and the three-axis taxonomy diagram. The body of the post (truncated at fetch time) walks through the remaining axes (action integration, architecture, plus a likely “VLA baselines” comparison and “what’s next” section) but those sections were not reached in the 30KB fetch window.

Why it’s interesting

Most directly relevant cross-link: this post is the clearest filed taxonomy of the WAM space and is the natural map for the wiki’s growing WAM/VLA literature. It explicitly catalogs many already-filed wiki entries: LingBot-VLA: A Pragmatic VLA Foundation Model / Advancing Open-source World Models (LingBot-World) (LingBot-VA), Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners (DVA, the inverse-dynamics WAM), Cosmos 3: Omnimodal World Models for Physical AI (Cosmos 3 / Cosmos Policy, the monolithic joint-prediction WAM), 2410.24164-class Pi-0 and Pi-0.5 references in the VLA-backbone branch, and π*0.6: a VLA That Learns From Experience (RECAP) in the same VLA-backbone family. It also names neighbours the wiki should consider filing — Sereact Cortex 2.0 (Sereact’s WAM bet), mimic-video (Mimic Robotics), Video Prediction Policy (VPP), Unified Video Action Model, Fast-WAM, GR-1 (the 2023 joint-prediction foundation), VLM2VLA, and Knowledge Insulation. Sharp contrast with A Functional Taxonomy of World Models: both posts try to taxonomize the same space in mid-2026, but Fei-Fei Li / World Labs’ three-way split (renderer / simulator / planner) operates at the purpose level (what is the WFM for), while Reuss’ three-axis grid operates at the architectural level (how is a WAM built). They are complementary maps. Also complements Robots Need More than VLA and World Models (Bou-Ammar’s position that VLA+WMs is insufficient and four extra interfaces are needed) — Reuss is arguing inside the VLA-vs-WAM frame that Bou-Ammar is arguing past. Useful for the team because Luma’s robotics-adjacent video-generation work sits exactly at the WAM intersection (large pretrained video backbones being fine-tuned for downstream tasks), and this post is the most recent and most concrete public articulation of why that lineage is a 2026 paradigm shift rather than a 2023 dead end.

See also

• VLA Models — the concept page this post is the strongest external taxonomy for; the VLM-backbone vs WAM-backbone split here is the same split Embodied-R1.5 / π*0.6 / Spirit-v1.5 vs Cosmos / LingBot-VA argue from inside the wiki.
• World Foundation Models — the WFM cluster page; WAMs are one of the named deployment patterns for WFMs (alongside WFM-as-backbone-for-VLA and WFM-as-reward).
• A Functional Taxonomy of World Models — complementary mid-2026 WFM taxonomy from World Labs (renderer / simulator / planner) operating at the purpose level rather than the architecture level.
• Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners — DVA, the cleanest filed instance of the inverse-dynamics WAM paradigm Reuss describes (causal video model as policy + inverse-dynamics translator).
• Advancing Open-source World Models (LingBot-World) — LingBot-World, the world-model side of LingBot-VA / Ant Group’s open WAM stack on Wan 2.2.
• Cosmos 3: Omnimodal World Models for Physical AI — NVIDIA Cosmos 3, the monolithic MoT joint-prediction WAM the post discusses as Cosmos Policy.
• Robots Need More than VLA and World Models — Bou-Ammar position paper arguing the VLA-vs-WAM debate skips four missing interfaces (autolabelling / retargeting / 3D physics reasoning / video-language reward inference).
• π*0.6: a VLA That Learns From Experience (RECAP) — π*0.6 / RECAP, the modern VLM-backbone VLA Reuss positions on the opposite side of the bet.
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models — Embodied-R1.5, a unified-VLM VLA that argues VLM-side capabilities (pointing/grounding) substitute for action pretraining — counter-evidence to the “VLM-VLA grounding wall” framing.
• Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models — Spirit-v1.5, the data-axis counter-bet inside the VLM-VLA camp.