Agentic World Action Models

TL;DR

AW-0 is a World Action Model that denoises code, future video, and robot actions jointly as three coupled streams inside a Mixture-of-Transformers backbone (code expert = LLaDA diffusion LM + frozen Qwen3-VL ViT, video expert = Cosmos-Predict2.5 DiT, action expert = separate DiT), with blockwise causal cross-expert attention in the order code → video → action. An automated annotation pipeline (Gemini 3.1 Pro + Claude Code / Opus 4.8) labels 2,000 h of YAM bimanual teleop + 1,000 h of egocentric human video with programs over a 50-skill library distilled from the data itself. At deployment the generated program conditions the action expert by default; a hybrid mode optionally executes the code against perception/control primitives Code-as-Policies-style, falling back to the action chunk when the slice fails validity checks. Reports 50.7% average on RoboTwin 2.0 (80.8% clean, 29.2% random in hybrid mode) and 71.1% average on eight real-world bimanual tasks on YAM parallel grippers and Tianji-Sharpa 22-DoF dexterous hands, vs 43.6% for π0.5 and 46.0% for DreamZero on the same tasks.

Key claims

Joint denoising of code + video + action in one MoT with blockwise causal cross-expert attention (code → video → action) beats autoregressive code-then-video-then-action on 7 of 8 real-world tasks (71.1 vs 54.5 avg) and is also faster [Ablations table].
Placing the code stream first in the attention chain (C → V → A) outperforms V → C → A on 7 of 8 tasks because semantic task structure shapes future prediction throughout denoising rather than only after the video is committed [Ablations §How should the code expert be integrated].
A dedicated ViT (Qwen3-VL) as the code expert’s visual input beats reusing the video expert’s Wan2.2 VAE features on all 8 real-world tasks, with the largest gains on fine-grained reasoning tasks: Pack Backpack 64.6 vs 39.6 and Solve Cube 72.1 vs 53.9 [Ablations table].
Full annotated programs with explanatory comments beat comment-free code on all 8 tasks, and full programs beat natural-language subgoals on 7 of 8 — program structure plus comments carries intent that free text alone does not [Ablations §Code as planning representation].
Removing skill calls from the program vocabulary hurts on all 8 tasks, with the largest drop on skill-heavy tasks (Deal Poker 58.3 vs 42.0) — the distilled 50-skill library is load-bearing, not a readability affordance [Ablations §Code as planning representation].
Headline real-world result: 71.1% average across 8 bimanual tasks (3×16 rollouts each, additive rubric) vs 46.0% DreamZero, 43.6% π0.5, 39.9% WLA, 35.9% EgoScale on the same hardware and tasks [Real-world experiments table].
On RoboTwin 2.0 with 50 clean demonstrations per task (2,500 episodes total), reaches 80.8% on the clean setting (+16.0 pp vs WLA at 64.8%) and 29.2% on the visual-random setting in hybrid mode (above every baseline including π0.5 at 27.7%) [§Simulation].
Hybrid execution preserves 46% of clean-setting performance under visual randomization vs 25% for default action-chunk execution, trading 17.7 pp on clean scenes for +8.6 pp on random scenes [§Simulation].
On the real-world Fold and Stack Towels task, hybrid execution reaches 90.6% vs 84.4% for pure action-chunk execution from the same post-trained checkpoint — dexterous folding stays on the action expert while structured approach + stacking motions are driven by executed code [§Hybrid execution in the real world].
The annotation pipeline freezes a 50-skill library (23 for YAM data + 27 for egocentric human data) distilled by Claude Code acting as coordinator over Gemini-authored per-episode programs, then re-annotates the corpus against the frozen vocabulary with a cache that only invokes the coding agent on contact/motion-pattern cache misses [§Agentic code annotation].

Method

AW-0 models three generative streams in parallel: a discrete code-token sequence C₀ (program for the current subtask), continuous video latents Z₀ from a pretrained VAE (future observations), and a continuous action chunk A₀ over a horizon of H steps. Conditioning is current multi-view observations + language instruction + robot state + a compact memory cache of K learnable slots holding compressed program slices from earlier in the episode. Each expert owns its own attention projections, LayerNorms, and FFNs; they communicate at every layer through masked joint attention, with bidirectional attention inside each expert and blockwise causal attention across experts in the order code → video → action.

Training targets each stream with its native objective: video and action use flow matching with a shifted noise schedule; code uses absorbing-mask corruption with a random mask ratio and cross-entropy on masked positions. During training only the active slice (one subtask’s primitive/skill calls) is masked and supervised, with the slices generated earlier in the episode compressed into the memory cache as conditioning. To ensure the action expert reads informative video features, the video noise level is resampled from a mixture of the pretrained distribution and a uniform one, with importance weights balancing video learning against predictive guidance.

At inference the policy appends a canvas of G mask tokens for the code expert and decodes them jointly with the video and action states over T denoising steps, committing the G/T most confident positions per step. The code stream supplies planning context to both continuous experts throughout generation; the action expert additionally attends to the video representations. In default mode the decoded code is not executed — it is only a plan that conditions the action expert, and the predicted action chunk drives the robot. In hybrid mode the decoded slice is executed Code-as-Policies-style against a library of perception + control primitives whenever it passes stage, dependency, syntax, whitelist, and vacuity checks; otherwise the action chunk from the same forward pass drives the robot until the next replan.

The code expert is initialized from LLaDA (a pretrained diffusion LLM) augmented with a frozen Qwen3-VL ViT so the current views are encoded into visual tokens that ground skill planning in the scene. The video expert is a pretrained Cosmos-Predict2.5 video DiT over VAE-encoded future frames. The action DiT generates the continuous action chunk with additional robot-state conditioning.

Programs for training come from an automated pipeline: Gemini 3.1 Pro writes subtask-level programs against a predefined, declaration-only primitive API on one episode per task; Claude Code (Claude Opus 4.8) acts as coordinator and distills recurring task-general routines into skills. The resulting 50-skill library is frozen (23 for YAM robot data, 27 for egocentric human data) and the sampled episodes are re-annotated against the final vocabulary, forming a cache of reusable subtask→program mappings. The remaining corpus reuses a cached program whenever a subtask’s contact and motion pattern matches an existing entry and invokes the coding agent only on cache misses. Pretraining uses ~2,000 h of YAM bimanual teleoperation + ~1,000 h of egocentric human manipulation, with every training window paired with its instruction and the aligned program slice. Downstream fine-tuning labels each target task’s demonstrations under the frozen skill library and jointly fine-tunes on code + video + action.

Results

RoboTwin 2.0 (50 demos/task, 2,500 episodes). Clean: 80.8% (default), 63.1% (hybrid), vs WLA 64.8%, DreamZero 51.9%, π0.5 48.4%, EgoScale 36.5%. Random: 20.6% (default), 29.2% (hybrid), vs DreamZero 17.6%, π0.5 27.7%, EgoScale 16.4%, WLA 15.5%. Average across both: AW-0 50.7% [§Simulation].
Real-world 8 tasks (YAM parallel grippers + Tianji-Sharpa 22-DoF hands, 3×16 rollouts each, additive rubric). AW-0 71.1% avg vs DreamZero 46.0%, π0.5 43.6%, WLA 39.9%, EgoScale 35.9%. Largest absolute gains: Arrange Fruits 91.7 vs best baseline 70.8 (+21), Pack Backpack 64.6 vs 49.7 (+15), Flip Scoreboard 68.8 vs 40.8 (+28), Solve Cube 72.1 vs 52.3 (+20) [Real-world experiments table].
Hybrid-mode on Fold and Stack Towels (YAM): 90.6% vs 84.4% for action-chunk execution with the same post-trained checkpoint [§Hybrid execution in the real world].
ViT vs VAE for the code expert’s visual input: ViT wins on all 8 real-world tasks; aggregated 71.1 vs 55.6 avg. Fine-grained-reasoning tasks show the widest gaps: Pack Backpack 64.6 vs 39.6, Solve Cube 72.1 vs 53.9, Fold Towels 84.4 vs 61.3 [Ablations table].
Joint denoise vs autoregressive teacher-forced code → video → action: joint denoise wins on 7 of 8 tasks (71.1 vs 54.5 avg) and is faster; AR loses most on contact-rich dexterous tasks (Fruits w/ Tong 54.2 vs 20.8, Flip Scoreboard 68.8 vs 39.3, Solve Cube 72.1 vs 55.8) [Ablations table].
Qualitative code-conditioning effect on predicted video. On the instruction “Arrange the fruits so that finally there is only mango in the left bowl”, AW-0 predicts the right wrist camera interacting with the mango; the natural-language-subgoal variant hesitates between apple and mango; the no-code variant goes for the apple. On Deal Poker card-slot insertion, AW-0 predicts the card aligning and entering; the two ablations produce blurred/distorted hand-card predictions [§What code changes in the predicted future].
Named failure modes. Six representative failure videos include contact-establishment misses on dexterous tasks and skill-grounding errors (right skill, wrong arguments) — syntactic validity of the generated program does not ensure correct execution [§Failure cases, §Limitations].

Why it’s interesting

AW-0 directly bridges two threads that the wiki has been tracking as structurally distinct cells on the VLA Models recipe-lever board. The World Foundation Models / WAM lineage (World Action Models are Zero-shot Policies (DreamZero) DreamZero, Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA, InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data InternW0-Δ, EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data EgoWAM) jointly denoises video and action for visual foresight but has no mechanism for compositional skill reuse. The coding-harness over frozen policy lineage (Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents Harness VLA, Playful Agentic Robot Learning RATs, RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI, RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks RoboClaw, OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA) composes reusable programs but has no predictive grounding in physical dynamics. AW-0 puts both inside one MoT backbone where the generated program is simultaneously (a) training-time conditioning that shapes the predicted future and the action chunk and (b) deployment-time executable code with action-chunk fallback — directly answering RATs’ open question of whether its play-learned skill library would compose with a video-prediction prior rather than sitting on top of a frozen VLM. The 23+27 skill library pattern inherits RATs’ play-stage frozen-library discipline but freezes against demonstration data rather than self-directed play rollouts.

The architecture is a sibling to Rho-1: Collapsing the Multimodal Stack Rho-1 on the “collapse the multimodal stack into one MoT” axis but keeps three experts (code is a first-class modality) rather than two, and the C → V → A attention ordering ablation gives the sharpest filed empirical answer so far to the “where should structured planning sit relative to future prediction and action generation” question that EgoWAM / UniWAM / ModAR had been circling without controlled multi-stream ablations. The ViT-vs-VAE finding for the code expert’s visual input quantifies the cost of the “one encoder for everything” factorization at the specific interface where language-mediated semantic selection has to happen — same shape of result as Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models LIT’s latent-interface bottleneck but on the planning side rather than the action side.