InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data

TL;DR

InternW0-Δ is a Shanghai AI Lab World Action Model that couples a pretrained Wan2.2-TI2V-5B video expert and an action expert through a directed Mixture-of-Transformers, with a frozen VLM supplying scene-grounded task semantics to the action side. Two training-only mechanisms carry predictive and geometric priors into representations the action expert consumes without future-video sampling at inference: Causal Imprint tokens are supervised against adjacent clean-latent differences on future frames, and a Track4World teacher distills clip-level 4D geometry/motion into the video expert via cached descriptors. Trained on a unified >20K-hour open corpus spanning robot demos, UMI, egocentric human, and Ego2Robot data under a canonical state-action representation, it reports SOTA on LIBERO-Plus (92.8%), RoboTwin 2.0 Clean2Random (71.9%), EBench (66.0), and RoboDojo (23.9%), with 152.8 ms controller-observed round-trip latency on a single RTX 5090. Code, weights, infra, and processed data (where licenses permit) are pledged.

Key claims

A directed Mixture-of-Transformers couples a pretrained video expert and an action expert through masked joint attention, with a frozen VLM providing scene semantics only to the action stream and T5 preserving the video expert’s inherited language pathway [§3.1, §3.2, §3.6].
Causal Imprint tokens aggregate recent/current visual features in the video expert’s self-attention layers and are supervised by (a) adjacent clean-latent differences on future frames and (b) representation alignment to a terminal-future-slice feature; the clean futures never enter the action expert’s forward path [§3.4, §3.7, Fig. 4].
A frozen Track4World teacher precomputes clip-level 4D-aware descriptors (geometry + 2D/3D motion + camera motion + visibility), and 16 learnable queries on top of the 15th VideoDiT block’s clean-condition hidden tokens are trained to reproduce them under an MSE auxiliary loss; the teacher and distillation branch are discarded at inference [§3.5, §3.7].
A canonical state-action representation unifies robot demos, UMI data, egocentric human demos, and Ego2Robot into >20K hours of processed training data under a shared semantic layout (joint positions, absolute EEF pose, gripper/hand configuration), with EEF actions encoded relative to the current EEF pose and other channels absolute [§3.2, Table 1].
A sparse memory of anchor-frame + previous-chunk + current observation feeds the video expert episode-level and recent-execution context without maintaining a dense history [§3.3].
At inference, actions are generated with a single video-expert prefill followed by iterative action-only denoising — no future-video sampling, no distillation-branch invocation [§3.8].
Post-trained on only unperturbed demonstrations, InternW0-Δ reports LIBERO-Plus 92.8% under distribution shift, RoboTwin 2.0 Clean2Random 71.9% and Clean2Clean 90.0%, EBench overall 66.0, and RoboDojo average 23.9% (≈2× the strongest prior WAM) [Abstract, §1].
Deployment runtime hits 152.8 ms average controller-observed round-trip latency on a physical dexterous-hand setup with a single RTX 5090 via context caching + compiled asynchronous action-chunk execution [§1].

Method

InternW0-Δ (part of the InternW series) is built around a directed Mixture-of-Transformers that keeps the video and action streams modality-specific but couples them block-wise through masked joint attention. The video expert inherits Wan2.2-TI2V-5B’s weights and T5 language pathway; the action expert consumes task-conditioned scene semantics from a frozen VLM applied to all current views together with view identities and the instruction. Multi-view observations are composed into an embodiment-dependent canvas, encoded by the frozen Wan VAE, and conditioned on proprioception via per-expert linear projections. A sparse visual memory holds an anchor frame, a frame from before the previous action chunk, and the current observation.

Two auxiliary supervision paths carry future information into training without ever letting the action expert attend to realized future tokens. Causal Imprint tokens live inside the video expert’s self-attention layers at the current-frame spatial resolution; their final-layer representation is supervised by adjacent clean latent differences Δ_i = z_{t+i+1} - z_{t+i} constructed from pre-noise clean future slices, and intermediate layers are aligned with the stop-gradient features of the terminal future slice at the same block depth. A separate 4D-aware distillation path takes projected clean-condition hidden tokens from the 15th VideoDiT block, decodes them through two Transformer layers with 16 learnable queries, mean-pools, and matches a cached clip-level descriptor from a frozen Track4World teacher. The attention mask (Fig. 3) formally guarantees that neither Causal Imprint nor the action tokens have any forward activation path from realized future-frame tokens.

Training optimizes L_FM(video) + L_FM(action) + L_imprint + L_align + L_dist jointly. Inference uses a single video-expert prefill (anchor + recent + current observations) followed by action-only iterative denoising; the Track4World teacher and student branch are not evaluated.

Results

LIBERO-Plus (distribution shift, post-trained on unperturbed demonstrations only): 92.8% — best reported [§1].
RoboTwin 2.0: 90.0% Clean2Clean, 71.9% Clean2Random — both leads [§1].
EBench (mobile bimanual manipulation): 66.0 overall — highest [§1].
RoboDojo (memory + precision + long-horizon): 23.9% average success, ≈2× the strongest prior WAM [§1].
Real-robot deployment on two gripper-based and two dexterous-hand platforms by post-training the same pretrained checkpoint to each embodiment’s control interface [§1].
Average controller-observed round-trip latency: 152.8 ms on a single RTX 5090, reported as a speedup over the standard runtime via caching + asynchronous action-chunk execution [§1].

Why it’s interesting

InternW0-Δ is the sharpest controlled instantiation to date of the “video co-training as supervision, not as inference-time rollout” WAM recipe — same thesis as Fast-WAM: Do World Action Models Need Test-time Future Imagination? Fast-WAM but pushed further: both future supervision (Causal Imprint) and geometric supervision (4D-aware distillation) are explicitly training-only, with mask-level guarantees that no activation path exposes realized future frames to the action expert. This matters because the recurring open-problem on the World Foundation Models page — Sitzmann’s chicken-and-egg of action-conditional WFMs — is sidestepped here by keeping clean futures and 4D teacher descriptors only in the training objective, not in the control loop.

The 20K+ hour unified corpus is the most direct contemporary challenger to Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s million-hour scaling-law claim and to Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1’s 100K-hour UMI scaling at the open-data tier, and the explicit Ego2Robot integration (Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data) means it sits astride the human→robot pipeline rather than treating human video as a separate pretraining substrate. The directed-information-flow argument contrasts cleanly with Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA’s unified causal AR flow-matching and with μ₀: A Scalable 3D Interaction-Trace World Model µ₀’s frozen 3D-trace WM + action-expert factorization — three structurally different answers to the same question of how to put future prediction into service of control.