InternW0: A Foundational Physical World Model for Efficient Real-World Interactions

ListenThe World Model That Doesn't Replan Every Step11:01

Show notes

InternW0 is Shanghai AI Laboratory's first physical world model in the InternW series — a dual-expert mixture-of-transformers that pairs a slow, high-capacity video predictor with a fast, lightweight action controller. The clever bit: instead of regenerating a future plan at every control step, a layerwise K/V editor lets each action chunk build its own observation-conditioned view of a cached video context. We dig into the architecture, the independent-denoising trick that lets action gradients shape the video representation, the 7,200-hour corpus, and what all of this might mean for how we think about video models. **Sources** - InternW0: A Foundational Physical World Model for Efficient Real-World Interactions

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

InternW0 is the first entry in Shanghai AI Laboratory’s InternW physical-world-model series — a mixture-of-transformers architecture that pairs a high-capacity video expert (future visual prediction) with a lightweight action expert (continuous robot control), coupled through an observation-conditioned video-context routing interface. The design rejects the standard WAM contract of regenerating a future plan at every control step: the video expert updates on a slower schedule while the action expert consumes a per-chunk-edited view of a cached video KV, so predictive context remains useful across many fast action updates. Pretrained on ~7,200 hours of heterogeneous robot + egocentric data (incl. the 275-hour EgoLab wet-laboratory corpus), with a contact-aware post-training stage that extends the action interface with force/tactile histories and jointly predicts future interaction wrenches. Evaluated on simulation benchmarks and real scientific workflows (15-stage MOF synthesis, 5-stage dexterous quantitative pipetting).

Key claims

InternW0 instantiates an asymmetric mixture-of-transformers WAM: separate token streams and parameters per layer for the video expert and action expert, with the video expert never consuming robot-action tokens or domain identities, keeping physical prediction broadly shared while control specializes per embodiment [§2.1].
Asynchronous duplex inference separates the two timescales — the video expert refreshes its prediction on a slower schedule, the action expert emits chunks from the latest plan and current observations — so action inference can continue while the next video prediction is computed [§2.1].
The load-bearing primitive coupling the two experts is a layerwise chunk K/V editor: a small set of observation-derived routing queries retrieves task-relevant information from the cached predictive context, routes it back to the original video-token positions, and a zero-initialized residual projection predicts corrections gated by a learned scalar — so the editor begins as identity and progressively learns observation-dependent corrections, without mutating the underlying video context [§2.1, Eqs. 2–4].
The video expert is trained with an independent-denoising condition stream that shares all VideoDiT parameters with the primary video flow-matching stream but uses independently sampled noise/flow-time, is kept clean with probability π₀, and whose K/V is consumed by the editor; critically, these K/V features are not detached, so gradients from the action flow-matching objective flow back into the shared video expert and shape the predictive representation by action usefulness, not visual reconstruction alone [§2.1].
Cross-embodiment training uses learned soft prompts prepended to every action chunk, with domains defined by data provenance + robot morphology + subset identity (25 domains, 24 statistics groups in the current corpus), plus per-dimension validity masks that zero unavailable inputs and exclude them from the loss — so joint training does not require identical control or sensing semantics across robots [§2.1].
Action-free egocentric videos supervise only the video expert, while action-labeled robot trajectories supervise both — the pretraining objective is a mixed flow-matching loss L_pretrain = λ_v L_video + λ_a 1[action supervision] L_action over samples drawn from the mixed distribution, with per-example / per-chunk flow times from the shifted continuous schedule and first-frame-causal video attention anchoring one clean reference frame [§2.1, Eq. 6].
Contact-aware post-training extends the action-chunk prediction target from a_t to [a_t; f_t; τ_t] (action + 6-D interaction wrench + tactile) — only pre-chunk contact history conditions, future contact channels are noised and denoised jointly with action channels, action channels decode for execution and contact channels represent predicted feedback [§2.1, Eq. 7].
Pretraining corpus is 7,233.5 hours / 811,969 episodes across 25 domains: InternData-A1 (3,494 h sim), AgibotWorld (2,620 h real incl. tactile), RoboCOIN (438 h, 13 real subsets), Galaxea R1 Lite (315 h), EgoLab (275 h human egocentric), MolmoAct YAM bimanual (70 h), RoboDojo (21 h sim); domain sampling proportional to √(effective anchors) to prevent any single corpus from dominating [§3, Table 1].
EgoLab is a new 275-hour wet-laboratory bimanual egocentric corpus (3,192 episodes, 50 fps DJI Action @ 2.5K, head-strapped), with 3D camera + bimanual hand motion reconstructed at native 50 fps; released as a laboratory-operation-prior substrate for WAM pretraining [§3].
Training stack separates data plane (shuffle metadata descriptors not payloads, materialize on demand, sample-identity-deterministic choices for exact replay, node-shared video cache), execution plane (Ray-managed co-location of data materialization with GPU training), and optimization plane (per-joint-layer FSDP2 with hierarchical replicate-and-shard mesh, BF16 compute + FP32 masters + FP32 gradient reduction, per-layer compile + selective activation checkpointing, transactional distributed checkpointing for mid-epoch resumption) [§2.2].
Target downstream deployment is specifically scientific workflows: a 15-stage metal–organic framework synthesis and a 5-stage contact- and force-aware dexterous quantitative pipetting pipeline inside InkStone, where Intern-S2-Preview provides scientific reasoning + tool use and InternW0 is the physical-execution layer [§1].

Method

InternW0’s backbone is a mixture-of-transformers where each joint layer contains two modality-specific blocks — a video expert and an action expert — with separate parameters and token streams. Video frames are encoded by a frozen Wan VAE for the predictive stream and by a frozen DINOv3 for current chunk-level observations; language instructions are precomputed text embeddings; proprioceptive states provide chunk-specific conditioning for the action expert. Both experts use continuous flow matching. The video stream uses a first-frame-causal attention mask (one anchor frame remains clean and is excluded from the video loss, future frames attend bidirectionally); flow times are sampled per video example and independently per action chunk from a shifted continuous schedule.

The structurally novel piece is the chunk K/V editor between the experts. The paper’s framing is that regenerating a video plan at every control step is too expensive, but executing against an unchanged plan risks conditioning the policy on an increasingly stale world view. The editor resolves this by letting the action expert consume a per-chunk edited view of the cached video context: at each layer, observation-derived routing queries attend to the valid video context K/V to retrieve task-relevant information, that retrieved information is routed back to the original video-token positions, and a zero-initialized residual projection — gated by a learned scalar σ(g) — produces corrections to the context tokens. The underlying cached video context is never mutated; each action chunk constructs its own view (Eqs. 2–4). Because the gate starts at zero, the editor begins as identity and learns observation-dependent corrections progressively.

The paper trains this with an independent-denoising second video stream — same VideoDiT weights, independently sampled noise realization and flow time, first frame kept clean, kept fully clean with probability π₀. This stream’s K/V is what feeds the editor. Crucially, these condition K/V features are not detached from the video network, so the action flow-matching loss propagates gradients through the editor into the shared video expert — the video representation learns to be useful for action generation, not just for reconstruction. At inference the ground-truth condition stream is replaced by generated video context; its K/V is cached and exposed through the same interface.

Cross-embodiment pretraining uses learned soft prompts prepended to every action chunk (domains = data source × morphology × subset — 25 total this release, not 1-to-1 with robot morphology), per-domain input/output projections, and per-dimension validity masks that zero unavailable channels and exclude them from the loss. Action-free egocentric videos contribute video supervision without action labels; the combined objective is a mixture of video and action flow-matching losses weighted by per-example indicators.

Contact-aware post-training augments the action prediction target with 6-D interaction wrenches (3 forces + 3 moments) and tactile representations. Only pre-chunk contact history is used as conditioning; future wrenches and tactile channels are noised and denoised jointly with action channels. Action channels decode for execution; contact channels serve as predicted feedback that supports contact-aware action selection without requiring force/tactile labels during the heterogeneous pretraining phase.

The data pipeline cleans trajectories for defects that corrupt supervision — static trajectories, head/waist motion intervals, brief gripper pulses, missing/out-of-range values — on each dataset’s native timeline, handled via validity masks or window invalidation to retain usable data while excluding unreliable supervision. Domain mixing is proportional to √(effective anchors) rather than raw hours. Training infra separates data/execution/optimization planes; the execution plane runs on Ray and the optimization plane layers composable FSDP2 (per joint layer sharded independently, narrow per-domain interfaces replicated) with BF16 compute + FP32 masters + FP32 gradient reduction and per-layer compilation under selective activation checkpointing.

Results

The abstract and §3 report the training-corpus composition (7,233.5 h / 811,969 episodes across 25 domains, Table 1) and name two downstream evaluation targets — simulation benchmarks and real-world scientific workflows (15-stage MOF synthesis, 5-stage dexterous quantitative pipetting). Quantitative benchmark numbers against RoboTwin / LIBERO / RoboCasa and head-to-heads vs filed WAM/VLA baselines (π0.5, Fast-WAM, LingBot-VA, µ₀, Flex-π, Long-WAM) are not retrievable in the first 30KB of the arxiv body and need the full PDF to extract. EgoLab is a 275-hour wet-laboratory corpus released as part of this work.

Why it’s interesting

InternW0 occupies a distinct cell on World Foundation Models’ recipe-lever board: it is the first filed WAM that combines (a) MoT dual-expert architecture (sibling to Long-WAM: Scaling the Context of World-Action Models, Fast-WAM: Do World Action Models Need Test-time Future Imagination?, Flex-π: A Multi-Stream World-Action Model with Compute Flexibility) with (b) a layerwise chunk K/V editor that mutates per-chunk views of a cached video context rather than refreshing the context itself — a finer parallelism axis than RealtimeWAM: One-Step Asynchronous World Action Models RealtimeWAM’s Cross-Expert Wavefront Pipelining, which overlaps the two experts via block-wise sharing of the same cache; InternW0 instead edits the cache per action chunk. The zero-initialized residual + non-detached condition-stream K/V gives the shared video expert a direct learning signal from the action loss without a separate reconstruction objective. The scientific-workflow framing (InkStone + Intern-S2-Preview + MOF synthesis + quantitative pipetting) also stakes a position adjacent to the Tactile sensing for manipulation contact-force-integration line (FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT, ReForce — force-aware retargeting via closed-loop residual force tracker ReForce, Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction T-Rex) by making wrench prediction and tactile conditioning a first-class post-training stage, with wrenches entering as a joint denoising target alongside actions.