DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

TL;DR

DreamX-Phi 1.0 is a bimanual-manipulation video world model from Alibaba’s DreamX Team: given an observed frame, a language instruction, and a prescribed end-effector-pose + gripper action sequence, it predicts the resulting future observation video. Its central contribution is a stack of faithfulness constraints on top of a standard video diffusion prior — per-arm SE(3) transformations injected via PRoPE-style geometric attention, a depth branch for scene geometry, and SAM3 object masks distilled from a frozen V-JEPA teacher to preserve manipulated-object identity through grasping. A distribution-matching distillation step compresses the multi-step teacher into a few-step student. The model placed first on Track 1 and second on Track 2 of the WorldArena 2.0 IROS Challenge.

Key claims

Realism does not imply faithfulness: a video model can hallucinate the wrong arm moving or lose the manipulated object mid-grasp even when frames look convincing, motivating explicit action-, geometry-, and object-consistency constraints [Abstract].
Per-arm SE(3) transformations are injected into attention via PRoPE-style geometric encoding, preserving arm identity and rigid-motion structure of the commanded path [Abstract].
A lightweight depth branch is added for scene-level geometry, and SAM3 masks combined with a frozen V-JEPA teacher are used to maintain object consistency through grasping [Abstract].
The multi-step generator is distilled into a few-step student via distribution-matching distillation for efficient deployment [Abstract].
DreamX-Phi 1.0 achieves first place on Track 1 and second place on Track 2 of the WorldArena 2.0 Challenge [Abstract].

Method

The paper frames the model as a conditional video generator taking a start frame, a language instruction, and an action sequence of end-effector poses + gripper states, and producing a future observation video. Standard action-conditioned video WMs treat actions as a scalar or embedded feature; DreamX-Phi routes them geometrically instead. Per-arm SE(3) transformations at each timestep are converted into a PRoPE-style relative encoding inside attention — the same recipe Cameras as Relative Positional Encoding uses for cameras — so the network sees each commanded rigid motion as structured geometry rather than a generic conditioning vector. This is the mechanism intended to prevent the “wrong arm moves” failure. Two auxiliary supervisions sit on top: a lightweight depth branch grounds scene-level geometry, and SAM3-derived object masks are distilled from a frozen V-JEPA teacher to keep the manipulated object’s identity coherent through the grasp (the “object disappears” failure). Deployment efficiency comes from distribution-matching distillation of the multi-step generator into a few-step student.

Results

WorldArena 2.0 Challenge: 1st place on Track 1 and 2nd place on Track 2 [Abstract].

The abstract does not report per-component ablations or intrinsic video-quality metrics (FVD, PSNR, action-following scores) at filing time; those, along with weights and inference code, are gated on the IROS Challenge conclusion per the release note.

Why it’s interesting

DreamX-Phi is a concrete instance of the “add per-arm SE(3) geometric structure + auxiliary geometry/object heads to a video WM” recipe that has been circling this cluster from several directions. The PRoPE encoding is a direct crossover: Cameras as Relative Positional Encoding introduced it for multi-view camera frusta, and DreamX-Phi ports the same relative-transform-into-attention machinery from the observer side to the actor side (arms), a nontrivial transfer of the same primitive. Contrast against RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation, which also predicts synchronized RGB + depth for embodied WMs but uses a tri-branch DiT with 3D RoPE rather than per-arm SE(3) attention; and against μ₀: A Scalable 3D Interaction-Trace World Model (µ₀), which pushes 3D structure into the interface of the world model (contact-point trajectories) rather than into attention geometry. The Xiaomi-Robotics team’s WorldArena leader on file — Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model — took the “one 38B AR omnimodel” route to the same benchmark; DreamX-Phi’s placement suggests a smaller specialized diffusion-WM with structured geometric attention is competitive with monolithic AR omnimodels at this specific challenge.