EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data

TL;DR

EgoWAM (Georgia Tech RL2, Danfei Xu’s group) is a controlled study of what world-model target makes human→robot co-training actually transfer under the World Action Model (WAM) recipe. The authors fix the backbone (HPT), action head (flow-matching), data mixture, and horizons, and swap only the world-prediction target across three choices: pixel VAE latents, DINO features, and camera-stabilized 3D motion flow. Across three real bimanual tasks (cup-on-saucer, fold-clothes, bag-grocery) and 1,800 real-world rollouts, WAM co-training scales with in-the-wild egocentric human data (the EgoVerse-A flagship split) where behavior-cloning co-training saturates or degrades; the pixel target transfers weakly, DINO delivers up to 4× better OOD object/scene generalization, and 3D flow delivers 20–30% in-domain gains. The paper argues an effective world representation must abstract appearance, stay consistent across embodiments, and factor out ego-motion — pixel VAE fails all three, DINO satisfies two, 3D flow satisfies all three by construction.

Key claims

BC co-training is the wrong baseline when human and robot action patterns are misaligned: the shared action decoder is the sole channel through which human data reaches the policy, forcing it to entangle transferable content (objects, scenes, semantics) with non-transferable execution (morphology, style), and the embodiment gap in the latter blocks transfer of the former [§1].
WAMs open a second supervision channel — an auxiliary world-model head predicts future states from a shared backbone, and because that channel operates on observations rather than actions it is largely indifferent to morphology and behavioral style, letting human data shape the shared backbone even when its action labels cannot [§1, §3.2].
A world-model target should satisfy three desiderata: (D1) appearance abstraction, (D2) cross-embodiment consistency, (D3) ego-motion factoring; pixel VAE targets violate all three, DINO features satisfy D1/D2 but not D3, and camera-stabilized 3D motion flow satisfies all three by construction [§4].
Under the same HPT backbone, action head, horizons, and data mixture, pixel-based prediction transfers weakly from in-the-wild human data because reconstructing raw appearance entangles embodiment- and scene-specific detail that does not transfer [§5.2, Fig. 3, Fig. 5].
The DINO-based WAM yields the strongest OOD object and scene generalization — up to 4× improvement on novel objects and novel scenes with varied backgrounds/table heights — while the 3D-flow WAM yields the largest in-domain spatial gains (20–30%) [§5.2, Fig. 3, Fig. 6].
BC co-training on unaligned human data (e.g. humans grasping cups sideways vs the robot gripper’s front approach) collapses below the robot-only baseline; the 3D-flow WAM stays robust and still surpasses robot-only, evidence that the world-prediction channel absorbs action mismatch that BC amplifies [§5.2 Q3, Fig. 8].
Under BC co-training, human/robot (and even human/human) embeddings sit in isolated clusters when actions do not align; WAM co-training pulls them into a shared space via the additional dynamics supervision [Fig. 4].
WAM benefits come from training-time representation shaping rather than test-time imagination — the world-model head is dropped at inference and the action head runs at 30 Hz matching BC latency, with the cross-embodiment representation retained in the trunk [§4.3].
3D-flow supervision requires camera-frame stabilization (Aria VIO poses fold into the 3D point tracker so returned point positions share a consistent world frame, then mapped back to the camera frame at time t) — without this, ego-motion dominates the flow signal on egocentric video and static background produces large apparent flow whenever the wearer turns their head [§4.2].
Aligned baseline construction matters: the paper unifies actions into a 14-D end-effector space, re-expresses human hand poses in the instantaneous device frame at time t to factor out head ego-motion, uses embodiment-specific windows (0.65 s human, 1 s robot) both discretized into 20 steps to absorb speed mismatch, and quantile-normalizes workspace range — so BC is the strongest action-aligned baseline the framework supports, not a strawman [§3.1].

Method

EgoWAM builds on the Heterogeneous Pretrained Transformer (HPT) backbone: embodiment-specific stems tokenize each input (ego-RGB via ResNet-18 or DINO, wrist-RGB on robot batches, end-effector proprioception via per-embodiment MLP) into a shared latent stream fed to a single transformer trunk. Two banks of learnable query tokens read out of the trunk: shared action tokens feed a flow-matching action head (alternating self/cross-attention, denoising a chunk A_t:t+H from Gaussian noise conditioned on time τ), and shared future tokens feed a swappable world-model head. Training is joint: L = L_BC + λ · L_WM with λ = 0.05, drawing minibatches from both human D_H and robot D_R corpora. At inference the WM head is dropped and only the action head is unrolled.

The three world-model heads instantiate the desideratum axis:

Pixel VAE. Target is the future ego frame encoded by a pretrained video VAE optimized for photometric reconstruction. Head is a VACE-1.3B DiT (no text conditioning), trained as noise prediction. Two variants: pretrained-init (Pixel-PT) and from-scratch (Pixel). Violates all three desiderata.
DINO features. Target is DINOv2 patch features at the future frame. Head follows the wide-DDT design of RAE (widened denoiser matching the high channel count of semantic latents, decoupling noise-prediction from feature-conditioning blocks), trained as noise prediction. Satisfies D1 and D2; spatially image-indexed so D3 only partially mitigated.
3D flow. Target is a dense 3D motion field over query pixels in the camera-stabilized frame at time t. Aria VIO camera poses are fed into a pretrained 3D point tracker so returned points sit in a consistent world frame; the future 3D position is then mapped back into the camera frame at t, so static background yields near-zero flow while manipulated objects retain motion proportional to physical displacement. Head is an alternating self/cross-attention decoder taking query points as additional conditioning, regressing velocity via smooth-L1. Satisfies all three desiderata.

Evaluation uses three EgoVerse flagship bimanual tasks (cup-on-saucer, fold-clothes, bag-grocery) on a two-arm ARX5 platform with Aria egocentric RGB + two wrist RealSense D405 cameras. Human data spans two regimes: In-Domain (1:1 with robot data, same scenes/objects, unmatched viewpoint/behavior) and EgoVerse (10:1 human:robot, full EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World flagship split with diverse scenes, objects, demonstrators, no deliberate alignment). Each method is scored on 20 in-distribution + 20 OOD rollouts per task (10 unseen objects in training scene + 10 seen objects in novel scenes).

Results

Q1 — WAM vs BC under natural human data. Across in-domain and OOD evaluations, WAM co-training scales and transfers more consistently from large-scale natural egocentric human data than BC co-training [§5.2 Q1, Fig. 3, Fig. 5]. On fold-clothes with EgoVerse data, BC still overfits to robot data and fails to adapt to novel scenes with lower table heights; the DINO-WAM leverages in-the-wild human data to generalize to new objects and scenes.
Q2 — world representation choice. Pixel-based WAMs hallucinate (free-space grasps on bag-grocery, incomplete final fold on fold-clothes). DINO-WAM yields the strongest OOD generalization to unseen objects/scenes (up to 4× improvement on novel-object / novel-scene axes). 3D-flow WAM yields the largest in-domain spatial gains (20–30%), succeeding at precise cup placement across the workspace [§5.2 Q2, Fig. 3, Fig. 6].
Q3 — unaligned human data ablation. Only bag-grocery naturally benefits from BC action-level co-training because pick-and-place motions are aligned between human and robot. Under a deliberately-unaligned human-demonstration set (humans picking objects in an unusual manner that maps to inexecutable robot actions), BC collapses below the robot-only baseline while the 3D-flow WAM stays robust and still surpasses robot-only [§5.2 Q3, Fig. 8].
Representation prediction quality (Fig. 7). Pixel prediction barely changes across time; DINO refines object shape from human data but only moderately; 3D flow benefits most from human data, recovering object motion that robot-only prediction leaves static — a mechanistic explanation for the 3D-flow in-domain wins.
Inference cost. WAMs are trained joint but the world-model head is dropped at inference; the deployed policy runs at 30 Hz, matching a same-size BC policy [§4.3].
Per-task quantitative tables (normalized sub-task score and success rate over 1,800 rollouts) reported in the paper; specific per-task numbers were not extracted into this filing.

Why it’s interesting

EgoWAM is the sharpest controlled study filed so far on which world-model target transfers under the WAM co-training recipe — a direct follow-up to Fast-WAM: Do World Action Models Need Test-time Future Imagination? (which established that WAM benefits come from training-time representation shaping, not test-time imagination) and complementary to HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining (the matched-volume head-to-head showing egocentric human video can outperform teleop as pretraining substrate at equal data volume). Together the three stake out a coherent position: the WAM channel is where human data enters, and the choice of world-target is the next critical design axis after “how much human data to add.” The Georgia Tech RL2 group producing EgoWAM is the same lineage that shipped EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World (the consortium egocentric corpus EgoWAM trains on) and Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (the reduced-DoF bridging-action recipe on the human-to-robot-retargeting axis) — so EgoWAM lands as an intra-lab argument that the representation channel matters at least as much as the action channel they previously reduced.

The DINO-vs-3D-flow contrast also reads directly against two adjacent recipes: Flow Matching in Feature Space for Stochastic World Modeling (FAIR FlowWM predicting in DINOv3 feature space, sibling in the DINO-target camp) and Back to the Features: DINO as a Foundation for Video World Models (DINO-world video WFM in frozen DINOv2 space) — EgoWAM adds the comparative datapoint those two lack (DINO beats pixel VAE by design, but 3D flow beats DINO on in-domain spatial precision when your task has a clean camera-stabilization signal available). For Luma’s world-model track this is the first filing to explicitly diagnose “predict pixels” as a bad co-training target and to offer two concrete alternatives with predictable trade-offs (semantic prior for OOD generalization vs geometric grounding for in-domain precision).