HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos

TL;DR

HumanEgo (UMD; Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, Yiannis Aloimonos) is a robot-data-free, hardware-agnostic recipe that trains a flow-matching manipulation policy from minutes of Project-Aria egocentric human video. Each demonstration is lifted to Interaction-Centric Tokens (ICTs) — 6-DoF poses of hands and objects expressed relative to other task entities rather than to camera or world frame, giving an embodiment-, viewpoint-, and environment-invariant scene representation. The policy is trained with three free auxiliary objectives (3D Object Motion forecast, 2D image-plane trace forecast, and latent-consistency forecast) that share the encoder with the flow-matching action head, turning the encoder into an in-policy world model of hand–object interaction. With 30 min of human video per task, HumanEgo reports 92.5% average real-robot success across four tasks (75% with only 15 min), outperforms matched-time robot teleoperation by 41%, and transfers zero-shot to novel robots, cameras, and environments.

Key claims

ICTs — entity-relative 6-DoF poses of hand and object expressed against other task entities (not the camera or a fixed world frame) — make the representation invariant to embodiment, viewpoint, and environment so the same tokens describe the same skill across human/robot, camera, and scene changes [Project page §“ICTs encode each entity by its 6-DoF pose relative to other task entities”].
Adding ICTs to raw human RGB lifts the Water Flowers real-robot success from 7.5% to 85% — a +77.5 pp gap that no amount of visual preprocessing closed in their ablations [Project page §“adding ICTs to raw human RGB jumps Water Flowers success from 7.5% to 85%”].
Three auxiliary forecasting objectives sharing the encoder with the flow-matching action head — 3D object-motion forecast, 2D image-plane trace forecast, and latent-consistency forecast K steps ahead — together act as a lightweight world model of hand–object interaction with no extra annotation cost beyond the existing perception pipeline [Project page §“three auxiliary objectives”].
With 30 minutes of human video per task, HumanEgo achieves 92.5% average success across four real-world tasks, and 75% with only 15 minutes [Project page §“30 minutes / 92.5%”].
HumanEgo outperforms matched-time robot teleoperation by 41% absolute success [Project page §“outperforms matched-time robot teleoperation by 41%”].
The trained policy transfers zero-shot across novel robots, cameras, and environments without any robot data in training [Project page §“zero-shot across novel robots, cameras, and environments”].
Aria Gen-1’s calibrated SLAM + MPS hand pose are necessary: precision of upstream SLAM and hand-tracking directly bounds downstream policy performance in the authors’ experiments [Project page §“the accuracy of the upstream SLAM and hand-tracking signal directly bounds downstream policy performance”].
The same pipeline works on third-person exocentric video from a fixed ZED with no SLAM step at all, since the head-camera-motion problem disappears — and the authors report policies “just as strong” as the egocentric variant [Project page §“third-person fixed ZED camera”].
Stated scope limit: ~1 cm precision floor inherited from off-the-shelf object pose estimators, no support for deformable objects, fine sub-cm manipulation, highly dynamic motions, or in-hand reorientation — the regime is coarse “eyes-closed-after-a-glance” pick-and-place [Project page §“Limitations”].
Robot embodiment is currently a parallel-jaw gripper; dexterous-hand support is a natural retargeting extension since the pipeline already tracks 21-keypoint hand pose per demonstration, but is untested [Project page §“Not yet on a real dexterous hand”].

Method

HumanEgo’s pipeline has three stages. (1) Capture and perception. Demonstrations are collected with Aria Gen-1 glasses. Meta’s MPS pipeline supplies calibrated 6-DoF SLAM (head/camera trajectory), 21-keypoint hand pose, and synchronized egocentric RGB out of the box — no per-session calibration. An off-the-shelf object-pose estimator tracks task-relevant objects in 6 DoF. (2) Interaction-Centric Token construction. Each task entity (each hand, each object) at each timestep is encoded as its 6-DoF pose expressed relative to other task entities — i.e. hand-relative-to-object, object-relative-to-target — yielding a unified, variable-length, embodiment-invariant token set per frame. ICT construction “adds no new inputs” beyond what the existing pose-estimation stack already produces. (3) Policy with auxiliary objectives. A flow-matching policy is trained to regress an action chunk from (image, ICTs). A shared encoder also feeds three forecasting heads: 3D Object Motion (future 6-DoF trajectory of each manipulated object), 2D Trace (image-plane projection of hand and object — “the path your eyes would follow watching the video”), and Latent Consistency (the encoder’s own representation K steps ahead). All three targets are computed for free from the same perception pipeline, so the supervision density per demonstration grows without new annotation.

Two design positions are explicit in the body. First, visual fidelity is not the bottleneck — the authors tried arm inpainting (masking the demonstrator’s arm) and keypoint rendering (drawing tracked keypoints back into the frame) and “none of it moved the needle”; the +77.5 pp Water Flowers gain came from adding ICTs rather than improving pixels. Second, the recipe is intentionally minimal: ICTs are not a hand-engineered task graph (no symbolic predicates, no operators, no goal predicates), they are just relative 6-DoF poses, so the structure is bought “essentially for free” relative to a TAMP pipeline.

Results

92.5% average real-robot success across four tasks with 30 min of human video per task; 75% with 15 min [Project page §“30 minutes / 92.5%”].
+41% absolute success vs matched-time robot teleoperation baseline [Project page §“outperforms matched-time robot teleoperation by 41%”].
ICT ablation on Water Flowers: 7.5% → 85% success (+77.5 pp) when adding ICTs to raw human RGB [Project page §“Water Flowers 7.5% to 85%”].
Zero-shot transfer reported across novel robots (i.e. policy trained on human data evaluated on a robot it never saw), novel cameras (RealSense vs ZED), and novel environments (table heights, layouts) [Project page §“transfers zero-shot across novel robots, cameras, and environments”].
Exocentric (fixed ZED) variant reported “just as strong” as the egocentric (Aria) variant on the same task suite; per-task deltas were not enumerated on the project page at filing [Project page §“third-person ZED camera”].
Released asset: HumanEgo dataset on Hugging Face (Leo-TX/HumanEgo) — 122 recordings across serve_bread and water_flowers tasks, with raw Aria recordings, MPS annotations, and full preprocessing output [Project page §“We publicly release the HumanEgo dataset”].
Quantitative per-task tables, baseline identities (matched-time teleop baseline architecture), encoder/head sizes, and auxiliary-loss ablation breakdowns were not surfaced on the project landing page at filing — likely in the PDF (arXiv 2605.24934) which was not retrieved at filing time.

Why it’s interesting

HumanEgo lands directly in the contested center of the VLA Models cluster’s biggest open question: is egocentric human video a credible alternative to teleop data, and if so what is the lever? Three concurrent positions are on the wiki. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining argued the lever is data filtering (matched-volume egocentric beats teleop by 24% val loss, 90% OOD success). ActiveMimic: Egocentric Video Pretraining with Active Perception argued the lever is signal design (treat camera motion as a learned viewpoint action). What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos? argued the lever is per-embodiment specialization plus accurate triangulated 3D hand labels. HumanEgo proposes a fourth lever: entity-relative geometric tokens + free in-policy world-model auxiliary losses, and is the most aggressive of the four on data efficiency (15–30 minutes of human video per task) and the only one to claim outperforming matched-time teleop in absolute terms rather than matching it. It is also the closest filed sibling to Do as I Do: Dexterous Manipulation Data from Everyday Human Videos in pipeline shape (RGB human video → reconstructed hand-object interaction → robot trajectory) but stops at parallel-jaw grippers with a 1 cm precision floor, while DO AS I DO retargets to multi-fingered dexterous hands; the two are clean complements on the dexterity / data-efficiency axes. The ICT construction echoes μ₀: A Scalable 3D Interaction-Trace World Model‘s thesis that the right intermediate representation between perception and action is interaction-centric 3D traces rather than dense pixels or embodiment-specific actions — HumanEgo provides the entity-relative pose variant with a flow-matching policy on top. The “vision is one of several inputs that surfaces spatial structure, not the structure itself” framing is also the most direct positive-evidence answer filed so far to the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor-level critique: HumanEgo’s policy is sensorimotor (poses, kinematics) with vision as a supporting signal, not a vision-first VLM with an action head.