Skip to content
Bud's Wiki

Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

TL;DR

Section titled “TL;DR”

Ego2Robot (AIM3 Lab RUC + Alibaba Qwen team + ShanghaiTech + BIGAI + BUAA) is a three-stage pipeline that converts egocentric human manipulation videos into rendered robot-format training data by combining hand-pose retargeting with visual arm synthesis — SAM 3 arm segmentation, ProPainter hand removal, base-pose search, IK solving, and depth-aware compositing that literally paints the target robot arm into each frame. Applied to ~1,940 hours of source ego video from four corpora (ANT, EgoDex, ViTRA, EgoVerse) rendered across 15 robot morphologies, it produces 18,561 hours of synthetic robot data — the largest ego-to-robot dataset filed to date. On a purpose-built RoboTwin 2.0 extension with 11 disentangled perturbation axes plus EBench, joint pretraining on Ego2Robot + real robot data at 1:1 mix lifts overall success rate from 62.2% (robot-only) to 68.1%, with the largest gains on out-of-distribution embodiment, visual, and task axes.

Key claims

Section titled “Key claims”
  • Prior ego-to-robot retargeting work has been limited to per-task effectiveness at small scale; whether the paradigm provides pretraining benefits for VLAs at scale is the open question this paper targets [§Abstract, blog intro].
  • The pipeline supports two input paths: Path A consumes ego datasets with existing hand-pose annotations; Path B estimates hand poses from raw video via WiLoR per-frame reconstruction plus DynHaMR temporal optimization — after which both paths share a unified rendering stage across 15 morphologies in parallel [blog §Pipeline].
  • Kinematic retargeting maps 4 hand keypoints (thumb, index/middle tips, wrist) into gripper TCP + width + orientation, then smooths trajectories with Savitzky-Golay filtering plus SLERP interpolation [blog §Retargeting].
  • Visual arm synthesis is the load-bearing engineering contribution: SAM 3 segments the human arm, ProPainter inpaints the removed hand region, a base-pose search + IK solver plants the robot arm, and depth-aware compositing places it into the inpainted scene [blog §Rendering].
  • Quality curation is three-tier: L1 pipeline-internal (IK failures, self-collision, kinematic outliers), L2 statistical (extreme values, temporal discontinuities), L3 VLM-scored video–action consistency [blog §Curation].
  • Source corpus is ANT (7h, in-house) + EgoDex (732h) + ViTRA (249h) + EgoVerse (954h) ≈ 1,940 h, rendered across 15 morphologies to yield 18,561 hours of synthetic robot data — the largest ego-to-robot dataset to date [blog §Dataset].
  • Existing OOD protocols conflate distribution shifts into a single score; the paper extends RoboTwin 2.0 with 11 independent perturbation settings plus EBench (higher-mounted camera closer to the ego view), decoupling four generalization axes: visual, scene, embodiment, task [blog §Benchmark].
  • Backbone is a VLA with Qwen3.5-4B language backbone + Diffusion Transformer action head, predicting 32-step camera-frame relative EEF chunks [blog §Model].
  • Comparisons are equal-compute rather than equal-data: every pretraining run uses 200K steps × batch 12 × 8 GPUs (~19.2M frames), so mix ratios do not confer a compute advantage [blog §Protocol].
  • At the 1:1 Ego2R:Robot pretraining mix, the model beats robot-only pretraining on Clean (68.1 vs 62.2), Randomized (53.5 vs 50.9), Visual (67.3 vs 61.4), Scene (56.9 vs 52.9), Embodiment (27.2 vs 23.8), and Task (54.1 vs 46.2) [blog Table 1].
  • Zero-shot embodiment transfer to unseen morphologies improves substantially: UR5 22.1 → 25.0, ARX 44.1 → 51.2 vs robot-only baseline; Franka collapses at ~5% for both, indicating a residual embodiment gap not closed by pipeline data alone [blog Table 2].
  • Ego-only ablation (no real robot data at all) supports a “raw ego is a 16th morphology” reading: raw-ego pretraining scores 28.1% on RoboTwin Randomized, single-morphology pipeline lifts this to 31.7% (+3.6), 15-morphology rendering to 33.5%, and adding raw ego alongside the 15-morphology rendered data reaches 37.3% [blog §Ego-only].
  • Few-shot real-robot deployment on an ARX ACone platform (5 long-horizon tasks, only 20 teleop demos per task): Mix + Ego2R Play — where pipeline-converted ego-play recordings (~7 min casual first-person hand manipulation in the target scene) are mixed into the finetuning data — beats both robot-only and Mix on all five tasks [blog §Real-robot].

Method

Section titled “Method”

Ego2Robot is a data-generation pipeline plus a pretraining recipe. Stage 1 estimates 3D hand poses from ego video (Path A: use existing annotations; Path B: WiLoR + DynHaMR). Stage 2 kinematically retargets thumb/index/middle tips + wrist into per-morphology TCP, gripper width, and orientation streams, then smooths temporally. Stage 3 is the visual synthesis contribution: for each frame, SAM 3 segments the human arm; ProPainter removes the hand and inpaints the underlying scene; a base-pose search selects a plausible robot mount point; an IK solver produces per-joint targets that match the retargeted TCP trajectory; and depth-aware compositing renders the target robot arm into the inpainted frame. This gives each source ego frame a visually plausible robot-format counterpart. Three curation layers then filter the output: L1 catches IK failures and collisions inside the pipeline, L2 rejects statistical outliers and discontinuities, L3 uses a VLM to score video–action consistency per clip. The resulting 18,561-hour synthetic corpus is co-trained with ~6,565 hours of real robot data (DROID + AgibotWorld + InternData) at four mix ratios (robot-only, 1:3, 3:1, 1:1 Ego2R:Robot). The VLA (Qwen3.5-4B + DiT action head) predicts 32-step camera-frame relative EEF chunks. Every pretraining run is 200K steps × batch 12 × 8 GPUs, then finetuned on RoboTwin 2.0’s 50 clean-setting tasks (Aloha-Agilex embodiment), and evaluated across the 11 disentangled perturbation settings plus EBench.

Results

Section titled “Results”
  • RoboTwin 2.0 overall table [blog Table 1]: Ego2R+Robot (1:1) reaches Clean 68.1, Rand 53.5, Visual 67.3, Scene 56.9, Embody 27.2, Task 54.1, EBench 49.8 — winning 6 of 7 columns vs the robot-only baseline (Clean 62.2, Rand 50.9, Visual 61.4, Scene 52.9, Embody 23.8, Task 46.2, EBench 39.6). Ego2R+Robot (3:1) is best on Embodiment (28.2), suggesting more synthetic data helps morphology transfer but the 1:1 mix is the overall optimum.
  • Per-perturbation breakdown [blog Table 2]: Largest 1:1-mix gains vs robot-only are Language (+5.4: 68.5 vs 63.1), BG (+3.7: 70.3 vs 66.6), Light (+7.6: 65.8 vs 58.2), Color (+6.4: 65.8 vs 59.4), Clutter (+3.7: 52.0 vs 48.3), Camera (+5.9: 56.3 vs 50.4), Object (+10.3: 39.6 vs 29.3), ARX (+7.1: 51.2 vs 44.1), UR5 (+4.8: 25.0 vs 20.2). Franka embodiment stays low for all mixes (~5-7%), a residual gap.
  • Ego-only ablation [blog §Ego-only]: Raw ego alone 28.1% → single-morphology pipeline 31.7% → 15-morphology pipeline 33.5% → 15-morphology pipeline + raw ego 37.3% on RoboTwin Randomized. Each stage of the pipeline adds monotonically.
  • Few-shot real-robot [blog §Real-robot]: On ARX ACone across 5 long-horizon tasks with only 20 teleop demos per task, Mix + Ego2R Play (finetuning-time addition of ~7-min ego-play recordings converted through the pipeline) beats Robot-only and Mix on all 5 tasks.
  • Dataset scale: 18,561 hours synthetic robot data across 15 morphologies from ~1,940 h of source ego video [blog §Dataset].

Why it’s interesting

Section titled “Why it’s interesting”

Ego2Robot occupies a distinct point on the Human-to-Robot Retargeting interface-location axis from EgoScale (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) and ACE-Ego-0 (ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining) — the two closest siblings — by pushing the retargeting all the way to the pixel level: rather than unifying action spaces (EgoScale’s 22-DoF joint retargeting + shared wrist-motion channel) or reweighting pseudo-action noise inside the loss (ACE-Ego-0), Ego2Robot literally synthesizes robot-arm imagery into the ego frames so that downstream training sees a visually robot-native corpus. This is the strongest filed instance of the “labels + renders on real video” pattern that Synthetic Training Data tracks — closer to a data-generation pipeline than to a training-recipe change. On the scale axis, the 18,561 h corpus is comparable to EgoScale’s 20,854 h (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) but multiplied across 15 morphologies from just ~1,940 h of source video, giving the pipeline a per-source-hour leverage that neither ACE-Ego-0’s 1.48K h pseudo-action corpus nor HumanScale’s (HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining) matched-volume comparison could reach. The disentangled 11-perturbation extension of RoboTwin 2.0 is a distinct methodological contribution — most filed VLA papers report a single OOD success number that mixes visual, scene, embodiment, and task axes; the Ego2Robot benchmark surface is the first to cleanly attribute pipeline gains to specific shift axes (embodiment ARX +7, task-object +10, language +5, lighting +8), making it a candidate evaluation substrate for other pretraining-substrate studies.

See also

Section titled “See also”