EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

TL;DR

EgoScale (NVIDIA GEAR; Jim Fan / Yuke Zhu / Danfei Xu / Trevor Darrell / Furong Huang labs) pretrains a flow-matching VLA on 20,854 hours of action-labeled egocentric human video — over 20× larger than any prior human-to-robot transfer study — and reports a clean log-linear scaling law between human-data scale and action-prediction validation loss (R² = 0.9983) that strongly predicts downstream real-robot performance. The training recipe is three-stage: large-scale unconstrained egocentric pretraining → small (~50h human / ~4h robot) aligned mid-training with matched viewpoints and motion-capture-grade hand tracking → per-task post-training. The full recipe lifts average success rate by 54% over a no-pretraining baseline on five highly dexterous 22-DoF tasks (shirt rolling, card sorting, tongs, bottle unscrewing, syringe transfer), enables one-shot transfer to new tasks (88% on shirt folding from a single robot demo + 100 aligned human demos), and transfers to a different embodiment (Unitree G1 tri-finger hand, +30%+ absolute success).

Key claims

Human-to-robot transfer for dexterous manipulation is fundamentally a scaling phenomenon: at 1k → 20k hours of egocentric pretraining data, average downstream task completion rises monotonically from 0.30 to 0.71 with no signs of saturation [§3.3, Fig. 5 right].
Optimal pretraining validation loss against human-data scale is a near-perfect log-linear law: $L \propto -\log(\text{hours})$ with $R^2 = 0.9983$ across {1k, 2k, 4k, 10k, 20k} hours [§3.3, Eq. 1, Fig. 5 center].
Validation loss on held-out egocentric video is a meaningful predictor of real-robot task success rate, not just an offline metric — the two curves track each other closely across data scales [§3.3, Fig. 5].
The full recipe (human pretraining + aligned mid-training + post-training) beats from-scratch training by ~55% average task completion across all 5 tasks, and beats human-pretraining-only and mid-training-only baselines, indicating large-scale human pretraining and aligned mid-training are complementary rather than substitutes [§3.2, Fig. 4].
Aligned mid-training (~50h human / ~4h robot of matched-viewpoint tabletop play data) is what unlocks one-shot transfer: with a single robot demonstration plus 100 aligned human demos, the policy reaches 88% on Fold Shirt and 55% on Unscrew Water Bottles — neither task appears in mid-training data [§3.4, Fig. 6].
Cross-embodiment transfer holds when an aligned-mid-training stage includes the new embodiment: on Unitree G1 (shorter arm, 7-DoF tri-finger hand), human pretraining yields >30% absolute success-rate gain on Pen-in-Bin and Dish-in-Rack vs G1-only training, indicating the 22-DoF Sharpa-hand pretraining encodes a reusable, embodiment-agnostic motor prior [§3.5, Fig. 7].
The choice of human hand action space matters: retargeted 22-DoF joint-space hand actions beat both wrist-only and fingertip-based representations across all 5 tasks; fingertip-based collapses on contact-sensitive tasks (cards, bottle) due to implausible joint configurations after MLP mapping [§3.6, Fig. 8].
Action representation is unified across human and robot via relative wrist motion in the local frame plus joint-space dexterous hand actions, with absent robot proprioception during human pretraining replaced by a learnable placeholder token — no architectural changes between human and robot training stages [§2.1, §2.3].
The model is a flow-matching VLA following GR00T N1’s structure: VLM backbone + DiT action expert + lightweight embodiment-conditioned MLP adapters for proprioception and hand action heads (shared wrist-motion prediction across embodiments) [§2.3].
Stage I training is large: 100k steps, 256 GB200 GPUs, global batch 8192, LR 1e-4, all parameters unfrozen; Stage II mid-training freezes the VLM backbone and updates only the vision encoder and DiT action expert [§2.4].
Pretraining substrate combines in-the-wild noisy egocentric recordings (~20k h across 9,869 scenes / 6,015 tasks / 43,237 objects, with off-the-shelf SLAM + hand-pose estimation) with 829 hours of high-precision EgoDex (Apple Vision Pro wrist + hand tracking) to anchor the noisy bulk [§2.2 Stage I].
The recipe explicitly decouples scale from alignment: Stage I provides diversity and semantic grounding from noisy at-scale data, Stage II supplies precise embodiment correspondence from a much smaller (~50h) aligned-viewpoint dataset [§2.2 Stage II].

Method

EgoScale is a three-stage training pipeline for a flow-matching VLA (GR00T N1-style: VLM backbone + DiT action expert + per-embodiment MLP adapters). The unified action representation is the load-bearing design choice: relative wrist motion in the local frame (invariant to camera/global pose) plus retargeted 22-DoF Sharpa-hand joint actions, shared across humans and robots; absent robot proprioception during human pretraining is filled with a learnable placeholder token. Stage I pretrains on 20,854 hours of egocentric human video (in-the-wild 20k h + EgoDex 829 h) with all parameters unfrozen, 100k steps × 256 GB200 GPUs × batch 8192 × LR 1e-4. Stage II mid-training co-trains on ~50 h human + ~4 h robot data with matched camera viewpoints and Vive + Manus motion-capture-grade hand tracking, freezing the VLM and updating only the vision encoder + DiT action expert. Stage III post-trains on 100 per-task teleop demonstrations (20 for shirt rolling) at LR 1e-5. Evaluation runs two seeds × 10 trials per task on a Galaxea R1Pro humanoid with Sharpa Wave 22-DoF hands, three RGB cameras (head + two wrist), and reports both task completion score and absolute success rate.

Results

Pretraining recipe ablation [§3.2, Fig. 4]: Across 5 tasks, full recipe > human-pretrain-only > mid-train-only > from-scratch. Human pretraining alone already beats mid-train-only on most tasks despite being noisy and unconstrained. Combined recipe lifts average task completion by ~55% over from-scratch.
Log-linear scaling law [§3.3, Fig. 5]: Validation loss vs hours follows $R^2 = 0.9983$ log-linear fit across {1k, 2k, 4k, 10k, 20k} h. Real-robot average task completion rises monotonically 0.30 → 0.71 with no saturation. Models trained on 1k–2k h overfit (loss reverses); 10k–20k h shows stable monotonic improvement.
One-shot transfer [§3.4, Fig. 6]: Pretrain + Midtrain, single robot demo + 100 aligned human demos: 88% success on Fold Shirt, 55% on Unscrew Water Bottles (3 bottle geometries). Models missing either large-scale pretraining or mid-training fail in this one-shot regime.
Cross-embodiment transfer [§3.5, Fig. 7]: Unitree G1 (7-DoF tri-finger, shorter arm, paired with Homie lower-body policy) — Pretrain + G1-inclusive Midtrain beats G1-only training by >30% absolute on Pen-in-Bin and Dish-in-Rack; same G1 data alone (no human pretraining) fails to match.
Hand action representation ablation [§3.6, Fig. 8]: Retargeted 22-DoF joint actions consistently top wrist-only (collapses on Tongs / Cards from premature/weak grasps) and fingertip-based (collapses on Cards / Bottle from implausible joints after MLP mapping).
Headline number: 54% absolute average success-rate improvement over no-pretraining baseline on 22-DoF hand tasks; transfers to lower-DoF hands.

Why it’s interesting

EgoScale is the strongest filed datapoint connecting egocentric human data scale to dexterous manipulation — orthogonal to the matched-data-volume comparison HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining ran (which holds volume fixed and varies source), and roughly 20× the scale of the prior human-to-robot-transfer literature that the What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos? / ActiveMimic: Egocentric Video Pretraining with Active Perception / HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos cluster operates at. The log-linear loss-vs-hours law (R² = 0.9983) is the first filed instance of a clean Hyperparameter scaling laws-style fit for an egocentric-video pretraining objective — extending the Kaplan/Chinchilla scaling-law toolkit to the embodied-data axis that Scaling Laws, Carefully explicitly excluded. On the Human-to-Robot Retargeting axis, EgoScale sits between the dropped-channels recipe of Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (3-DoF wrist translation only, rotation discarded as too noisy) and the keep-everything-with-noise-aware-loss recipe of ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: EgoScale keeps the full 22-DoF dexterous hand channel from retargeted human keypoints and shows it beats both wrist-only and fingertip-based alternatives on contact-sensitive tasks — direct evidence that, at scale, the high-DoF joint-space channel is not too noisy to learn from. Finally, the aligned-mid-training recipe is a more compact version of Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners‘s “small amount of paired data anchors a large unconstrained corpus” thesis, and the one-shot post-training result (88% from one demo) is a stronger downstream-efficiency claim than HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos‘s minutes-of-demos result at much smaller pretraining scale.