ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

TL;DR

ACE-Ego-0 (ACE Robotics + CUHK MMLab, overlapping authors with the Kairos-HomeWorld scene-generation effort) is a unified VLA pretraining framework that jointly trains on heterogeneous embodied data — real-robot trajectories, simulation rollouts, and pseudo-action-labeled egocentric human videos — under a single objective. The pipeline converts raw egocentric human videos into robot-format trajectories via a scalable video-to-action labeler, then unifies their representation with teleop via camera-space actions, morphology conditioning, and time-aligned chunking, and absorbs the resulting label noise through a reliability-aware training objective with a human-stream auxiliary loss. Instantiated on 4.53K hours of robot + sim data plus 1.48K hours of pseudo-action-labeled egocentric human data, ACE-Ego-0 reports SOTA on RoboCasa GR1 TableTop and RoboTwin 2.0 and strong transfer to real-world bimanual manipulation.

Key claims

Scaling robot trajectory collection by teleoperation is costly and labor-intensive; large-scale egocentric human videos provide complementary real-world supervision in the pretraining stage [§Abstract].
Joint training on human + robot data is structurally hard because of divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality — these are the four axes the unification recipe targets [§Abstract].
A scalable egocentric video-to-action pipeline converts raw human videos into robot-format pseudo-action trajectories, supplying the missing action labels for the human stream [§Abstract].
The unified action representation has three components: camera-space actions, morphology conditioning, and time-aligned action chunking — designed to make egocentric pseudo-actions and teleop demonstrations directly comparable inside the same training batch [§Abstract].
A reliability-aware training objective with a human auxiliary loss concentrates supervision on reliable signals to robustly leverage the noisy pseudo-action labels from egocentric video [§Abstract].
Pretraining mix is 4.53K hours of robot + simulation data alongside 1.48K hours of pseudo-action-labeled egocentric human data — roughly a 3:1 robot-to-human ratio on the human side [§Abstract].
Incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning [§Abstract].
Reports state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, plus strong transfer to real-world bimanual manipulation [§Abstract].

Method

The contribution is a pretraining recipe across three components. (1) Video-to-action labeling pipeline. Raw egocentric human videos are converted into pseudo-action trajectories in a robot-compatible format; the abstract does not describe the labeler internals, but the output is intended to be plug-compatible with teleop demonstrations in the same batch. (2) Unified action representation. Camera-space actions (express wrist/end-effector motion in a head-/wrist-camera frame to absorb embodiment differences in body kinematics), morphology conditioning (so the policy knows which embodiment it is acting through at any sample), and time-aligned action chunking (same chunk length and discretization across human and robot samples) jointly bring the two streams into a shared action manifold. This is conceptually adjacent to the Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots line — drop the unreliable channels, use a low-DoF embodiment-shared signal — but ACE-Ego-0 keeps the full action space and absorbs the gap via representation choices rather than by reducing the action DoF. (3) Reliability-aware training objective. Pseudo-action labels from human video carry per-sample reliability that teleop labels do not, so a reliability-aware loss reweights samples online and a human auxiliary loss concentrates supervision on the reliable subset of the human stream. The downstream evaluation surface includes RoboCasa GR1 TableTop and RoboTwin 2.0 sim benchmarks, plus real-world bimanual manipulation; per-task tables, ablation deltas, the backbone identity, and the labeler internals were not retrievable from the abstract alone at filing time.

Results

State-of-the-art on RoboCasa GR1 TableTop [§Abstract].
State-of-the-art on RoboTwin 2.0 [§Abstract].
Strong transfer to real-world bimanual manipulation [§Abstract].
Pretraining mix: 4.53K hours of robot + simulation data plus 1.48K hours of pseudo-action-labeled egocentric human data [§Abstract].
Consistent improvement in both unified joint pretraining and supervised fine-tuning regimes under reliability-aware weighting [§Abstract].
Per-task success rates, exact ablation deltas (reliability weighting on/off, human-auxiliary loss on/off, each representation choice on/off), backbone identity, and the labeler architecture were not retrievable from the abstract; code + data repo is up but release pending at filing time (github.com/ACERobotics-VLA/ACE-Ego-0).

Why it’s interesting

ACE-Ego-0 stakes out a fifth distinct lever for the egocentric-video-vs-teleop axis that HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining, ActiveMimic: Egocentric Video Pretraining with Active Perception, What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?, and HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos have already split four ways. The previous four levers were: filtering + labeling pipeline quality at matched volume (HumanScale), signal-design via viewpoint-action factoring (ActiveMimic), per-embodiment specialization in vision + policy heads (Li et al.), and entity-relative interaction-centric tokens + free auxiliary forecasting heads (HumanEgo). ACE-Ego-0’s lever is reliability-aware joint training — explicitly model the per-sample noise in the pseudo-action labels and reweight, rather than filtering them out upstream (HumanScale) or routing them around dedicated network capacity (Li et al.). The 1.48K-hour egocentric figure also sets a new public datapoint for labeled human-stream pretraining volume — between HumanScale’s matched-volume experiments and the ~28h TriHands or 532-video HumanEgo regimes — and the framework is the first filed to claim both joint-pretraining and SFT gains under the same reliability mechanism. The recipe is complementary to Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (drop-channels-on-the-human-side) and Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (kinematic-reconstruction-as-retargeting) on the Human-to-Robot Retargeting cluster: ACE-Ego-0 keeps the full action space and pays for noise with reliability weighting, where Translation-as-Bridging-Action shrinks the action space to drop the noise and Do as I Do tries to fix it upstream via better hand-object reconstruction.