HuRo: Robotizing Human Videos for Scalable VLA Pretraining

TL;DR

HuRo (Yonsei University; Jinho Jeong et al.) is a robotization pipeline that converts heterogeneous human-video sources into robot-aligned observations + retargeted action trajectories, then trains a VLA on the resulting HuRo dataset — ~630K episodes, 142M frames from five human-video sources. On four real-world manipulation tasks, increasing robotized pretraining scale lifts overall completion 51.5% → 80.3% and OOD completion under spatial/visual shifts 34.9% → 72.2%. Sits on the “convert both observations and actions” recipe branch of the human-video-to-robot debate, with the sharpest published ablation-level claim that end-to-end pretraining with retargeted actions beats visual-only transfer.

Key claims

A single robotization pipeline can convert heterogeneous human-video sources into both robot-aligned observations and retargeted action trajectories, and can infer missing intermediate signals across annotation levels [§Abstract].
Scaling robotized pretraining data monotonically improves downstream VLA performance across four real-world manipulation tasks: overall task completion 51.5% → 80.3% [§Abstract].
OOD completion under spatial and visual perturbations climbs 34.9% → 72.2% as robotized pretraining data scales — the OOD gain (+37.3 pp) exceeds the in-distribution gain (+28.8 pp) [§Abstract].
Visual robotization (converting human observations into robot-style observations) specifically improves OOD robustness in an ablation [§Abstract].
End-to-end pretraining with retargeted actions outperforms visual-only transfer — action retargeting adds signal beyond observation alignment alone [§Abstract].
The resulting HuRo dataset contains ~630K robotized episodes and 142M processed frames aggregated from five distinct human-video sources [§Abstract].

Method

HuRo is a two-part pipeline. First, observation robotization: heterogeneous human videos are visually transformed so they look like a robot’s viewpoint — closing the appearance gap that would otherwise leave a VLA seeing hands in scenes it will never see hands in at deployment. Second, action retargeting: hand motion in the source video is converted into a robot action trajectory in the target embodiment’s action space. Because the source corpora sit at different annotation levels (some have hand keypoints, some don’t; some have language, some don’t), the pipeline is designed to infer missing intermediate signals rather than filter clips down to a common denominator — which is what unlocks the 630K-episode / 142M-frame scale from five distinct sources.

The VLA is then pretrained end-to-end on the robotized dataset (both robotized observations and retargeted actions as supervision), and evaluated on four real-world manipulation tasks under both in-distribution and out-of-distribution spatial + visual conditions.

Results

In-distribution completion (4 real tasks): 51.5% → 80.3% as robotized pretraining scales [§Abstract].
OOD completion under spatial/visual shifts: 34.9% → 72.2% as robotized pretraining scales [§Abstract].
Ablation 1 (visual robotization): improves OOD robustness [§Abstract].
Ablation 2 (action retargeting): end-to-end pretraining with retargeted actions outperforms visual-only transfer [§Abstract].
Dataset: ~630K robotized episodes / 142M frames from five human-video sources [§Abstract].

Per-task numbers, baseline breakdowns, dataset composition per source, and specific comparisons against filed retargeting-recipe siblings (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data, Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data) are not disclosed in the abstract.

Why it’s interesting

HuRo lands squarely on the “robotize both channels and pay for retargeting-noise with scale” branch of the human-to-robot retargeting debate that Human-to-Robot Retargeting tracks — the same branch as Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data Ego2Robot (which also renders human video into robot-format training data via a multi-stage pipeline) and complementary to ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining ACE-Ego-0 (which absorbs the human-robot embodiment gap inside the training objective rather than in the pipeline). Distinct from Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s drop-the-noisy-channels 3-DoF wrist-only recipe by keeping and retargeting the full action channel, and distinct from EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale’s single-source (egocentric-only) 20,854 h axis by aggregating five heterogeneous sources into a common robotized substrate. The stronger OOD-gain-than-ID-gain result (+37.3 pp vs +28.8 pp) is a load-bearing datapoint for the visual-robotization-improves-OOD-robustness claim and is what makes this a distinct recipe row on the VLA Models lever board rather than a straightforward EgoScale/Ego2Robot scale-up.