EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning

TL;DR

EgoInfinity is a modular data engine — not a fixed dataset — that converts arbitrary internet RGB videos (egocentric or third-person, any-view, partial human visibility) into agent-agnostic 4D hand-object trajectories and then retargets those trajectories into executable joint commands for diverse robot morphologies. The pipeline composes perception, segmentation, reconstruction, interaction-aware refinement, and a motion retargeter under cross-module metric calibration so each component can be swapped as upstream perception models improve. The authors validate the engine on perception fidelity, kinematic feasibility, contact consistency, cross-embodiment generalization, and real-robot skill acquisition (grasping, cutting, wiping, pouring).

Key claims

The pipeline lifts in-the-wild RGB into agent-agnostic, metric 4D representations — hand trajectories, 6-DoF object poses, and contact-relevant states — without human-in-the-loop annotation [§Abstract].
Cross-module metric calibration plus an interaction-aware refinement stage are presented as the load-bearing additions over naively concatenated perception models, reducing drift and contact inconsistencies typical of pure visual reconstruction [§Abstract].
A novel motion retargeter compiles recovered 3D hand motions into joint trajectories that execute on robots of differing morphologies, and tolerates arbitrary viewpoints and shot sizes including footage where the human body is only partially visible [§Abstract].
The engine is validated end-to-end against five axes — perception fidelity, kinematic feasibility, contact consistency, cross-embodiment generalization, and real-robot skill acquisition (grasping, cutting, wiping, pouring) [§Abstract].
The modular framing is positioned as the lever for web-scale data generation: each block (perception, segmentation, reconstruction, refinement, retargeting) can be replaced with a stronger version as the corresponding sub-field advances [§Abstract].

Method

Single-paragraph reconstruction from the abstract (full technical details not in the fetched body): RGB video → per-frame perception and segmentation (presumably hand/object detection and SAM-class masks) → reconstruction recovering 3D hand pose and 6-DoF object pose → interaction-aware refinement that uses contact constraints to suppress visual-reconstruction drift → cross-module metric calibration tying the modules into a single metric coordinate frame → a motion retargeter that maps the agent-agnostic 4D hand-object trajectory to joint commands for a target robot morphology, robust to viewpoint and to footage where the human body is partially out of frame. The authors emphasize the engine is the contribution rather than any one component, and that the abstraction lets the system inherit gains from advances in any incorporated module.

Results

The paper’s stated evaluation axes are perception fidelity, kinematic feasibility, contact consistency, cross-embodiment generalization, and real-robot skill acquisition with named tasks (grasping, cutting, wiping, pouring) [§Abstract]. Specific quantitative numbers are not in the fetched abstract body; the headline framing is qualitative — a demonstrated scalable bridge from internet videos to executable robot behavior across these five axes.

Why it’s interesting

EgoInfinity is the third paper this week (alongside Do as I Do: Dexterous Manipulation Data from Everyday Human Videos and ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control) to argue that the kinematic-reconstruction-plus-retargeting path from monocular human video to robot actions is now a credible alternative to teleop-data scaling and to direct VLA action pretraining. EgoInfinity’s distinct bet is the engine vs dataset framing: where HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining and ActiveMimic: Egocentric Video Pretraining with Active Perception argue the lever is data-curation/filtering or viewpoint-action signal design, EgoInfinity treats the lever as a modular pipeline that compounds upstream perception gains — closer in spirit to the data-infrastructure story in EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World and HumanNet: Scaling Human-centric Video Learning to One Million Hours than to the model-architecture VLA papers (π*0.6: a VLA That Learns From Experience (RECAP), Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models). The interaction-aware refinement and any-view retargeting directly address the brittleness that ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control frames as the unsolved bottleneck.