EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization

TL;DR

EgoCPT retargets dexterous hand trajectories from a single ego-centric human video onto a real robot hand via a contact-aware optimization that iteratively refines the contact points between the robot hand and manipulated object. The inputs are an ego-centric RGB video plus a reference hand-pose trajectory (from off-the-shelf estimators or specialized capture hardware), and the method’s robustness to noisy hand poses comes from treating object motion as the goal and hand motion as a reference, searching for a robot-feasible contact configuration that realizes the demonstrated object trajectory rather than copying the demonstrated contacts verbatim. Reports overall task success of 67.3% on noisy EgoDex monocular hand-pose inputs and 92.0% on high-quality TACO mocap-grade inputs across five manipulation tasks, outperforming multiple baselines on both benchmarks.

Key claims

Learning dexterous manipulation from human ego-centric video is bottlenecked by reconstructing physically grounded robot trajectories from noisy hand-pose estimates — the embodiment gap plus hand-pose noise make naive copy-the-contacts retargeting fail [§Abstract].
A contact-aware optimization loop that iteratively refines robot-hand/object contact points produces executable robot trajectories from a single video with only RGB + reference hand pose as inputs [§Abstract].
The right design choice is to treat object motion as the goal and hand motion as reference, searching near the demonstrated hand gesture for contacts reachable by the robot — rather than enforcing exact copying of demonstrated contacts, which breaks under noisy reconstruction and embodiment change [§Overview].
Pipeline is a three-stage inner loop per interaction segment: (1) sample contacts near the demonstrated gesture that are reachable by the robot, (2) refine the pose to remove penetration and close contact gaps, (3) allocate contact forces consistent with target object motion, then re-refine as needed [§Overview].
Reports 67.3% average success on EgoDex (noisy monocular hand-pose reconstructions) and 92.0% on TACO (mocap-grade reconstructions) across five manipulation tasks, outperforming multiple baselines on both benchmarks [§Abstract].
The ~25-point gap between noisy (EgoDex) and clean (TACO) inputs quantifies the method’s residual sensitivity to upstream hand-pose quality — the optimization absorbs much of the noise but not all of it [§Abstract].
Rollouts cover pouring, lint rolling, sweeping, scooping, pick & place, bowl & ladle, container manipulation, and surface brushing — mostly tool-use and contact-rich tasks where contact location dominates success [§Rollout gallery].

Method

EgoCPT takes an ego-centric RGB video of a human performing a manipulation task plus a reference 3D hand-pose trajectory (from off-the-shelf hand estimators on EgoDex-style monocular video, or mocap capture on TACO-style sequences). The output is a dexterous robot-hand trajectory executable on a real robot. The core move is to invert the usual retargeting problem: instead of mapping human hand pose → robot hand pose and hoping the contacts transfer, EgoCPT fixes the object motion as the retargeting target and searches the robot’s configuration space for a trajectory of hand poses + contact forces that realizes that object motion.

Per interaction segment the inner loop runs: (a) Sample contacts — perturb around the demonstrated hand gesture and sample candidate contact point sets reachable by the robot’s kinematics; (b) Refine the pose — adjust wrist and finger joint angles to eliminate interpenetration and close any gaps between hand links and object surface at the sampled contact points; (c) Allocate forces — solve for a set of contact forces at those contact points consistent with the target 6-DoF object motion (dynamics-feasible wrench equilibrium), looping back to pose refinement if the force allocation step fails to find a feasible solution. This makes the retargeting procedure contact-aware in both the geometric sense (contact points are optimized, not copied) and the dynamic sense (contact forces must realize the target object motion).

Results

Overall task success — EgoDex (noisy monocular hand pose): 67.3% average across five manipulation tasks [§Abstract].
Overall task success — TACO (mocap-grade hand pose): 92.0% average across five manipulation tasks [§Abstract].
Baseline comparison: outperforms multiple baselines on both benchmarks — specific baseline numbers and per-task breakdowns not disclosed on the project page [§Abstract].
Task coverage: pouring, lint rolling, sweeping, scooping, pick & place (EgoDex); plus bowl & ladle, container manipulation, surface brushing (TACO) [§Rollout gallery].
Qualitative hand-object reconstruction: side-by-side comparisons on each benchmark showing EgoCPT finds new grasps via the contact search where the input reconstruction has broken contact or interpenetration [§Hand-object reconstruction gallery].

Why it’s interesting

EgoCPT stakes out an explicitly contact-first position on the interface-location axis tracked by Human-to-Robot Retargeting: the retargeting target is the object’s trajectory and the contact point set, not the hand pose itself — hand pose is only a search prior. This puts it close to C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video C2Dex, which also uses object-side contact topology (aggregated across frames in canonical object space) as the shared interface, but EgoCPT differs in attacking the problem per-segment through geometric + dynamic contact-force optimization rather than across-frame contact aggregation + Laplacian geometry preservation + residual RL. It is also complementary to Do as I Do: Dexterous Manipulation Data from Everyday Human Videos Do as I Do’s kinematic 4D-reconstruction recipe by moving the retargeting target one layer out from hand kinematics to object dynamics, and sibling to CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation CHORD on the “object-centric contact representation” axis — CHORD at the wrench level, EgoCPT at the combined geometric-point + force-allocation level. The 92% success on TACO at single-demo scale is the strongest filed single-video dexterous retargeting result in the TACO regime, with the remaining question being how it composes with scaled pretraining substrates like EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale.