Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

TL;DR

DO AS I DO is a pipeline that converts monocular RGB human videos — egocentric or third-person, in-the-wild — into robot-executable manipulation trajectories for multi-fingered dexterous hands. It reconstructs hand-object interaction in 4D from a single video, then retargets the recovered hand-object dynamics into action sequences for a target dexterous robot. The paper claims SOTA over prior work on both (a) hand-object interaction estimation from RGB and (b) extraction of dexterous manipulation trajectories from RGB videos, evaluated on datasets with ground truth and on internet-collected video clips. The output is “robot-complete” trajectory data — actions executable on a real platform — from disparate human videos the operator never captured themselves, targeting the data bottleneck that has stalled dexterous-manipulation learning.

Key claims

Monocular RGB-only human videos can be turned into robot-complete dexterous manipulation data by a single algorithm that handles both the perception (4D hand-object reconstruction) and the embodiment-gap (retargeting to multi-fingered robot hand) steps in one pipeline [§Abstract].
The method outperforms prior state of the art in estimating hand-object interactions from RGB human videos, measured on datasets with ground truth [§Abstract].
The method outperforms prior state of the art in extracting dexterous manipulation trajectories from RGB videos, measured on a dataset of video clips collected online [§Abstract].
The pipeline accepts both egocentric and exocentric in-the-wild video sources without per-source specialization [§Abstract].
The paper proposes an “efficacy playbook” for practitioners collecting human data for manipulation, derived from the experiments [§Abstract].

Method

DO AS I DO is described in the abstract as a two-stage cascade. Stage one reconstructs hand-object interactions in 4D from a monocular RGB clip — that is, time-varying 3D hand pose plus 3D object pose plus contact — covering both egocentric and exocentric in-the-wild sources. Stage two retargets those hand-object-interaction estimates onto a target multi-fingered dexterous robotic hand, emitting “a sequence of actions executable in the real world.” Together the two stages convert a clip of a person manipulating an object on YouTube/Ego4D-style footage into a trajectory that a robot can replay. The abstract does not specify the architectural choices (pose estimator, IK solver, contact model); code and a project site are released alongside the paper (do-as-i-do.com, github.com/malik-group/do-as-i-do). Method details and quantitative comparisons are in the body, which was not retrieved at filing time.

Results

Headline numbers are not surfaced in the abstract. The paper claims, qualitatively: (a) SOTA on hand-object interaction estimation against ground-truth datasets, and (b) SOTA on dexterous-manipulation trajectory extraction on an online-video evaluation set [§Abstract]. The Malik tweet announcing the release frames the deliverable as “convert human videos to robot hand-object interaction trajectories in 4D” with sim/real video demos on the project page; no comparative benchmark scores are quoted in the tweet. Filing this on abstract + tweet only — a refresh would help once the body is parsed.

Why it’s interesting

This is the data side of the dexterous-VLA stack — directly addressing the bottleneck the VLA Models cluster keeps circling back to. The wiki has filed three plausible levers for getting better VLAs: clean teleop data (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), egocentric video pretraining (ActiveMimic: Egocentric Video Pretraining with Active Perception and EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World), and generated-video imitation (Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations). DO AS I DO sits in a fourth lane: rather than pretraining on raw human video or generating synthetic video, it converts raw RGB human video directly into the action-labeled trajectory format VLAs and behavior-cloning policies consume — closing the gap between ActiveMimic: Egocentric Video Pretraining with Active Perception‘s “egocentric video as a pretraining substrate” and ABC’s behavior-cloning training distribution (ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation). The retargeting step is the same problem ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting (Disney ReActor) attacks for character animatronics, but DO AS I DO targets multi-fingered dexterous hands with hand-object contact rather than full-body humanoid morphologies, and uses kinematic reconstruction rather than physics-in-the-loop RL — orthogonal tradeoffs worth watching. It also extends Pose Estimation and Motion Capture from body/skeleton pose into 4D hand-object pose with contact, which is the harder regime that markerless body capture work like MAMMA: Markerless & Automatic Multi-Person Motion Action Capture explicitly steers around. Malik himself authored the recent Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level post arguing VLA work under-weights the sensorimotor / contact level; this paper is his group’s positive proposal for cheaply getting contact-grounded data at scale.