Skip to content
Bud's Wiki

LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition

TL;DR

Section titled “TL;DR”

LUCID (CMU; Gupta, Shi, Yuan) pairs two independently scalable supervision sources for dexterous manipulation by separating intent (what should change in the scene) from control (how the robot achieves it). An intent model is trained on unstructured internet human video to predict short-horizon 3D object flow plus a palm-pose reference from a live RGB-D observation; a generalist sensorimotor policy is trained in massively-parallel simulation to realize that reference on a specific embodiment. Because the intent interface is shared, the same intent model drives both a LEAP dexterous hand and a parallel-jaw gripper. On three web-supervised tasks (stirring, wiping, binning) LUCID reports 73% average success in closed-loop vs 28% for a Veo-3.1-generated open-loop plan, and cross-embodiment tasks (push-T, cable routing) supervised by 1 hr of self-collected smartphone video hit 63% on both hand and gripper.

Key claims

Section titled “Key claims”
  • Decoupling intent (from human video) from control (from simulation) lets the intent model transfer zero-shot to novel scenes, camera viewpoints, and object instances without robot teleop or per-embodiment adaptation [§1, §3].
  • Closed-loop intent — re-querying the intent model against the live scene rather than emitting a one-shot plan — is the load-bearing choice: LUCID gets 73% vs a Veo-3.1 open-loop planner’s 28% averaged across stirring, wiping, and binning (10 trials × 3 OOD scenarios each) [§4.1, Fig. 5A].
  • The same intent model drives both a dexterous hand and a parallel-jaw gripper for push-T and cable routing, with matching 19/30 aggregate success per embodiment; cable routing actually favors the gripper because two opposing jaws suit the thin cable [§4.3, Fig. 5B].
  • Real-world binning success rises predictably with training-corpus size and the held-out intent loss follows a power law over the observed range (1k → 20k clips) [§4.2, Fig. 6].
  • The sensorimotor policy uses an eigen-grasp basis (fit to retargeted human grasps) plus per-joint residuals; the ablation removing the basis and using per-joint-only actions loses stable-grasp behavior [§3.2, §4.4, Fig. 7A].
  • Both teacher and student policies use an AME-derived cross-attention encoder where each object query point becomes a permutation-invariant token; replacing this with a plain MLP encoder loses the per-point detail needed for stable grasp contacts [§3.2, §4.4, Fig. 7A].
  • Student policy is distilled with a hybrid PPO-on-student-rollouts + MSE-to-teacher objective (PHP-style) with the distillation weight annealed; a DAgger-BC teacher/student rollout mix underperforms because the student can’t fully exploit its own wrist-depth camera [§3.2, §4.4, Fig. 7B].

Method

Section titled “Method”

Two learned components communicate through a hand-designed short-horizon reference of object flow plus palm pose. The intent model adapts CoTracker3 as a point-token transformer conditioned on frozen DINOv3 patch tokens with depth fused through a residual adapter; a single palm-pose token is appended to object tokens so flow and palm are predicted jointly. It is trained on ~20k human video clips per task (predominantly YouTube; also EPIC-KITCHENS, Something-Something, Panda-70M) supervised by hindsight ground truth extracted through a four-stage pipeline: ViPE for camera and metric depth, SAM 3.1 for object and human masks, DenseTrack3Dv2 for 3D point tracks, and WiLoR for per-frame MANO hand meshes rigidly aligned to the object mask to lift a palm pose. Human pixels in the input are heavily augmented so the intent model doesn’t lean on demonstrator hand appearance (important because at deploy the visible hand is a robot).

The generalist sensorimotor policy is trained once per embodiment via goal-conditioned RL in Isaac Lab on procedurally-generated blob/tool/plate primitives under randomized physics, chained through four-segment reference trajectories (approach → in-hand → goal → disengage). Action space combines arm joint-position deltas with eigen-grasp coefficients (fit to retargeted human grasps) plus per-joint hand residuals. A PPO teacher trained on privileged simulator state is distilled into a student that observes only external RGB-D + wrist depth + proprioception, with a curriculum that tightens gravity, wrench perturbations, and success tolerances as the population mean of per-environment difficulty rises. At deploy, a slow intent cycle re-queries the intent model against the live scene while a fast control loop runs the student policy every step; between intent cycles a sliding-window 3D point tracker maintains current point positions.

Results

Section titled “Results”
  • Web-supervised tasks (stirring, wiping, binning), 10 trials × 3 OOD scenarios each: closed-loop LUCID 73% average success vs Veo-3.1 open-loop planner 28% [§4.1].
  • Cross-embodiment tasks (push-T, cable routing) supervised by 1 hr smartphone video: LEAP dexterous hand and parallel-jaw gripper each reach 19/30 (63%); cable routing favors the gripper due to jaw geometry [§4.3, Fig. 5B].
  • Intent-data scaling on binning: held-out intent loss decreases as a power law from 1k → 20k clips, and real-world 30-trial success rises monotonically; at 1k–2k clips references are too poor to identify the bin, at 5k–10k container localization emerges but placement alignment remains weak [§4.2, Fig. 6].
  • Emergent task composition without task-specific supervision: placing a bin near the workspace after wiping causes the robot to pick up the soiled tissue and deposit it, attributed to incidental binning in the broader wiping video pool [§4.1].
  • The sensorimotor policy handles deformable objects (tissue, towel, cloth) at deploy despite being trained only on rigid procedural objects [§4.1].
  • Ablations (sim, 3 seeds each): MLP encoder loses per-point detail for stable contacts; removing eigen-grasp basis massively enlarges exploration space; removing wrist camera leaves object geometry unresolved; DAgger-BC distillation underexploits the student’s wrist-depth input [§4.4].

Why it’s interesting

Section titled “Why it’s interesting”

LUCID takes a distinct position on the Human-to-Robot Retargeting design axis: rather than retargeting human motion to robot joint commands (kinematic reconstruction as in Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, physics-in-the-loop as in ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting, object-tracking-as-constraint as in ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control, or a reduced 3-DoF bridging action as in Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), it retargets task intent while delegating morphology-specific execution entirely to a sim-trained policy. This is complementary to the scale-of-human-data lever in EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: LUCID uses 20k clips per task (~two orders of magnitude less than EgoScale’s 20,854h) but converts them into an embodiment-agnostic intent surface that a sim-only policy can consume. As a VLA-adjacent recipe it also stakes out a counter-position to end-to-end VLM-with-action-head approaches like π*0.6: a VLA That Learns From Experience (RECAP) and Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models — no robot action data, no VLM backbone, and the same intent module trivially spans a dexterous hand and a parallel-jaw gripper. The 73% vs 28% closed-loop-vs-Veo3.1 gap is also the sharpest filed evidence that video-generation-based open-loop planning (Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models-style WAM rollouts) is not currently a substitute for continually re-queried intent during execution.

See also

Section titled “See also”