Skip to content
Bud's Wiki

Human-to-Robot Retargeting

TL;DR

Section titled “TL;DR”

Human-to-robot retargeting is the problem of closing the kinematic gap between cheap, abundant human demonstration sources (mocap, egocentric RGB, third-person video) and target robot embodiments whose morphology + contact mechanics differ from a human. The filed papers disagree on where the right interface lives: physics-in-the-loop bilevel optimization over a parameterized retargeting (ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting), kinematic 4D hand-object reconstruction + retargeting (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), dual-variable RL between object-tracking constraint and motion fidelity (ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control), or dropping the unreliable channels entirely and using a low-DoF embodiment-shared action signal — 3D wrist translation in the head-camera frame — with interleaved-action-token training that absorbs heterogeneous data (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots).

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Danfei Xu — two root-level paradigm shifts in robot learning: Sim2Real for locomotion, Behavior Cloning for manipulation: Danfei Xu’s “BC-for-manipulation as the paradigm shift” framing is the implicit motivation for the retargeting subfield — extracting demonstrations from human video is a direct response to BC’s data-hunger.
  • [2026-07-20] Scaling Behavior Foundation Model for Humanoid Robots: ScaleBFM (Zeng et al.) treats retargeted human motion as the pretraining substrate for a humanoid BFM rather than as a per-clip control target: 102M frames aggregated across multiple open-source motion datasets and retargeted onto the target humanoid, with PPO-based motion tracking as the proxy task. The load-bearing finding for this concept: reference-motion scaling splits into two regimes — homogeneous (more in-domain data, similar behavior coverage) yields only marginal gains, while heterogeneous (diverse sources expanding behavior coverage) yields substantial OOD gains. Contrasts with ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting‘s physics-in-loop retargeting-quality lever by showing source-diversity is a distinct lever that matters even under a fixed kinematic retargeter; sharpens the “retargeting quality vs retargeting diversity” open question on the page with a quantitative answer favoring diversity for OOD generalization.
  • [2026-07-18] HandUMI — Robot-free bimanual data collection retargetable to any parallel-gripper arm (Brik H. Meza launch tweet): HandUMI (Brik H. Meza et al.) packages the make-collection-tool-embodiment-invariant position as an open-source, Apache-2.0 hardware+software stack: robot-free wearable capture that replays onto four distinct bimanual parallel-gripper arms (Agilex PiPER, OpenArm, TRLC-DK1, I2RT YAM) by swapping the gripper module, with LeRobot v3-compatible output, built-in calibration/QA, and both sim replay and real teleoperation supported. Complements Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training‘s at-scale UMI-based pretraining evidence by making the same collection substrate broadly available to small teams and across multiple embodiments rather than a single vendor rig.
  • [2026-07-18] Towards Predictive, Aligned, and Scalable Robot Learning (Lumo-2): Lumo-2 (Astribot) headlines “Human-to-Robot learning” as one of three flagship capabilities alongside predictive world dynamics and long-horizon manipulation; the launch showcase demonstrates 20+ household tasks on a real humanoid with cross-embodiment learning from human video contributing to the training substrate.
  • [2026-07-17] WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time: WAM-TTT (Feng, Lyu, Wang et al.) proposes a distinct position on the interface-location axis: don’t retarget human motion to robot embodiment at all, and don’t retarget task intent either — instead absorb the raw human video into a frozen WAM’s key–value memory via self-supervised video prediction, and rely on a paired-human-robot meta-training stage with a KV reconstruction objective to align memory content with robot-behavior-relevant features. The human-robot correspondence lives inside the frozen model’s internal state rather than in an explicit action-space reduction (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), 4D kinematic reconstruction (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), or intent abstraction (LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition). At deployment only unlabeled human videos are needed; no robot demos, no annotations, no fine-tuning. Reports consistent wins over in-context human-video conditioning baselines across manipulation and generalization settings.
  • [2026-07-17] Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation: Mixture of Frames Policy (Wang, Song, Bohg et al.) reframes the “which frame to represent motion in” problem — a recurring headache for cross-embodiment retargeting — as an MoE routing decision inside a diffusion policy: denoise the bimanual action chunk in base / left-hand / right-hand / rel-traj frames in parallel and learn per-subtask expert weights. Measures a 15.0-point gap between best- and worst-per-task frame choice and shows learned per-subtask routing beats even the oracle-per-task frame choice (66.8 vs 63.8), suggesting the frame problem is at least partly architectural rather than purely a data/kinematics engineering concern.
  • [2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic robotics stakes out a hardware-first position on the interface-location axis: fix the embodiment-invariant interface at the hand-kinematics level by shipping a wearable exoskeleton (U1) whose rigid linkage mechanically constrains the human hand into the motion space of the robot hand (M1), eliminating retargeting at collection time. This is the same shortcut as Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training‘s UMI-based collection but extended one embodiment upward from two-finger grippers to a fully dexterous 15-DoF anthropomorphic hand; complements the algorithmic positions (physics-in-loop ReActor, reduced-DoF Translation-as-Bridging-Action, task-intent LUCID) by moving the fix into the collection hardware itself. Tactile + joint-encoder + wrist-camera sensing is mirrored 1:1 across M1 and U1 so demonstrator observations match robot observations exactly.
  • [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) proposes an eighth position on the interface-location axis: make the collection tool itself embodiment-invariant — hand-held UMI grippers are the retargeting target at collection time, so the kinematic + contact-mechanics gap between demonstrator and robot is closed at data-gathering, not by post-hoc reconstruction (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos), reduction (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), or intent abstraction (LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition). The recipe’s headline evidence is that this shortcut scales cleanly: 100,000 hours of UMI pre-training across 1,700+ scenarios yields smooth pre-train → post-train real-robot success curves with no saturation, and 75% vs π0.5’s 40% at <10 h/task of downstream robot data. Sits alongside EgoScale’s 20,854 h egocentric-video result as one of the two filed data points supporting scale-as-the-lever on non-teleop substrates.
  • [2026-07-13] HUG: Human Universal Grasping: HUG (NYU / Pinto group) stakes out a seventh position on the interface-design axis: keep the full 51-DoF MANO articulation, normalize hand shape to a canonical β (so θ denotes the same grasp across collectors), and pay for retargeting with a high-quality capture stream — 1M-HUGs, 1M egocentric Aria-Gen-2 RGB-D frames from 6,707 in-the-wild human grasps across 41 buildings + camera-pose back-propagation for view multiplication. Reaches 73% tabletop and 62% in-the-wild real-world grasp success on 30 unseen HUG-Bench objects, beating sim-trained Dex1B (44%) and parallel-jaw CAP (33%), with zero-shot deployment across Ability, WUJI, and Aria/ZED cameras. Sits between Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s reduce-to-3-DoF and EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data‘s scale-with-22-DoF-joints — keeps full articulation but with 750× less pretraining data than EgoScale.
  • [2026-07-11] Cross-Embodiment Robot Manipulation via a Unified Hand Action Space: UHAS (UT Dallas / Casas, Xiang) proposes a sixth position on the interface-location axis: a geometric fingertip-surface interface — a canonical sphere with lateral rotation and radial displacement parameters — shared across Allegro / LEAP / Shadow / MANO Human hands, with a Cascade IK solver mapping the shared sphere back to per-embodiment joint targets. A single RL policy trained in the sphere-deformation space controls all four hands on in-hand cube reorientation and transfers zero-shot to unseen embodiments; real-world deployment on LEAP and Allegro from the same policy. Contrasts most directly with Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (drop rotation) and EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data (keep 22-DoF joints + scale) by reparameterizing rotation into a geometry that is embodiment-invariant by construction.
  • [2026-07-11] LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition: LUCID (CMU; Gupta, Shi, Yuan) proposes a sixth lever: don’t retarget human motion at all — instead retarget task intent (short-horizon 3D object flow + palm-pose reference) and delegate morphology-specific execution to a sim-trained sensorimotor policy. The same intent model drives a LEAP dexterous hand and a parallel-jaw gripper at matching 19/30 success on push-T + cable routing supervised by 1 hr of smartphone video, and closed-loop intent re-querying beats a Veo-3.1 open-loop planner 73% vs 28% on stirring/wiping/binning. Structurally distinct from the motion-retargeting recipes on this page: the interface between human data and robot control is a scene-level prediction (what should happen next) rather than a joint-level or end-effector-level trajectory, which is why one intent module trivially spans two very different embodiments.
  • [2026-07-10] LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control: Human demonstration videos are used as in-context visual task prompts (§2.3.4) to the causal video-action policy, and human-robot co-training brings egocentric human videos into the shared world model (§2.3.5). Complements the reduction/filtering positions (HumanScale filtering; Translation-as-a-Bridging-Action retargeting) with a shared-latent position: the human video is consumed at the same tokenizer layer as robot video, and the latent-action tokenizer’s bottleneck yields action-relevant supervision from human video without pseudo-action labels.
  • [2026-07-09] From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0): LingBot-VLA 2.0 takes the keep-both-channels-at-scale position: 10,000 hours of egocentric human video are mixed directly into the same pretraining stage as 50,000 hours of robot trajectories spanning 20 configurations, rather than as a separate mid-training phase (ACE-Ego-0), a filtered auxiliary channel (HumanScale), or a reduced 3-DoF bridging interface (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots). Provides a corpus-scale reference point ~50% the size of EgoScale’s egocentric-only 20,854 hours but combined with 2.4× more robot data.
  • [2026-07-02] HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation: HABIT (KAIST; Song et al.) sits adjacent to but distinct from this concept: it does not retarget human motion to a robot embodiment (the robot is teleoperated), but it does retarget the coordination interface — every trigger cue in the recorded data must exist in the camera stream, and Supervisor tasks force the robot to infer intent from human gesture rather than shortcutting to workflow knowledge. Complements the motion-retargeting recipes on this page by showing that on the interaction-cue channel, the analogous “drop the unobservable signals” principle likewise produces policies that transfer.
  • [2026-06-30] TrueLabel — Physical AI Dataset Directory: Robotics, Humanoid & Egocentric Data: TrueLabel Physical AI Dataset Directory provides a side-by-side license-and-scale view across the egocentric corpus (Ego4D, EPIC-KITCHENS, HOI4D, DexYCB, xperience-10m, 10Kh-RealOmin-OpenData, UniHand_Preview) and the teleop corpus (AgiBot World, DROID, LeRobot releases) that this concept’s filed papers debate as pretraining substrates.
  • [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale (NVIDIA GEAR) is a sixth lever on the action-channel-design axis: keep the full 22-DoF retargeted Sharpa-hand joint actions plus relative wrist motion, and pay for the noise with raw pretraining scale (20,854 h of egocentric video). Directly contradicts the Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots drop-the-noisy-channels recipe by ablating wrist-only and fingertip-based representations and finding the 22-DoF joint channel wins across all 5 contact-sensitive tasks. Cross-embodiment transfer to a Unitree G1 7-DoF tri-finger hand (+30%+ absolute) is the first filed evidence that a high-DoF retargeting target functions as a reusable motor prior across substantially different hand kinematics.
  • [2026-06-30] datasets.bot — Curated catalog of robotics and embodied-AI training datasets: datasets.bot — third-party browseable catalog of robotics/embodied-AI training datasets (LeRobot, AgiBot World, HIW-500, GR00T X-Embodiment Sim, FTP-1, ABC-130k, DreamDojo, …) with per-dataset modality tags (rgb/depth/ee_pose/proprioception/tactile/force_torque/ego), license, and popularity proxies; useful as a scoping aid for “what human/robot data exists under what license” before picking a retargeting substrate.
  • [2026-06-30] ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: ACE-Ego-0 absorbs the human-robot embodiment gap inside the training objective rather than at the representation or physics level: keeps the full action space, unifies streams via camera-space actions + morphology conditioning + time-aligned chunking, and pays for the resulting pseudo-action label noise with a reliability-aware loss and human auxiliary loss — directly contrasting Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s drop-the-noisy-channels recipe on the same problem.
  • [2026-06-30] Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots: Fourth distinct lever for the cluster — action-representation design rather than physics/perception/RL-constraint. Proposes a 3-DoF bridging action (relative wrist translation in head-camera frame, 30-step chunk) shared by humans and robots, plus interleaved-action-token + attention-masking training that absorbs three data sources with different action coverage. Outperforms 6-DoF baseline from scratch on the ByteMini bi-manual suite (44.58 vs 34.67% task progress, 22.50 vs 12.50% success rate) and shows that adding large-scale human-action pretraining strongly outperforms pick-and-place-only training.
  • [2026-06-19] ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control: ConTrack (UCSD; Xiaolong Wang group, ECCV 2026) — dual-variable RL update reallocates authority between object-tracking constraint and hand-style fidelity online, eliminating per-clip reward tuning. Closed-loop-control half of the same dexterous-hand pipeline that Do as I Do attacks at the kinematic-reconstruction level.
  • [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO (Malik group; Berkeley + Meta) — kinematic-only pipeline: 4D hand-object reconstruction from monocular RGB + retargeting to multi-fingered robot hand, end-to-end. Targets in-the-wild data the lab-captured corpora can’t reach; positions itself as Malik group’s positive answer to its own sensorimotor counter-position.
  • [2026-05-24] ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting: ReActor (Disney Research Switzerland) — bilevel optimization with physics in the outer retargeting loop and an RL tracking policy in the inner loop. Targets full-body humanoids + a quadruped (Disney animatronics), validated on hardware. The framework derives an approximate gradient on the outer loss so the joint optimization doesn’t require unrolling through the RL inner loop.

Open questions

Section titled “Open questions”
  • At what level should the embodiment-invariant interface live — joint pose with adaptive correspondences (ReActor), 4D hand-object reconstruction with kinematic retargeting (Do as I Do), object trajectory as a constraint (ConTrack), or a deliberately reduced action space that drops the noisy channels (Translation-as-Bridging-Action)?
  • Do the physics-in-loop and action-representation-reduction recipes compose — i.e. does dropping wrist rotation on the human side help even when a downstream physics-simulator-grounded policy is solving the dynamics? No filed paper combines them.
  • Does the full-body humanoid retargeting problem (ReActor) want a different interface than the dexterous-hand retargeting problem (ConTrack / Do as I Do), or is there a unified parameterization?
  • How well does the bridging-action / interleaved-token recipe generalize to full bi-manual dexterous tasks (not just parallel-gripper bi-manual ByteMini tasks where rotation can be cleanly discarded)?
  • ReActor and Do as I Do both bypass the teleop vs human-video substrate debate by treating human motion as a data source and retargeting as the recipe — does this make HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining‘s matched-volume head-to-head measurement moot, or does retargeting quality become the new lever?

Papers

Section titled “Papers”

2026-06

Section titled “2026-06”

2026-05

Section titled “2026-05”