HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos

TL;DR

HDMI (HumanoiD iMitation for Interaction) is a general framework for learning whole-body humanoid-object interaction skills directly from monocular RGB video of humans interacting with objects — no task-specific reward engineering, no VLM/model-based planners. The pipeline has three stages: (i) extract and retarget human and object trajectories from unconstrained video using off-the-shelf pose estimation (GVHMR + LocoMujoco), (ii) train an end-to-end RL control policy to co-track both robot and object states with three key design choices — a unified object representation in the robot’s root frame, a residual action space centered on the reference pose, and a unified contact-promoting interaction reward, and (iii) zero-shot deploy on real hardware. On a Unitree G1 humanoid, HDMI achieves 67 consecutive bi-directional door open + traversals (34 minutes of continuous operation), 6 distinct loco-manipulation tasks in the real world, and 14 tasks in simulation, spanning kneeling, box transport, staircase climbing, sitting, and bowing. The Sep 8 2026 update announces an open-sourced rewrite of the codebase with an mjlab backend and multi-motion support per object.

Key claims

HDMI is the first general framework that enables learning autonomous whole-body humanoid-object interaction skills directly from RGB video, spanning different body parts (hand, foot) and different object types (articulated / rigid, fixed-base / floating-base) with no per-task architectural changes [§I, §III].
Unified object representation — expressing object pose in the robot’s local root frame plus reference contact points transformed into the same frame — is spatially invariant and lets a single policy architecture handle heterogeneous object geometries and types [§III-B1].
Residual action space grounds initial exploration around the reference pose rather than the default standing pose: the policy outputs a corrective offset added to the reference joint targets, which is load-bearing for learning motions far from standing (kneeling, sitting on stairs) — without it, the policy converges to a suboptimal “bend-at-waist” strategy on suitcase pickup instead of the intended kneeling motion [§III-B2, §V-B, Fig. 9].
Unified contact-promoting reward combines a position term (align end-effector with target contact point) and a bounded force term (encourage stable contact force capped at a threshold), gated by the reference contact signal. This handles imperfect retargeted references — the reward lets the policy deviate from a flawed reference to establish a proper grasp; without it, the policy rigidly follows the flawed reference and fails to interact [§III-B3, §V-A, Fig. 7].
Interaction reward is also load-bearing for precise contact locations: without it, the L-shaped end-effector for pushing a box is placed on the vertical face rather than the edge, producing unstable contact [§V-A, Fig. 7c].
Termination conditions (root pose error 0.5 m / 1.2 rad, body local pose error 0.5 m / 1.2 rad, object pose error 0.5 m / 1.2 rad, lost contact 0.2 m + 1.0 N) are min-step-gated to 25 steps to avoid killing episodes during transient contact establishment [Table II].
Sim-to-real headline numbers: 67 consecutive door open + traversals (34 min continuous, with initial positional offset randomized 10–30 cm per round); 7 consecutive suitcase manipulation runs with smooth kneel → lift → walk transitions; 2 full bread-box carrying trials; foam mats relocation with sidestepping; 3 full continuous executions of the multi-stage “Truman’s Bow” task (staircase climb → bow → sit → wave → jump off → walk back); robust under terrain changes (removing the wooden floorboard) [§IV].
Simulation ablations (4096 parallel envs per condition): the full method with both residual action and body tracking termination consistently achieves the lowest joint and body tracking errors across 8 tasks and converges much faster than the “actions relative to default standing pose” variant [§V-B, Fig. 8].
All policies trained in IsaacSim with the same hyperparameters across all tasks, zero-shot deployed on real hardware with mocap markers on pelvis + objects for policy observations [§IV].
Named limitations: (i) system depends on ground-truth motion capture data (object poses) at deployment — a next step is on-board sensing (RGB/depth); (ii) one specialist policy per skill — no unified generalist model yet [§VI].

Method

The three-stage pipeline is deliberately simple. Stage 1 (data) takes an RGB video of a human interacting with an object and produces a structured reference dataset. GVHMR provides world-grounded SMPL pose estimation; LocoMujoco retargets human poses onto the humanoid morphology. Post-processing annotates object trajectory (position, orientation, joint state for articulated objects like doors or folding chairs) and a binary contact signal per frame. Each frame of the reference is q_t^R = (p^R, R^R, q^R) (root position, root orientation, joint positions) plus q_t^O = (p^O, R^O, q^O_joint) for articulated objects, and a contact signal c_t.

Stage 2 (policy training) frames interactive skill acquisition as a robot–object co-tracking problem trained via PPO in IsaacSim. Three DeepMimic-style ingredients are inherited: reference state initialization (episode starts from a random frame with small perturbations), a phase-variable observation φ ∈ [0,1], and tracking-error-based termination. On top of these, the three HDMI-specific components layer in: (i) the unified object observation in the robot’s root frame plus reference contact points also in the root frame; (ii) the residual action space q_target = q_ref + Δq_π so exploration is centered on the reference; (iii) the interaction reward r_interact averaging position + bounded-force terms over all active end-effectors, gated by c_t. Domain randomization on robot + object inertial and friction properties improves sim-to-real.

Stage 3 (deployment) loads the trained policy onto a Unitree G1 with mocap markers on pelvis + object; object poses are transformed into the robot’s root frame at runtime to serve as policy observations.

The Sep 8 2026 open-source rewrite (announced in the pointer tweet) swaps IsaacSim for mjlab as the physics backend and adds single-object multi-motion support — the same object can now carry multiple reference motion clips, sharpening the “one policy per skill” limitation the paper flagged but does not yet address in the released version.

Results

Door Open + Traversal: 67 consecutive runs (34 min) before failure; 10–30 cm initial positional offset per round; robust to terrain change (wooden floorboard removed) [§IV-A, Fig. 4a].
Box Loco-Manipulation: Suitcase manipulation 7 consecutive successful runs (kneel → lift → walk with load); Bread box carrying 2 full trials (180° turn is the failure mode); Foam mats relocation succeeds (walk → grasp → sidestep → place) [§IV-B].
Truman’s Bow (multi-stage: staircase climb → bow → sit → wave → jump off → walk back): 3 continuous full executions on real hardware, with light human back support during stair climbing for safety [§IV-C].
Simulation task coverage: 14 distinct simulation tasks; 4096 parallel envs per ablation; interaction-reward removal does not significantly impact final performance on the majority of tasks but is essential specifically for (a) tasks with imperfect retargeted references (suitcase grasp) and (b) tasks requiring precise contact placement (pushing box with L-shaped end-effector) [§V-A].
Residual action ablation: full method (residual + body-tracking termination) consistently lowest joint + body tracking error across 8 tasks; residual grounds exploration around reference and enables kneeling motion that the “actions relative to default” baseline cannot learn (baseline collapses to bending-at-waist) [§V-B, Fig. 8, Fig. 9].
Real-world validation robustness: same set of hyperparameters used across all trained policies; no per-task hyperparameter sweep; zero-shot sim-to-real works across contact-rich, dynamic, and long-horizon regimes [§IV Experiment Setup].

Why it’s interesting

HDMI stakes out a joint robot-and-object motion-tracking interface for humanoid loco-manipulation from human video — sibling to Scaling Behavior Foundation Model for Humanoid Robots (ScaleBFM) at the methodology level (both are PPO motion-tracking + retargeted human motion + IsaacSim) but distinct in target: ScaleBFM studies scaling laws on locomotion + expressive motion, whereas HDMI’s load-bearing scope is contact-rich object interaction — the exact case where kinematic-only retargeting breaks and where ScaleBFM’s “reference motion is the objective” formulation would need a contact reward augmentation to work. And unlike ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting (ReActor), which puts physics inside an outer retargeting loop to jointly adjust retargeting parameters and inner RL policy, HDMI takes the opposite bet: assume off-the-shelf kinematic retargeting will produce imperfect references and design an inner interaction reward that lets the policy deviate from the flawed reference. That’s a testable methodological split — ReActor makes retargeting the tunable object; HDMI makes the RL reward the tunable object. The 67-door benchmark is also the sharpest published contact-consistency claim on this concept page, and the paper’s unified interaction-reward-across-object-types + residual-action-space combination is the closest published analog to a “single recipe that spans doors, boxes, staircases, and bowing” that the ScaleBFM eight-mode control interface aspires to reach on the locomotion side. Related to Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots (Translation-as-Bridging-Action) as an opposite-corner design choice on the retargeting interface — HDMI keeps the full joint-space retargeted reference and closes the gap in the reward + action-space; Translation drops most of the reference channel dimensions and closes the gap in the action representation.