AdaDex: Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

TL;DR

AdaDex is a goal-conditioned RL framework that turns a single hand-object trajectory (from mocap or in-the-wild RGB(D) video) into an adaptable dexterous skill. The core recipe, Hybrid Dexterous Mimicking, trains one shared policy on two populations of massively parallel environments simultaneously: motion-tracking environments with a dense tracking objective (to instill a human-like prior) and generalization environments with noised initial states/goals and a sparse object-success reward (to force the skill to work far off the reference). The resulting policies cover functional grasping, in-hand reorientation with finger gaiting, dynamic throw-and-catch, pouring, hammering, and sweeping; they transfer across simulators and to the real world, and a coding agent (Codex w/ GPT-6 Astra) can author new wrist+object waypoint sequences for downstream tasks.

Key claims

A shared policy trained jointly across two parallel environment populations — motion tracking (dense) and generalization (sparse, noised) — outperforms either population alone on both the trained motion and on randomly generated goals [§Hybrid Dexterous Mimicking; Tables i and iii].
On GRAB with one motion-per-object training, AdaDex reaches 96.74% progress on seen motions and 80.02% on similar-but-unseen motions, versus 35.20% / 7.28% for ManipTrans and 9.72% / 13.68% for SimToolReal [Table setting (i), setting (ii)].
Removing the generalization environments (w/o GenEnv) scores higher on seen motions (97.47%) but collapses on unseen motions (31.75%) and on random goals (14.53% progress / 0.52% SR), isolating GenEnv as the lever responsible for off-reference adaptation [Table ii, Table iii].
A single policy trained in Isaac Gym transfers zero-shot, closed-loop to MuJoCo on all 36 GRAB benchmark motions without retraining [§Transfer].
The same recipe retargets to multiple dexterous embodiments (ArtiMANO, XHand, Sharpa, Wuji) without method changes [§Hands].
On real hardware with an XHand mounted on an xArm, generated joint targets are passed open- and closed-loop at 1× speed (30 Hz) [§Real].
The HOI reconstruction pipeline accompanying AdaDex reports better reconstruction quality and shorter runtime than Do-As-I-Do and EgoInfinity on the same casual RGB-D video, enabling a few seconds of phone video → simulation-ready reference [§HOI reconstruction].

Method

AdaDex treats motion tracking and goal reaching as the same goal-conditioned problem — bring object + wrist to a target pose, differing only in where the goal comes from and how progress is rewarded. The policy is trained with PPO on massively parallel environments split into two populations: (a) motion-tracking environments, where the goal is the reference trajectory’s next step and the reward is a dense tracking signal that encodes the human prior; (b) generalization environments, where the initial state and goal are noised and the reward is sparse object-success only. Weight updates come from both populations, so the shared policy must satisfy both regimes. Reference trajectories can be sourced from mocap (GRAB) or from the paper’s own HOI reconstruction pipeline over casual RGB-D phone video. At deployment, downstream tasks are expressed as new sequences of wrist + object pose waypoints, which the coding agent (Codex with GPT-6 Astra) emits and the trained skill executes.

Results

Headline numbers on 12 GRAB sequences (36 benchmark motions across three settings):

Seen motions: progress 96.74% (vs 35.20% ManipTrans, 9.72% SimToolReal); obj. pos. err 0.72 cm; obj. rot. err 4.16°; wrist pos. err 2.82 cm.
Similar but unseen motions: progress 80.02% (vs 7.28% ManipTrans, 13.68% SimToolReal); obj. pos. err 6.07 cm; obj. rot. err 15.83°.
Random goals (far off reference): progress 51.80%, SR 27.08% (vs 44.60% / 31.71% SimToolReal, 1.72% / 0.00% ManipTrans).

Ablations isolate the two populations: w/o TrackEnv drops to 8.45% / 9.62% / 4.34% progress across the three settings; w/o GenEnv is near-perfect on seen (97.47%) but collapses to 31.75% / 14.53% off-reference, showing both legs are load-bearing.

Transfer: policies trained in Isaac Gym execute zero-shot and closed-loop in MuJoCo on all 36 benchmark motions. The same method with no changes trains on four different robot hands (ArtiMANO, XHand, Sharpa, Wuji). Real-world rollouts on an XHand+xArm run open-loop and closed-loop at 30 Hz at 1× speed.

Why it’s interesting

AdaDex sits at a specific point in the Human-to-Robot Retargeting interface-location debate: keep the full dexterous channel + reference trajectory, but mix motion-tracking and goal-reaching RL inside one shared policy. This is a different lever from REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation REGRIND (retargeting as a reference-only prior that RL then departs from) and from EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization EgoCPT (per-segment contact-aware kinematic optimization rather than policy learning). The ablation isolating w/o GenEnv as the off-reference lever is the sharpest filed evidence that generalization environments are a distinct reward-side intervention — not a data augmentation — complementing ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control ConTrack’s dual-variable object-tracking-as-constraint recipe. Also notable as a datapoint that a single human motion clip plus a modern HOI reconstruction pipeline is enough per skill, in contrast to the scale-side bets in EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale and The Will to Act — Natura-1 Natura-1.