ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

TL;DR

ConTrack (UCSD; Yutong Liang, Quanquan Peng, Ri-Zhao Qiu, Xiaolong Wang; ECCV 2026) is a reinforcement-learning framework that turns long-horizon, contact-rich human hand-object demonstrations into executable dexterous robot motions by treating object tracking as the constraint rather than as one of many reward terms. Instead of forcing the robot hand to copy the human hand’s joint trajectory (the wrong target across the kinematic gap), ConTrack tracks the object motion and spends the remaining control authority on style/contact fidelity, with a dual-variable update that re-allocates the trade-off online when object tracking slips. A second contribution is an adaptive mid-trajectory reset library that reuses policy-reachable simulator states near recent failure boundaries, stabilizing long-horizon learning. Validated in simulation (GRAB, ARCTIC, DexterHand) and on real XHand and Sharpa Wave robot hands, with one policy per clip trained from the same recipe.

Key claims

The kinematic gap between human and robot hands makes joint-pose copying the wrong tracking target; what the demonstration actually specifies is the object trajectory, and treating object tracking as a hard constraint rather than a reward term avoids per-sequence reward tuning [§Abstract, project page Overview].
A dual-variable update reallocates control authority online between object tracking (the constraint) and motion/style fidelity (the secondary objective), eliminating the manual per-clip reward-weight tuning that prior reference-tracking methods require [§Abstract, project page Method].
An adaptive mid-trajectory reset library that reuses policy-reachable simulator states near the current failure boundary stabilizes long-horizon, contact-rich learning, and the failure boundary is moved earlier as the policy improves [§Abstract, project page Method].
Reference contact events and object-local contact points are used as shaping signals to encourage natural contact patterns, since many robot trajectories support the same object motion but only some have realistic contact [project page Method].
ConTrack reports improvements in success rate and object-pose accuracy over prior arts on simulation tracking benchmarks (GRAB bimanual, ARCTIC articulated tool use, DexterHand in-hand rotation) and demonstrates real-robot transfer to XHand and Sharpa Wave platforms [§Abstract, project page Results].

Method

ConTrack takes a reference hand-object trajectory (from human mocap or video reconstruction) and trains a per-clip RL policy with object tracking as a hard constraint. The optimization keeps the object close to its reference trajectory while letting the robot hand deviate from the human hand pose; a dual-variable controller monitors the ratio of current object-tracking return to the best return reached on the clip, shifting capacity toward the object term when tracking slips and back toward style/contact terms after recovery. The adaptive reset library samples mid-trajectory simulator states near the most recent failure boundary, so the policy practices reachable contact phases more often than easy initial states, and the failure boundary is annealed earlier as success improves. Contact shaping is driven by reference contact events and object-local contact points rather than world-space contact poses, so the same shaping signal generalizes across hand morphologies. The same recipe is run end-to-end on three datasets covering bimanual manipulation (GRAB), articulated tool use (ARCTIC), and in-hand object rotation (DexterHand), with policies deployed to two real robot hands (XHand, Sharpa Wave).

Results

The project page presents qualitative simulation and real-robot videos across GRAB (bimanual object manipulation), ARCTIC (articulated tool use), and DexterHand (in-hand rotation) for both XHand and Sharpa Wave robot hands [project page Results]. The abstract states ConTrack improves success rate and object-pose accuracy significantly over prior arts while preserving joint and contact fidelity, but specific benchmark numbers and ablations are in the body, which was not retrieved at filing time [§Abstract]. The framing positions ConTrack against (a) hand-crafted reward-tuning baselines that require per-sequence weights and (b) prior reference-tracking RL that breaks under limited interaction budgets [§Abstract]; head-to-head numbers vs. specific named baselines would need a refresh once the PDF is parsed.

Why it’s interesting

ConTrack is the physics-in-the-loop sibling to Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (Do as I Do, Malik group), surfaced in the same Slack thread. Where Do as I Do does kinematic 4D hand-object reconstruction followed by retargeting to a robot hand, ConTrack accepts a reference (possibly produced by Do-as-I-Do-style reconstruction) and trains an RL policy that physically executes it under an object-tracking constraint. The two approaches are complementary along the same data pipeline: Do as I Do is the perception + retargeting stage that emits a hand-object reference; ConTrack is the closed-loop control stage that makes that reference dynamically executable on a real dexterous hand. Both are direct answers to Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s critique that VLA work skips the sensorimotor / contact level — ConTrack’s claim is even sharper: not just contact-grounded data but contact-grounded training-time interaction with the simulator. The bilevel-with-physics framing also rhymes with ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting (Disney ReActor) at full-body morphology — ReActor adapts retargeting parameters in an outer loop while the inner RL policy tracks; ConTrack instead holds the reference fixed and lets a dual variable redistribute reward weight online, which is the cheaper formulation when reference quality is already good but reward tuning is the bottleneck. For the VLA Models cluster, ConTrack is a sensorimotor-level lever orthogonal to the policy-architecture debate (π*0.6: a VLA That Learns From Experience (RECAP) vs Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models) and to the data-collection debate (ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation vs Do as I Do: Dexterous Manipulation Data from Everyday Human Videos): it shows that the per-clip RL post-processing step itself can be made tuning-free, which is the precondition for scaling the human-video-to-robot pipeline beyond a handful of demos.