Skip to content
Bud's Wiki

C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video

TL;DR

Section titled “TL;DR”

C2Dex is a video-to-dexterous-manipulation framework built around a shared interaction representation: object-side contacts recovered by aggregating noisy per-frame hand-object interaction observations in the canonical object space. These stable contacts do double duty — first as trajectory-level constraints that guide monocular HOI reconstruction toward temporally coherent and physically plausible human trajectories, and then as explicit transfer targets for the dexterous robot hand, where Laplacian interaction optimization preserves local hand-object geometry across embodiments and residual RL refines the trajectory in simulation. Reports end-to-end trajectory success of 57.78% on DexYCB and 26.67% on TACO, roughly 3× and 2.7× the strongest baselines under matched evaluation, with real-robot replay demonstrating physical feasibility on contact-rich tasks.

Key claims

Section titled “Key claims”
  • Per-frame monocular HOI reconstructions produce temporally unstable contacts and physically implausible interactions; aggregating them into a stable set of object-side contacts in canonical object space yields a shared representation usable for both reconstruction cleanup and downstream retargeting [§Abstract].
  • Using the recovered stable contacts as trajectory-level constraints during reconstruction produces temporally coherent, physically plausible human HOI trajectories that per-frame methods do not [§Abstract].
  • Laplacian interaction optimization on the same stable-contact target preserves local hand-object geometry across hand embodiments — i.e. contact-topology transfer, not joint-space kinematic transfer [§Abstract].
  • Residual reinforcement learning in simulation refines the retargeted trajectory to close the remaining physics gap before real-robot deployment [§Abstract].
  • On DexYCB, C2Dex reaches 57.78% end-to-end trajectory success versus 17.78% for the strongest baseline; on TACO, 26.67% versus 10.00%, under identical evaluation criteria [§Abstract].
  • Real-robot replay on diverse contact-rich manipulation tasks confirms the retargeted trajectories are physically executable, not just simulator artefacts [§Abstract].

Method

Section titled “Method”

C2Dex organizes a monocular-video-to-dexterous-manipulation pipeline around a single canonical-object-space contact representation. Noisy per-frame HOI reconstructions are aggregated to recover a stable object-side contact set. That contact set is then re-used in two places: (1) as constraints on the human HOI reconstruction pass to fix temporal jitter and interpenetration; (2) as explicit targets for a Laplacian interaction optimization that maps the human hand’s local contact geometry onto the dexterous robot hand’s mesh, preserving where and how contact occurs rather than raw joint angles. A residual RL policy in simulation cleans up the remaining dynamics gap before deployment.

Results

Section titled “Results”
  • DexYCB: 57.78% end-to-end trajectory success vs 17.78% strongest baseline [§Abstract].
  • TACO: 26.67% vs 10.00% strongest baseline [§Abstract].
  • Real robot: physical feasibility confirmed across diverse contact-rich manipulation tasks via replay [§Abstract].

Why it’s interesting

Section titled “Why it’s interesting”

C2Dex stakes out a distinct position on the interface-location axis tracked by Human-to-Robot Retargeting: the shared embodiment-invariant interface is object-side contact topology in the object’s canonical frame, not joints (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos Do as I Do), not a reduced 3-DoF wrist bridge (Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots), not object-tracking as an RL constraint (ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control), and not task intent (LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition). This puts it closest to CHORD (CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation), which also uses an object-centric contact representation (contact wrenches on the object) — the difference is that C2Dex operates at the geometric level (contact locations + Laplacian preservation of local hand-object geometry) rather than the force/wrench level, and closes the loop with residual RL rather than a pure sim policy. The 3× improvement on DexYCB over the strongest baseline is the strongest evidence yet that fixing the contact-topology channel is more informative than fixing the joint-angle channel for contact-rich dexterous transfer.

See also

Section titled “See also”