CARD: Contact-Anchored Retargeting for Dexterous Manipulation

TL;DR

CARD (Contact-Anchored Retargeting for Dexterous Manipulation) is a simulator- and object-free framework that retargets human demonstrations onto robot hands by explicitly preserving contact force targets rather than only kinematics. A shared hand–contact representation unifies three human sources (smartphone video, mocap, tactile gloves) with one contact anchor per hand region; CARD generates both kinematics and explicit contact force targets, and a tactile tracking controller maintains those targets during open-loop replay on hardware. For monocular video (no instrumentation), Vid-HACO is a dedicated video-to-contact model that sets a new state of the art. Across the paper’s video / glove / mocap data plus the GRAB dataset, CARD maintains contact on 43–78% of contact frames vs <10% for kinematic retargeting and 22.5% for object-aware retargeting, and dropping CARD’s contact objectives into physics-based retargeting lifts strict GRAB task success from 12% to 32% [§Abstract].

Key claims

Kinematic-only retargeting ignores real-world contact dynamics, which is where downstream task failures concentrate; preserving explicit contact-force targets per hand region is a sharper interface than matching joint angles or end-effector pose [§Abstract].
A single hand–contact representation — one contact anchor per hand region — admits human demonstrations from video, mocap, and tactile gloves without a per-source retargeter [§Abstract].
Contact-preservation measured on contact frames: CARD 43–78% vs kinematic baselines <10% vs object-aware retargeting 22.5%, averaged across video / glove / mocap data and the GRAB dataset [§Abstract].
Integrating CARD’s contact-force objectives into physics-based retargeting lifts strict task success on GRAB from 12% → 32% [§Abstract].
Vid-HACO, the paper’s video-to-contact model, beats the image-based HACO baseline on held-out HRDexDB: vertex F1 63.3 → 67.2, AP 54.3 → 72.2 (per author thread; HRDexDB).
A tactile tracking controller — integral-dominant admittance control driven by the tactile sensor — is required to realize the demonstrated forces during hardware replay. Position-only replay vs replay with the tactile tracker: 🔨 hammer lift 1/10 → 8/10 (per author thread, position-only vs with-tracker).

Method

CARD factors the human-to-robot retargeting problem into three stages, all anchored on contact.

Extract contact. From any source that provides hand pose + contact (smartphone video, tactile gloves, motion capture), map into a unified hand-contact representation with one contact anchor per hand region. For monocular video that lacks instrumented contact, Vid-HACO predicts per-vertex hand contact directly from the video stream and feeds the shared representation.

Preserve contact. Given the unified representation, align the corresponding robot-pad contacts with the human contact-force targets — matching both the position (which region of the hand touches) and direction (how the force vector acts), rather than only aligning joint angles. This is the retargeting step proper, and it produces not only kinematics but explicit per-anchor force targets.

Track contact. On hardware, an integral-dominant tactile admittance controller tracks the desired contact-force targets using tactile sensor feedback as the error signal. Because the retargeter emits force targets explicitly, the controller has a well-defined set-point even for open-loop replay of a human demonstration.

The paper reports evaluation across video, glove, mocap, and the GRAB dataset, with a dedicated per-stage ablation of the tactile tracker.

Results

Contact preservation (fraction of contact frames where contact is maintained), across video + glove + mocap + GRAB: CARD 43–78% vs kinematic baselines <10% vs object-aware retargeting 22.5% [§Abstract].
Physics-based retargeter with CARD’s contact objectives bolted on: strict task success 12% → 32% on GRAB [§Abstract].
Vid-HACO vs image-based HACO on held-out HRDexDB: vertex F1 63.3 → 67.2, AP 54.3 → 72.2 (author thread).
Tactile tracker ablation (position-only → with tracker) on hardware: 🔨 hammer lift 1/10 → 8/10 (author thread).
Project headline: open-loop rollouts of card-flipping and other force-sensitive dexterous tasks on real hardware, no simulator in the loop (author thread).

Why it’s interesting

CARD stakes a distinct position on Human-to-Robot Retargeting‘s interface-location axis: the shared interface is per-region contact-force targets, and the retargeting step is designed to preserve those force targets explicitly rather than minimize kinematic error. It sits between CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation CHORD (object-centric wrench-based interface, embedded as a policy reward) and C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video C2Dex (object-side contact topology aggregated into a trajectory-level constraint) by keeping the force channel first-class and letting it survive the sim-free open-loop replay regime via the tactile tracker. The multi-source unification — one representation for video + glove + mocap — contrasts with the per-pipeline specialization in Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (video-only kinematic reconstruction) and EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization EgoCPT (per-segment contact optimization from monocular RGB). On Tactile sensing for manipulation, CARD is a counter-position to the “touch enters at the policy input layer” lineage (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction T-Rex per-finger tactile VQ-VAE, FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT per-layer force modulation, ReForce — force-aware retargeting via closed-loop residual force tracker ReForce residual force tracker atop a learned reference): CARD moves tactile integration into a controller that tracks explicit force set-points from the retargeter, making the demonstration-to-execution pipeline force-aware end-to-end without a learned policy in between.