Skip to content
Bud's Wiki

REGRIND: A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation

TL;DR

Section titled “TL;DR”

REGRIND is a minimalist retargeting-guided RL pipeline that learns contact-rich dexterous manipulation policies from a single human demonstration. The recipe: retarget the human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, then train a residual RL policy in simulation to track object-centric keypoints along that reference, then transfer zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands (LEAP and WUJI) across contact-rich tool-use tasks including operating scissors and turning a screwdriver, and the authors surface a systematic hardware analysis of the factors governing sim-to-real transfer in dexterous manipulation.

Key claims

Section titled “Key claims”
  • The humanoid whole-body-control recipe — retarget human motion to robot kinematic references, then RL-track them — extends to dexterous manipulation when the retargeting target explicitly preserves hand-object spatial and contact relationships rather than joint pose alone [§Abstract, §Method Overview].
  • A residual RL policy trained in simulation to track object-centric keypoints along the retargeted reference (not joint pose along the reference) produces fluid contact-rich policies that transfer zero-shot to hardware [§Method Overview].
  • The same recipe transfers across two multi-fingered hands (LEAP and WUJI) on the same tasks (screwdriver, scissors) with no per-hand algorithmic changes [§A General Recipe for Different Tasks and Robots].
  • Data augmentation during RL training yields policies robust to ±5 cm position perturbations and ±30° orientation perturbations of the initial configuration [§Generalizing to Different Initial Configurations].
  • Careful system identification at transfer time — not policy-side robustification alone — is a load-bearing part of the sim-to-real story for contact-rich manipulation [§Abstract].

Method

Section titled “Method”

REGRIND is presented as a three-stage pipeline. First, a retargeting stage converts a single human demonstration (hand-object trajectory) into a robot reference that preserves the human’s hand-object contact and spatial structure rather than trying to preserve raw joint angles — this is the “hand-object-relationship-preserving” retargeting the abstract emphasizes. Second, a residual RL policy is trained in simulation whose objective is to track object-centric keypoints along the retargeted reference; the residual formulation lets the policy correct the reference where contact dynamics demand it without abandoning the demonstration’s structure. Third, zero-shot transfer to hardware relies on careful system identification (the abstract flags this as a key factor). Data augmentation over initial configurations diversifies the RL training distribution enough that the deployed policy tolerates 5 cm / 30° perturbations. The tool-use tasks demonstrated — scissors and screwdriver — are chosen because both require regulating contact mode and force (not just making contact), which is precisely the regime where kinematic-only retargeting-then-RL recipes typically fail.

Results

Section titled “Results”

The project page is qualitative-first: policy rollouts are shown in simulation and on hardware at 1× speed for the LEAP hand and the WUJI hand, on scissors and screwdriver tasks, with additional 2× videos of 10 different initial configurations per task demonstrating the ±5 cm / ±30° robustness envelope. The claim is that fluid, human-like behavior emerges on contact-rich tool use from a single human demonstration — this is the headline efficiency claim over teleop-heavy or multi-demonstration recipes. The abstract also promises a systematic hardware analysis identifying the factors that govern sim-to-real transfer for dexterous manipulation, but the specifics of that ablation (which factors, magnitudes) sit in the paper PDF rather than the project page.

Why it’s interesting

Section titled “Why it’s interesting”

REGRIND is the seventh distinct answer to the Human-to-Robot Retargeting concept’s central question of “where should the embodiment-invariant interface live?” — and it stakes out a position that is orthogonal to the ones already filed. Where ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting puts the physics simulator in the outer retargeting loop (bilevel co-optimization of reference and tracking policy), REGRIND keeps the retargeting stage kinematic + relationship-preserving and puts the physics-side effort into a residual RL policy that tracks object-centric keypoints rather than robot-joint pose — a subtle but consequential redirect from “make the robot pose match the human” to “make the object trajectory match the human”. This is closest in spirit to ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control‘s reframing of object-tracking as a hard RL constraint, but where ConTrack uses a dual-variable update over the object-vs-hand trade-off, REGRIND simply defines the reward on the object and lets the residual policy figure out the hand. Complements Do as I Do: Dexterous Manipulation Data from Everyday Human Videos‘s in-the-wild-video-to-dexterous-trajectory pipeline: DO AS I DO produces the kinematic reference, REGRIND is the recipe that turns a single such reference into a hardware-executable policy.

See also

Section titled “See also”