DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

TL;DR

DONIC is the first generalist bimanual dexterous tracker — a single policy that controls both hands and generalizes across operation categories and heterogeneous human-motion sources without per-clip fine-tuning. It combines PPO with behavior cloning in a feedback loop that recycles verified successful executions back into the demonstration pool, an object-centered per-hand reward that supervises each hand according to the reference contact role, and Clip-Centered Advantages that remove reference-dependent offsets in the PPO update. Trained jointly on BiHOI (1,575 sequences, 229 objects, 32-camera multi-view captures retargeted to robot hands) plus mocap sources, one checkpoint reaches 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects — +77.36 and +45.73 points over bimanual-adapted DexTrack and Dexplore baselines [Abstract].

Key claims

A single policy, trained once across heterogeneous operation categories and human-motion sources (BiHOI + reconstructed mocap), achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects without fine-tuning, surpassing bimanual-adapted DexTrack and Dexplore by 77.36 and 45.73 percentage points [Abstract, Table 2].
Object tracking is kept active throughout the motion while each hand is supervised according to its reference’s demanded contact role — an object-centered per-hand reward — so one hand’s progress is not masked by the other during bimanual coordination [§Method, project-page Method panel].
A PPO+BC feedback loop where successful rollouts are replay-verified and recycled into a growing demonstration library that supplies behavior-cloning targets in later updates is the training recipe behind the generalist checkpoint [§Method, project-page Method panel].
Clip-Centered Advantages subtract each clip’s mean estimated advantage before the PPO update, removing reference-dependent offsets (different clips have different achievable reward baselines) while preserving within-trajectory differences — a normalization specifically shaped by multi-reference joint training [§Method, project-page Method panel].
BiHOI is a 1,575-sequence multi-view bimanual HOI dataset spanning 229 objects, captured with 32 calibrated 1920×1232 RGB cameras at 30 fps, with reconstructed hand+object motion retargeted to robot hands as the policy’s reference stream [§Dataset, project-page BiHOI panel].
Metrics follow AdaDexTrack extended to two hands: hand errors use the worse hand and a clip only counts as a success when both hands meet thresholds — a stricter aggregation than per-hand averaging that is load-bearing for the reported margins [Table 2 caption].

Method

DONIC trains one policy jointly across BiHOI’s multi-view captures and public mocap datasets, retargeting human hand+object motion into robot references. The architecture is a single network controlling both hands; the reference stream dictates when each hand engages the object. Three design choices distinguish it from prior single-hand trackers (DexTrack, Dexplore):

Object-centered per-hand reward. Object-tracking reward stays active for both hands throughout the clip, but each hand is additionally supervised against its reference-dictated contact role. This prevents the classic bimanual failure mode where one hand’s progress reward obscures the other hand’s drift.
Clip-Centered Advantages (CCA). Each reference clip has a different achievable reward baseline (different objects, grasps, horizons), so joint training across clips makes raw PPO advantages carry a reference-dependent offset. CCA subtracts each clip’s mean estimated advantage before the update, preserving within-trajectory contrast while removing the cross-clip bias.
Verified-demo recycling. PPO rollouts that succeed under the stricter two-hand metric (both hands meet thresholds, worse-hand error used) are replay-verified in Isaac Gym, then added back to a demonstration library that supplies BC targets in subsequent updates. This closes a self-improvement loop between PPO exploration and BC regression without any external demo collection stage past the initial BiHOI/mocap seed.

Evaluation is in Isaac Gym; comparisons against DexTrack-bi and Dexplore-bi use the same retargeted-reference conversion pipeline for Dexplore-bi (DexTrack-bi uses its own converted reference and scene, so views are not spatially aligned in the side-by-side gallery).

Results

Generalist, no fine-tuning: 86.79% success on unseen trajectories of training objects; 57.32% on held-out objects. Deltas over bimanual-adapted SOTA baselines (DexTrack-bi, Dexplore-bi) are +77.36 and +45.73 points [Abstract, Table 2].
Stricter two-hand metric (worse-hand error, both-hands-meet-thresholds success) is what the above numbers are measured under — not a per-hand average [Table 2 caption].
Qualitative evidence on the project page: BiHOI rollouts, held-out-object generalization clips (shorter source clips hold their final frame), and a side-by-side gallery where one transport drives DONIC and both baselines simultaneously.
No per-ablation isolation of the three design choices (object-centered per-hand reward / CCA / demo-recycling loop) is reported in the abstract or Table 2 caption snippet; the ablation stack has to be inferred from the paper body.

Why it’s interesting

DONIC lands cleanly in two filed debates. On Human-to-Robot Retargeting, it stakes a bimanual contact-role-aware position: keep the full dexterous reference, but let each hand be supervised against its own reference-dictated contact pattern — directly complementary to ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control ConTrack’s single-hand dual-variable RL (object-tracking as hard constraint) and sibling to Do as I Do: Dexterous Manipulation Data from Everyday Human Videos DO AS I DO on the “bypass teleop, convert human video” axis. On Synthetic Training Data, the verified-demo recycling loop is a concrete instance of the self-improvement / self-distillation pattern (closely related in spirit to Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning on the policy-improves-its-own-training-set axis), scoped specifically to dexterous bimanual tracking rather than long-horizon manipulation. The 77-point margin over bimanual-adapted single-hand baselines also sharpens the standing question of whether bimanual coordination needs a dedicated reward structure or whether single-hand recipes scale by just doubling the action space — DONIC’s answer is that per-hand contact-role supervision + clip-aware advantage normalization is load-bearing.