SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

TL;DR

SimToolReal reformulates dexterous tool use as a single object-centric, goal-pose-conditioned RL problem: procedurally generate handle-plus-head tool primitives in simulation, train one policy on a 22-DoF Sharpa dexterous hand + 7-DoF KUKA arm to move each tool to random goal poses, and at test time compose it by conditioning on a sequence of 6-D goal poses extracted from a single RGB-D human video. The policy only sees the tool’s current 6-D pose and a coarse 3-D grasp-region bounding box — recoverable in the real world via SAM 3D + FoundationPose — so it transfers zero-shot to novel tools without any object- or task-specific engineering. On DexToolBench (24 tasks, 12 tools, 6 categories, 120 real-world rollouts), it beats kinematic retargeting and fixed-grasp baselines by +37% task progress and matches specialist RL policies trained per-object/per-task.

Key claims

Reframing dexterous tool use as goal-pose reaching over procedurally generated primitives induces the skills required (stable grasping from a flat surface, in-hand reorientation, contact-force regulation) without task-specific reward shaping [§III-A].
The object-centric policy input — 6-D tool pose plus a coarse 3-D grasp-region bounding box (center + extents in object frame, held fixed per episode) — is minimal enough to be reliably extractable from vision foundation models yet sufficient for zero-shot transfer [§III-B].
Real-world deployment recovers this representation from a single third-person RGB-D human video: SAM 3D extracts the metric-scale mesh, SAM 2 segments the grasp region, and FoundationPose provides both offline goal-pose trajectory and 30 Hz online tracking during execution [§III-C].
Reward r = r_grasp + r_shape + 1[grasped] · r_goal, where r_goal = Δd_min + β · 1[success], uses a minimum-distance-so-far baseline to prevent the agent from farming reward by hovering near the goal, and only credits monotone progress [§III-A, Eq. 2].
Procedural tool generation: each tool is a handle + head, both sampled as cylinders/cuboids with varying dimensions and independent densities (mimicking real tools like hammers where head mass dominates handle mass) [§III-A].
SAPG (a PPO variant maintaining a policy population to escape exploration bottlenecks in massively parallel sim) is essential — standard PPO gets stuck [§III-B(a)].
Asymmetric actor-critic: actor sees only test-time-available inputs, critic sees privileged clean state — improves training stability [§III-B(c)].
On DexToolBench 120-rollout evaluation, SimToolReal averages strong Task Progress across all 6 categories, degrades on thinner tools (1cm flat spatula) and heavier tools (331g mallet vs 36g claw hammer), and lowest on screwdrivers which require continuous spinning [§IV-A, Fig. 4].
Failure-mode breakdown: 43.7% pose-tracking loss, 34.5% object drops, 18.2% failure to complete in-hand rotation, 3.6% grasp failure — with strong observed re-grasp recovery behavior [§IV-A].
Beats kinematic-retargeting baseline (which never grasps the brush — kinematic-only motion fails to establish contact) and fixed-grasp baseline (which succeeds on no-rotation variant but collides with table on the 90°-rotation variant) by an average of +37% Task Progress [§IV-B, Fig. 5].
Matches specialist RL policies trained per-object/per-task on their own training setup (Obj-A/Traj-A), while the specialists collapse on any object or trajectory variation — specialist policies overfit hardest to object identity [§IV-C, Fig. 6].
Introduces DexToolBench: 24 daily tool-use tasks, 12 tool instances across 6 categories (hammer, marker, eraser, brush, spatula, screwdriver), each paired with an RGB-D human video and digital-twin sim environment; raw videos, processed data, sim envs, and tool purchase links are released [§III-D].

Method

Simulation training. Procedurally generate tool primitives = handle (cylinder or cuboid) + head (cylinder or cuboid) with randomized dimensions and independent densities. Each episode places a random tool on a table, samples a random robot joint configuration, and samples a sequence of goal object poses {g₁, …, g_T} — first goal random in reachable workspace, later goals close to the previous. Robot is 7-DoF KUKA iiwa 14 + 22-DoF Sharpa left hand.

Policy inputs: current 6-D object pose, proprioception, and a coarse 3-D grasp-region bounding box (center + extents in object frame, fixed per episode). LSTM backbone lets the policy implicitly infer latent physical/geometric properties from interaction history — following DexFunc / RMA. Asymmetric actor-critic: critic sees privileged clean sim state; actor sees only test-time-available noisy/delayed observations.

Reward: r_grasp + r_shape shape early exploration; after grasp, r_goal dominates. r_goal = Δd_min + β · 1[d < ε] where d_min tracks the minimum-distance-so-far and resets on new goal — preventing the “hover-for-reward” degenerate solution. Distance uses keypoint positions in local pose frames (Yang et al. formulation).

Domain randomization: observation delays, action latency, noisy 6-D pose estimates, perturbed bounding boxes, random force/torque perturbations on the object (encouraging strong grasps). Training via SAPG (population-based PPO variant).

Real-world deployment. From the first RGB-D frame of a third-person human video, SAM 3D reconstructs a metric-scale mesh; SAM 2 masks the grasp region and converts it to a 3-D bounding box (fixed per-episode input to the policy). FoundationPose is run offline on the full RGB-D video conditioned on the extracted mesh to produce a downsampled-to-3-Hz goal-pose trajectory. At execution, FoundationPose runs online at 30 Hz for tool-pose feedback; the policy is conditioned on proprioception + current pose + fixed bounding box + current goal-pose target, and advances to the next goal when tracking error falls below ε.

Results

Real-world zero-shot on DexToolBench (24 tasks, 12 tools, 6 categories, 5 trials each = 120 rollouts): strong average Task Progress across all categories; highest on eraser (translation-dominated) and marker (though thin geometry hurts grasp reliability); intermediate on hammer/brush/spatula (require 90-180° in-hand rotation); lowest on screwdriver (requires functional reorientation plus continuous spinning) [§IV-A, Fig. 4].
Baseline comparison on brush-sweep-forward (2 variants, 5 rollouts each): kinematic retargeting fails to grasp on both variants (kinematic motion doesn’t establish stable contact); fixed-grasp succeeds on the no-rotation variant but collides with the table on the 90°-rotation variant because the arm alone can’t rotate the tool without in-hand manipulation; SimToolReal succeeds on both. Averaged across the benchmark, SimToolReal beats the two baselines by +37% Task Progress [§IV-B, Fig. 5].
Specialist comparison in simulation (6 category specialists, 10 rollouts × 6 tools × 3 variations): SimToolReal matches specialists on the exact object+trajectory the specialist trained on, and outperforms specialists on any object variation (specialist collapses hardest here) or trajectory variation (specialist can only track the first few goals then loses the object) [§IV-C, Fig. 6].
Failure-mode taxonomy: 43.7% pose-tracking loss (FoundationPose limitation, worst on thin/occluded tools), 34.5% object drops, 18.2% incomplete in-hand rotation, 3.6% grasp failure [§IV-A].
Ablations (Appendix): removing domain randomization, force perturbations, or SAPG each substantially degrade sim performance; naïve PPO fails to learn the tool-primitive task [§IV-E].

Why it’s interesting

SimToolReal is a clean argument for the object-centric + goal-pose factorization as the right level of abstraction for sim-to-real dexterous manipulation. The insight is that if you specify tasks as sequences of tool-frame goal poses, you can procedurally generate an effectively infinite training curriculum of tool primitives and let RL discover the underlying skills (stable grasp, in-hand reorientation, contact regulation) without ever engineering per-task rewards or per-object simulation models. This complements Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (which does functional retargeting from human video for dexterous hands but trains per-task) and contrasts with the imitation-learning direction taken by Envisioning the Future, One Step at a Time (Myriad) and the tactile-in-loop direction of Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks and Tactile sensing for manipulation — SimToolReal shows that for a large class of tool-use tasks, tactile feedback is not essential if you can extract accurate object-frame goal poses from a single human video. It’s also a strong empirical validation of the vision-foundation-model stack (SAM 3D + SAM 2 + FoundationPose) as the perception layer that closes sim-to-real for dexterous policies — the same three components appear across World Foundation Models work but here they carry the entire real-world visual gap on their own. The 43.7% pose-tracking-loss failure mode is the honest open problem: SimToolReal is only as good as FoundationPose’s tracking on thin/occluded tools.