GraspXL: Generating Grasping Motions for Diverse Objects at Scale

TL;DR

GraspXL (ETH AIT, ECCV 2024; now HKUST-GZ) is an RL framework that synthesizes objective-driven grasping motions across four dexterous hand morphologies (MANO, Allegro, Shadow, Faive) for 500k+ unseen objects — without any 3D hand-object interaction data during training. A single policy trained on only 58 PartNet/ShapeNet objects generalizes to Objaverse-scale test sets at 82.2% success while adhering to multiple simultaneous motion objectives (graspable area, heading direction, wrist rotation, hand midpoint position). The load-bearing recipe pieces are (a) hand-morphology-agnostic distance features (finger-joint-to-nearest-object-vertex vectors) as observations, (b) a two-phase curriculum that trains on stationary objects with objective-reward-heavy first then fine-tunes on non-stationary objects with grasp-reward-heavy, and (c) objective-driven wrist-PD-bias guidance that pipes target-vs-current deltas directly into the 6-DoF wrist controller bias term. Ships a released grasping-motion dataset covering the 500k+ Objaverse subset × multiple hand models, plus a 2026-05 update adding a tabletop grasping setting.

Key claims

Trained on 58 objects, the policy grasps 500k+ unseen Objaverse objects at 82.2% average success rate across small/medium/large size buckets, with 3.32 cm midpoint error and 0.279 rad heading error [Table 3].
Beats SynH2R (the only prior baseline offering heading-direction controllability, extended by the paper with wrist-rotation and midpoint terms in its optimization) by ~30% success-rate on the PartNet test set (95.0% vs 82.3%) and by 30-50% on objective errors (Midpoint 2.85 vs 4.06 cm, Heading 0.270 vs 0.522 rad, Rotation 0.306 vs 0.568 rad) [Table 2].
Grasp success on the ShapeNet test set drops less than baselines (81.0% vs 65.8% for SynH2R), evidencing better generalization out-of-training-distribution [Table 2].
Runs in real-time inference in simulation vs SynH2R/SynH2R-PD which require ~1 week of grasp-reference generation for the ShapeNet test set [§4.2].
Cross-embodiment transfer without pipeline changes: Allegro 95.3%, Shadow 94.0%, Faive 95.8%, MANO 95.0% success on PartNet [Table 5]. Only per-embodiment adjustments are hand state / action space dimensions.
Generalizes to reconstructed and generated objects: HOLD-reconstructed YCB 74.0%, in-the-wild reconstructed 77.5%, DreamFusion-generated 88.4% [Table 4]; performance on HOLD-reconstructed objects matches ground-truth YCB (73.7%), evidencing robustness to reconstruction noise.
Distance features (finger-joint-to-object-surface vectors partitioned into graspable/non-graspable) are the shape encoder — no PointNet, no VN, no per-object embedding [§3 Feature Extraction].
Reward is decomposed into an objective term (approach + heading + wrist rotation + midpoint) and a grasp term (contact + force + velocity), with an anatomy reward added only for the MANO hand [§3 Reward Function, Eq. 2-6].
The two-phase curriculum (stationary-object precision phase → non-stationary-object stability phase) is load-bearing: ablation section 4.4 quantifies its contribution [§4.4].
Objective-driven guidance — applying target-vs-current differences (heading, wrist rotation, midpoint) directly as bias terms to the 6-DoF wrist PD controller — improves exploration and convergence [§3 Objective-driven hand guidance, §4.4].
Dataset release: grasping-motion sequences for 500k+ Objaverse objects across multiple hand models (initially MANO + Allegro, later LEAP subset + Shadow), released on Hugging Face (ethHuiZhang/GraspXL) under CC BY-NC 4.0; 2026-05 update added tabletop grasping setting for MANO/Allegro/Shadow and fixed MANO little-finger artifacts.

Method

Task setup. The hand is modeled as N links with joint angles q, global orientation split into a heading-direction vector v and a wrist-rotation angle α about v, plus a midpoint m defined as the mid-point between the thumb tip and the middle-finger’s third joint. The object is a rigid mesh with vertices partitioned by the user into graspable set G and non-graspable set NG. The user optionally specifies target heading v*, wrist rotation α*, midpoint m*, and partition G/NG. Only v* is mandatory; unspecified fields fall back to the current values or defaults.

Feature extraction. State conversion produces a vector containing: joint angles q, joint-tracking-error d (against PD targets), hand and object velocities, per-link contact indicators c, per-link contact-force magnitudes f, and the three objective-delta terms (Δv, Δα, Δm). Object shape features are the per-link nearest-vertex offset — for every link i, compute the difference vector from the link position to the closest point in G (giving distance matrix DG) and separately to the closest point in NG (giving DNG). These distance features are what carries object shape into the policy — no learned encoder, no point-cloud MLP.

Reward. R = R_obj + R_grasp. R_obj combines an approach-and-avoid term (weight the nearest-graspable-vertex distance minus the non-graspable distance) with three per-objective error terms (heading angle mismatch, wrist rotation error, midpoint L2). R_grasp combines a contact score (fingers-in-graspable-area minus fingers-in-non-graspable-area), a force term (positive on graspable contacts up to a weight-proportional cap, negative on non-graspable contacts), and a velocity-penalty for stability. When the hand is MANO, an additional anatomy reward is added for natural finger poses.

Two-phase curriculum. Phase 1: train PPO on stationary objects (fixed pose, no free rigid-body simulation) with a large R_obj weight — this teaches precise finger placement without the exploration cost of a moving object. Phase 2: fine-tune on non-stationary objects with a large R_grasp weight — this teaches the wrist and finger closures needed for a stable grasp that survives contact.

Objective-driven wrist guidance. Compute Δv, Δα, Δm each step and add them as bias terms directly to the 6-DoF PD controller for the wrist. This is a hand-engineered gradient that biases exploration toward the objective without waiting for the reward signal to shape the policy.

Training + simulation. PPO with RaiSim as the physics simulator; single Nvidia RTX 6000 + 128 CPU cores. During training, target heading, rotation, and midpoint are randomly sampled per rollout under mild feasibility constraints (graspable-part narrower than 12 cm along the hand y-axis to avoid oversized grasps; midpoint on the graspable surface). Inference is real-time in simulation — no per-episode optimization needed.

Results

PartNet test set (48 unseen objects, 25 random objective samples each): 95.0% success vs SynH2R 82.3% and SynH2R-PD 26.5%. Midpoint 2.85 cm vs 4.06 / 4.30. Heading 0.270 rad vs 0.522 / 0.767. Rotation 0.306 rad vs 0.568 / 0.857. Contact Ratio 86.7% vs 53.4 / 13.0 [Table 2].
ShapeNet test set (3993 unseen objects, 5 random objectives each): 81.0% success vs SynH2R 65.8% and SynH2R-PD 21.9%. All objective-error metrics also lead. Absolute drops are smaller than baselines, evidencing better generalization [Table 2].
Objaverse test set (503k unseen objects): 82.2% average success. Small 85.9%, Medium 84.5%, Large 79.0%. Smaller objects have higher grasp success but larger heading/rotation error (easier to grasp, easier to rotate in-hand) [Table 3].
Reconstructed / generated objects: DreamFusion-generated 88.4%, HOLD-reconstructed (in the wild) 77.5%, HOLD-reconstructed (YCB) 74.0%, ground-truth YCB 73.7% [Table 4]. HOLD-YCB matching GT-YCB is the sharpest evidence for reconstruction-robustness.
Cross-embodiment (PartNet / ShapeNet): Allegro 95.3 / 83.4, Shadow 94.0 / 83.2, Faive 95.8 / 82.4, MANO 95.0 / 81.0 [Table 5]. Allegro has larger midpoint error due to its larger size; Shadow has slightly higher objective errors due to broad flat palm limiting dexterity — but success rates are within 1 pp of MANO across the board.
Inference time: real-time in simulation vs approximately one week of grasp-reference-optimization for baselines on the ShapeNet test set [§4.2].
Dataset released (Hugging Face ethHuiZhang/GraspXL): grasping-motion sequences over 500k+ Objaverse objects for MANO, Allegro, LEAP (subset), Shadow hands; 2024 diverse-approach-direction setting + 2026-05 tabletop-grasping-setting update.

Why it’s interesting

Two things worth pulling out for Luma. First, GraspXL is the procedural, physics-simulation-only counterpart to the recent data-driven grasp-generation work on the wiki. GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training (GraspGen) trains a DDPM diffusion model on ~53M Isaac-simulated grasps over ~36K Objaverse-LVIS meshes — and uses GraspXL as one of its grasp-generation baselines for the ACRONYM comparison. GraspXL sits at the upstream generation side (RL policy in simulation, no offline dataset needed), while GraspGen sits at the downstream diffusion-generator + discriminator side that consumes such offline datasets. The two-paper pair together spans the design space: procedural physics rollouts as the substrate (GraspXL) vs a diffusion model fit to those rollouts + On-Generator Training to close the gap (GraspGen).

Second, on the retargeting axis this is a hand-morphology-agnostic policy rather than a hand-morphology-agnostic interface. Compare to the interface-location debate on Human-to-Robot Retargeting: recipes like Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots drop channels to make one action-space work across embodiments, and Cross-Embodiment Robot Manipulation via a Unified Hand Action Space designs a geometric fingertip-surface interface shared across hands. GraspXL instead keeps the full per-embodiment action space but makes the policy inputs (distance features from links to object surface) morphology-agnostic — a fourth position on the “where to put the shared abstraction” question that is orthogonal to the interface-representation-reduction axis those papers debate. The 2024-vintage curriculum and objective-guidance mechanisms remain load-bearing across all four hands with no per-embodiment retraining of the recipe, which is what makes it a plausible substrate for anyone building a dexterous-grasp data engine on top of arbitrary CAD meshes.