GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping

TL;DR

GraspGen-X extends the GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training SE(3) diffusion-grasping recipe with a gripper embedding so that a single model produces high-quality 6-DoF grasps zero-shot on gripper embodiments never seen in training — including novel real hands (Robotiq-2F, Unitree G1, Inspire, Surge, Barrett, EZGripper, Sharpa, and any URDF + meshes onboarded via an interactive wizard). The load-bearing choices are (i) a hand-designed 12-dim Swept-Volume gripper representation — two axis-aligned bounding boxes describing the space the fingers sweep through at fully-open and half-open configurations — encoded by a 3-layer MLP, and (ii) training on 32 procedurally generated grippers across 6 kinematic families rather than the ~20 real grippers available off-the-shelf, since real grippers are too few and too clustered to cover the test distribution. The released model is trained on over 2 billion sampled grasps in Isaac Sim on 8000+ objects.

Key claims

Conditioning on a 12-dim Swept-Volume gripper vector (open box + half-open box) fed through an MLP outperforms every prior gripper-encoding baseline for zero-shot cross-gripper generalization: AUC 0.528 vs AdaGrasp-TSDF 0.432 (+25%), UniGrasp-cVAE 0.418, PointNet++ 0.349, on 10 held-out real grippers [Table 2].
Training on 25–50 procedural grippers beats training on 10 real grippers across every encoder tested (Swept-Volume, TSDF, PointNet++); real training grippers leave large non-overlapping regions of the Swept-Volume distribution vs the test set, procedural grippers cover the target distribution [Fig. 5, Fig. 8].
On 10 held-out real test grippers, GraspGen-X reaches 0.506 average AUC vs 0.398 for GraspGen-Retargeting (z-offset heuristic) and 0.126 for GraspGen-Direct-Transfer, split as parallel-2F 0.502 / revolute-2F 0.413 / high-dof-3F 0.699 [Table 1].
Zero-shot transfer to two 5-finger hands never seen at training (Surge, Inspire) reaches AUC 0.404 and 0.363 respectively — non-trivial performance on kinematic categories outside the training taxonomy [§5.1].
GraspGen-X serves as a better finetuning initialization than either from-scratch or single-embodiment (Franka) pretraining: converges fastest on all three metrics (translation error, rotation error, positive-grasp recall) with only 140K sampled grasps and 28 GPU-hours per gripper [Fig. 7].
Ablations on parameterization: gripper-type-only or z-offset-only vectors fail to learn; fully-open-only Swept-Volume underperforms open + half-open, because revolute 2-finger grippers (Robotiq-2F140, XArm hand) move fingers forward along the z-axis during closure and the closing dynamics need to be encoded; adding gripper-type one-hot on top of Swept-Volume does not help, suggesting cross-type parameter sharing is important [Fig. 8].
Training substrate: 2 B sampled grasps across 32 procedural grippers × 8000+ Objaverse-derived objects; the CVPR generator was trained on 175 M grasps (8.7 K GPU-hours) with 780 K steps on 8×A100, the discriminator on-generator data at 5.2 K GPU-hours [§4, Appendix].
The contact side of Ben’s Slack framing lives in the Swept-Volume: the swept box approximates the region of space in which finger-object contact will occur during closure, so the shared embodiment interface here is effectively “space where contact happens” rather than joint kinematics or contact points on the object surface [§3.1].

Method

Base model. GraspGen (single-embodiment) is an object-conditioned SE(3) DDPM grasp generator (factored SO(3) × ℝ³, 24 denoising steps) paired with a small MLP discriminator that scores each denoised sample; both share a frozen PointTransformer object encoder. GraspGen-X extends both blocks by concatenating a 512-dim gripper embedding to the object embedding.

Gripper embedding = Swept Volume. For a URDF + open/half-open joint states, compute two axis-aligned bounding boxes approximating the volume swept by the fingers between the base frame and the fingertips at each configuration. Each box is 6 dim (3 dim center translation + 3 dim extent), so the input is a 12-dim vector. A 3-layer MLP projects to 512 dim.

For always-parallel grippers (Franka Panda, Robotiq-2F-85), the box covers the space between the two fingers.
For revolute grippers whose fingers rotate relative to the base (Robotiq-3F, high-dof), the box approximates the space the finger will sweep through during subsequent closure.
For 3-finger high-dof hands (Unitree G1 7-DoF), the same antipodal-sampling grasp pipeline used for parallel grippers extends without modification.

Procedural gripper generation. Rather than train on 10 real grippers (biased and sparse in Swept-Volume space), procedurally synthesize a training distribution with Infinigen-Sim (Blender geometry nodes) over three kinematic families — parallel 2F, revolute 2F, revolute 3F high-dof. Randomization ranges are tuned so the resulting Swept-Volume distribution covers the 10 held-out real test grippers, which the 10-real-training-gripper distribution does not.

Dataset. ACRONYM-style antipodal sampling + Isaac Sim shaking-test labels, per gripper × per object. CVPR version: 25 grippers × 3.5 K objects × up to 2 K grasps = 175 M generator grasps + 5.2 K GPU-hours of on-generator discriminator data. Released model: 32 grippers × 8.5 K objects, 2 B+ grasps.

Integrating a new gripper (Ben’s angle). Zero-shot inference on any gripper needs only a URDF + meshes + a config.json describing open/close joint states and the two swept-volume boxes. A browser-based wizard walks the user through six panels (align base frame, confirm open pose, annotate open box, annotate half-open box, review animation, save) and writes the config file. Everything downstream (visual mesh merge, collision mesh, point-cloud caches) is produced automatically at first inference.

Results

Zero-shot on 10 held-out real test grippers: GraspGen-X 0.506 avg AUC vs Retargeting 0.398 vs Direct-Transfer 0.126. High-dof 3-finger is the biggest win: 0.699 vs 0.503 (RTG) vs 0.136 (DTR) [Table 1].
Zero-shot on 5-finger hands never in training taxonomy: Surge Hand 0.404, Inspire Hand 0.363 [§5.1].
Encoder head-to-head at matched training set (32 procedural grippers, 453 test objects): Swept-Volume 0.528 > TSDF 0.432 > UniGrasp-cVAE 0.418 > PointNet++ 0.349 [Table 2].
Real→procedural training-set swap: procedural training substantially outperforms real-gripper training for every encoder, and adding more procedural grippers (25 → 50) continues to improve [Fig. 8, Appendix Fig. A3].
SFT convergence: GraspGen-X-SFT reaches lower translation/rotation error and higher positive-grasp recall faster than Franka-SFT and from-scratch on 10 novel test grippers, with 140 K grasps × 28 GPU-hours per gripper [Fig. 7].
Real-hardware demos (in the code release, not the paper tables): Franka Panda + Unitree G1 + Piper arm + Surge hand doing clutter removal, battery removal, pick-and-place — all zero-shot from the same checkpoint via the wizard-based onboarding.

Why it’s interesting

Ben’s Slack framing is the right lens: this paper’s core mechanism is a gripper-shape → space-of-contact model that lets one policy retarget across embodiments without per-gripper retraining. Structurally, the Swept-Volume box sits at exactly the interface position c2dex-style contact-topology retargeting takes on the object side (C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video) — but here the contact geometry lives in the gripper’s base frame as a 12-dim vector, not on the object surface — so it’s cross-gripper generalization rather than human→robot retargeting. The Luma-adjacent lesson is that when a designed 12-dim geometric prior beats learned TSDF (+25% AUC) and learned PointNet++ (+51%) at cross-embodiment generalization, that’s a data point for the Human-to-Robot Retargeting “hand-designed low-dim interface beats high-dim learned features at generalization” position that Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s 3-DoF wrist-translation channel also stakes out. Complements GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training by moving the training substrate from real → procedural on the gripper axis (same move that GraspGen already made on the object axis via Objaverse) and lifts the 3-gripper GraspGen dataset (~53 M grasps) to a 32-gripper dataset (2 B+ grasps), giving Synthetic Training Data its largest filed multi-embodiment grasp release. Model checkpoints + code + Blender-based procedural gripper generator are open-source (Apache-2.0 code, NVIDIA Open Model License for weights); Sharpa hand + YAM gripper + DROID VLA pick-and-place data generation are flagged as upcoming community-contribution items.