GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training

TL;DR

GraspGen (NVIDIA) is a 6-DOF object-centric grasping framework that pairs a DDPM diffusion generator with a lightweight discriminator, both built on a PointTransformerV3 (PTv3) object encoder. The core technical bet is On-Generator Training: instead of training the discriminator on the same offline (positive, negative) grasp dataset used for the generator, run inference with the trained generator over the training set, simulate the resulting grasps, and use those pass/fail labels to train the discriminator so it learns the generator’s specific failure modes. Ships with a new open synthetic dataset of ~53M simulated grasps across three grippers (Franka Panda, Robotiq-2F-140, 30mm suction) generated by Isaac shaking-simulations over ~36K Objaverse-LVIS meshes.

Key claims

On-Generator Training closes a measurable distribution shift between offline (positive, negative) grasp data (e.g. ACRONYM) and the actual grasp distribution the diffusion model emits; Earth Mover’s Distance between offline and on-generator sets is substantial, especially for negatives, and the discriminator trained on on-generator labels alone reaches highest AUC vs offline-only (~10% lower AUC) or mixed [§4.2, Fig. 6].
Beats prior SOTA on ACRONYM-Franka single-object grasping by >48% AUC over Contact-GraspNet, M2T2, DexDiffuser, and SE3-Diffusion Fields in a precision-coverage evaluation over 815 objects × 2K grasps = 1.6M grasp attempts [§4.1, Fig. 3].
On FetchBench (cluttered manipulation with ground-truth collision meshes and cuRobo motion planning, 100 scenes × 60 tasks), GraspGen achieves SOTA against Contact-GraspNet and M2T2 with the reported margin — task success is bottlenecked by planner reachability, so the oracle-grasp planner itself is not at 100% [§4.1, Fig. 4].
Training the diffusion model on a 50-50 mix of partial (single-view) and complete (fused multi-view) point clouds recovers a single model that generalizes across both observability regimes, while models trained on either alone transfer poorly [§4.1, Fig. 5].
The reused-encoder discriminator design (frozen PTv3 encoder from the generator + a small MLP over (object_embedding, SO(3) grasp) concatenation) is 6.7% AUC higher and uses 21× less memory than the SOTA architecture in Mousavian et al. 2019 [§4.3, Appendix 6.6.2].
Translation normalization by a dataset-derived multiplier s = 1/std(‖positive_translation‖) (empirically 3.27 for Franka) lands near the optimum of a convex normalization vs performance curve, matching a hyperparameter grid search without one [§3.1, Fig. 7].
PointTransformerV3 as the object encoder reduces translation error by a reported margin and increases recall over PointNet++ [§4.3, Appendix 6.6.4].
Ships as data: ~17M grasps per gripper × 3 grippers = ~53M total, over 36,366 Objaverse-LVIS CC-BY meshes (with an 8,515-subset matched to ACRONYM size) — Isaac shaking-simulation success labels for Franka Panda and Robotiq-2F-140, analytical vacuum-model labels (Mahler et al. 2018) for the 30mm suction cup [§3.3].

Method

The framework has two components sharing a frozen PTv3 point-cloud encoder over an object-centric input:

Generator. DDPM in the SE(3) Lie group, factored as SO(3) × ℝ³. Two dedicated denoising schedulers run for translation and rotation separately (empirically better than one combined DDPM). Both grasps and point clouds are transformed to the point cloud mean center. Only 24 denoising steps at inference — much lower than image-diffusion budgets, justified by the low intrinsic dimensionality (3+3) of the grasp target. Trained with standard denoising loss on position + orientation deltas.
Discriminator with On-Generator Training. After the diffusion model finishes training, run inference on the ~7K-object training set with 2K grasps per object, then simulate each grasp in Isaac to produce a pass/fail label. This on-generator dataset — roughly matching the offline dataset in size — is used to train a small MLP that takes (encoder_embedding, SO(3) grasp) and outputs a sigmoid success score. The PTv3 encoder itself is frozen from the generator stage; only the MLP is trained from scratch with binary cross-entropy.

The key motivation: the generative model’s own failure modes (grasps slightly in collision with the object, outliers far from the object surface) are absent from offline datasets like ACRONYM where negatives are always collision-free, so a discriminator trained only offline systematically misses the real false-positive regime.

Inference: sample a batch of noise vectors, denoise for 24 steps, score each candidate with the discriminator, return top-k.

Results

ACRONYM Franka full point cloud (single-object test, 815 objects × 2K grasps): GraspGen area-under-Precision-Coverage-curve exceeds the next-best baseline (DexDiffuser) by >48%. All discriminative methods (GraspGen, DexDiffuser, M2T2) beat purely generative SE3-Diffusion Fields scored by log-likelihood, isolating discriminator quality as a load-bearing lever [§4.1, Fig. 3].
FetchBench clutter: GraspGen surpasses Contact-GraspNet and M2T2 by margins the paper reports as “SOTA” (specific per-scene numbers behind the pdf-truncation cutoff in the fetch here); the oracle planner using ground-truth grasps from ACRONYM achieves only ~63% task and ~68% grasp success, dominated by non-grasp bottlenecks (unreachable retracts, planner limits) [§4.1].
Multi-gripper: On Robotiq-2F-140-sim, GraspGen’s lead over M2T2 widens to 42% (vs 30% on Franka-sim), because M2T2’s contact-point representation is tuned for symmetric non-adaptive grippers and mis-transfers to adaptive parallel jaws [§4.4].
Real-robot (UR10 + Robotiq-2F-140 + single RealSense D435, SAM2 segmentation, FoundationStereo depth, cuRobo planning, NVBlox collision): GraspGen’s lead over M2T2 widens to 89% relative — a much larger sim-to-real preservation than baselines [§4.4, §4.5].
On-Generator Training ablation: offline-only discriminator scores worst (~10% lower AUC), on-generator-only scores highest, mixing them is between — the paper’s argument is that offline data contains too many easily-scored samples that dilute training signal for the actual failure modes [§4.2, Fig. 6].
Discriminator: 6.7% AUC higher and 21× lower memory than the SOTA baseline [§4.3, Appendix 6.6.2].

Why it’s interesting

Two things worth pulling out for Luma. First, On-Generator Training is a general recipe: it says “if you’re training a scorer over your own generator’s outputs, you have to train it on the outputs, not on the ground-truth dataset the generator was fit to” — this is structurally the same argument Synthetic Training Data tracks about verification gates on synthetic data, and it has a clean measurement (EMD between offline and on-generator sets is non-zero and larger on negatives). Second, the ~53M-grasp dataset over CC-BY Objaverse-LVIS meshes with three gripper types is a sizable open synthetic-labels release — parallels the substrate-building work in OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction (hand-object interaction repository via grasp transfer) and 3D-Object Perception Transformer (3PT) (Blender-rendered ~1B image-render pairs for industrial 6-DoF pose), but on the grasp-label axis specifically. Rundong’s Slack framing — “useful for generate synthetic grasps in sim” — pins the near-term relevance: this is a plug-in tool for producing training/eval grasps at scale on arbitrary CAD meshes.