HUG: Human Universal Grasping

TL;DR

HUG (Human Universal Grasping, NYU / Lerrel Pinto group) trains a dexterous multi-fingered grasping model purely on 1M-HUGs, a new 1M-frame egocentric dataset of in-the-wild human grasps captured with Aria Gen 2 smart glasses across 41 buildings and ~1.5K objects. The model is a point-conditioned flow-matching predictor that maps a single RGB-D image + a user click to a MANO hand grasp (wrist translation + 6D rotation + 15 finger joints), which is then retargeted zero-shot to different robot hands (Ability, WUJI) without any per-embodiment training. On the new HUG-Bench (90 hard, unseen objects with metric-scale meshes) HUG hits 73% real-world success on a tabletop xArm+Ability setup and 62% in an uncontrolled house on a WUJI-handed mobile manipulator, beating Dex1B (sim-trained, 44%) and CAP (parallel-jaw, 33%) by 22–30 absolute points. It is, to the authors’ knowledge, the first grasping framework trained purely on human data that deploys across multiple robot embodiments.

Key claims

Purely-human-data training + kinematic retargeting can beat sim-trained multi-fingered grasping (Dex1B) and modern parallel-jaw grasping (CAP) on hard, unseen objects: HUG reaches 66.7% (RGB+PC) / 73.0% (with extras) tabletop success on the 30 HUG-Bench test objects vs 43.7% Dex1B and 32.7% CAP [Table 3].
Aria-Gen-2-style egocentric collection + camera-pose back-propagation turns each physical grasp into hundreds of (object-only image, grasp) training pairs from diverse viewpoints, yielding 1M frames from 6,707 recordings at ~28 hours of raw capture [§3, Table 1].
The dual-modal RGB+PC design is genuinely complementary: RGB-only collapses to ~27% val SR and 108.6 mm FC error, PC-only holds at 64%/25.6 mm, and the full RGB+PC model reaches 71.5%/19.0 mm — RGB grounds the semantics of where to grasp while PC provides the metric geometry [Table 2].
Explicit 3D fingertip-placement supervision through the MANO decoder is the single most critical ingredient: removing the auxiliary L1 landmark loss (Eq. 1) drops test SR from 73.0% to 32.7% and doubles FC error to 35.7 mm [Table 2].
The model is not yet capacity-bound at 1M frames: sweeping 25K → 1M frames raises test SR from 33% → 73% and drops FC error 54.2 → 14.6 mm on a log-linear trajectory with no visible saturation [Fig. 2].
Zero-shot cross-embodiment transfer works: the same MANO grasps deployed on a WUJI 20-DoF hand + AgileX arm + Aria vision in an unseen household achieve 62% in-the-wild success — only ~11 points below tabletop, with no on-site retuning [§5.3, Table 3].
Failure modes concentrate at the closing / lifting stages, not at reaching, suggesting motion planning (to avoid table intersection during closure) and force-aware closing (to prevent post-grasp slip) would recover most of the remaining gap [Fig. 11, §5.3].

Method

HUG is a point-conditioned flow-matching model over a 105-dim MANO grasp state (3-dim wrist translation in camera frame + 6D wrist rotation + 15 × 6D finger rotations). Inputs are a single RGB image, a metric point cloud back-projected from stereo depth, and a 2D user click that is lifted to a 3D query point using its depth value. Encoders: a frozen DINOv2-Base ViT with register tokens on the RGB, and a trainable PointNeXt U-Net on a 0.4 m crop of the point cloud around the query. The two streams are fused by point painting — each PC token is concatenated with the DINOv2 patch feature at its 2D projection — and refined by a 6-layer pre-norm transformer that cross-attends to the query token, producing scene-conditioning tokens.

The flow transformer decomposes the grasp into three tokens (translation, wrist rotation, finger pose) — separated to prevent geometrically distinct components from over-mixing — passed through 6 DiT blocks with AdaLN-Zero timestep conditioning that cross-attend to the scene tokens. Training combines a velocity-prediction MSE with an auxiliary L1 loss on the 3D MANO hand landmarks obtained by decoding the predicted clean state (weighted w(t) = (1−t)² to concentrate on near-clean steps). MANO shape β is held fixed at a canonical value across the dataset so the model learns only articulation + placement. 100K steps, AdamW, LR 1e-4, batch 64, on two RTX 5090s in ~19 h. Predicted MANO grasps are then retargeted with an off-the-shelf optimizer (dex-retargeting for Ability, ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control-style constraints for WUJI) to the target hand and executed open-loop as pre-grasp → grasp → 0.3 m lift.

Results

On HUG-Bench in MuJoCo: full RGB+PC hits 71.5% val SR / 19.0 mm FC and 73.0% test SR / 14.6 mm FC vs a human-oracle upper bound of 90.3% / 9.4 mm and 94.0% / 7.4 mm — within ~19–21 SR points of the recorded-human ceiling [Table 2]. Ablations (test SR): −3D loss 32.7%, −crop 58.0%, −point-paint 58.3%, PC-only 70.7%, RGB-only 29.7% [Table 2]. Data scaling: 25K → 33%, 50K → 43%, 100K → 55%, 250K → 63%, 500K → 68%, 1M → 73% [Fig. 2] — no saturation.

Real-world tabletop (xArm 7-DoF + Ability 6-DoF hand + ZED stereo, 10 trials × 30 objects = 300 trials each): HUG 66.7% / 30-of-30 objects grasped ≥ once, Dex1B 43.7% / 27-of-30, CAP 32.7% / 20-of-30 [Table 3]. The augmented HUG variant (unclear which knob from the table) reaches 73.0% / 30-of-30. HUG wins decisively on large prismatic objects (storage bin 10/10 vs 0/10 both baselines), handles/irregular structure (picnic basket 9/10 vs 1/10 and 0/10; easel 8/10 vs 2/10 and 4/10), and remains competitive on small objects (eraser 6/10, strawberry 7/10). Common failures are Ability-hardware-specific (football, wipe dispenser — too large to wrap). In-the-wild (mobile YOR + AgileX NERO + WUJI 20-DoF hand + Aria Gen 2, 300 trials across kitchen/living room/bedroom/office of an unseen home): 62.0% overall, 29-of-30 objects grasped ≥ once [Table 3].

Why it’s interesting

HUG plants a new distinct position on the human-to-robot-retargeting design axis that Human-to-Robot Retargeting is tracking: rather than the physics-in-loop optimization of ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting, the 4D reconstruction pipeline of Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, the RL-constraint dual-variable update of ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control, or the drop-the-noisy-channels 3-DoF wrist bridging of Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots, HUG keeps the full 51-DoF MANO grasp and pays for the retargeting gap purely by scale + a canonical-hand-shape normalization step (fix β so θ denotes the same grasp across collectors). This is a sharp counter to the reduction-based Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots position and a leaner cousin of EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data‘s 22-DoF-joints-plus-scale bet (EgoScale keeps joints and pays with 20,854 h; HUG keeps MANO articulation and pays with only 28 h × 6,707 recordings, aided by the multi-view repropagation trick).

It also contrasts with Do as I Do: Dexterous Manipulation Data from Everyday Human Videos which shares the Fouhey/Malik lineage (Dandan Shan and David Fouhey are co-authors here) but attacks the problem from monocular RGB videos with 4D reconstruction, whereas HUG uses stereo RGB-D from smart glasses and skips explicit hand-object reconstruction — a cleaner input signal at the cost of requiring Aria-class hardware for collection. The 62% in-the-wild success on an unseen household is comparable to HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos‘s zero-shot claim but at multi-finger dexterous granularity that HumanEgo defers to future work.