EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics

TL;DR

EgoHand v1.0 (NVIDIA) is an 883.5M-parameter monocular hand reconstruction foundation model that takes a single 256×256 RGB hand crop plus camera intrinsics and emits SOMA-hand parameters (joint pose + 27D MHR shape/scale coefficients) and a full-perspective camera translation. A DINOv3 vision backbone initialized from SAM 3D Body feeds a TransformerDecoder head with learnable init queries for pose/shape/camera; shape/scale coefficients parameterize MHR’s rest-pose identity, which SOMA’s rig then deforms via forward kinematics to produce the final 3D hand mesh. Trained on ~14M images (~9M in-the-wild+mocap, ~5M synthetic) + ~126 h of video with keypoint-2D, keypoint-3D, and MHR/SOMA parameter losses, with HOT3D + H2O as the main evaluation substrates. Released under OpenMDW 1.1 / Apache 2.0 on Hugging Face (25K downloads at filing) and positioned explicitly as training-supervision infrastructure for robot foundation models — not a product in itself.

Key claims

The model consumes a single 2D-bounding-box-cropped 256×256 RGB image plus a 3×3 intrinsics matrix and outputs SOMA pose + 27D MHR shape/scale + 3D camera translation; the MHR identity is then posed via SOMA’s FK rig to produce the full 3D hand mesh [Model Architecture, Input, Output].
The intended downstream use is to supply accurate 3D hand keypoints and MHR/SOMA mesh parameters as training signal for robotic foundation models and manipulation policies that need per-frame hand trajectories from human demonstration video [Use Case].
The backbone is DINOv3 initialized from Meta’s SAM 3D Body checkpoint; the head is a SOMAHandTransformerDecoderHeadV2 with learnable init pose/shape/camera queries [Model Architecture].
Training mixes ~9M in-the-wild + mocap images with ~5M synthetic frames from NVIDIA’s RigPlay/DigitalHuman renderer plus ~16 h of marker-based mocap (1360×1360 @ 30 FPS and 2464×2064 @ 60 FPS); losses include keypoint-2D, keypoint-3D, and MHR/SOMA parameter targets computed in the crop frame [Training Dataset].
Evaluation is on HOT3D (9 subjects, 1.5K clips, 227K frames with UmeTrack mocap ground truth) and H2O (4 subjects, 184 clips, 114K frames with multi-view MANO fits); EgoDex (338K episodes, ~90M frames, 194 tasks, Apple Vision Pro) is used only for qualitative evaluation [Testing Dataset, Evaluation Dataset].
Benchmark score is undisclosed on the model card — only qualitative inference-pipeline + finetuning repos are linked [Evaluation Dataset, References].
Licensed under OpenMDW 1.1 with Apache 2.0 additional terms; test hardware is L40 / L40s with PyTorch runtime on NVIDIA Ampere Linux [License, Inference].

Method

EgoHand factorizes monocular 3D hand recovery into (a) a DINOv3-based image encoder that consumes a 256×256 RGB hand crop, (b) a TransformerDecoder head with three learnable query sets (pose, shape, camera) that cross-attend to encoder features, and (c) a parametric-identity + skeletal-pose output head. The 27D shape/scale coefficients parameterize MHR’s rest-pose hand identity (a Meta-released human rig model distinct from MANO), and SOMA — NVIDIA’s rig format — is then used to drive forward kinematics from the predicted pose onto the identity-conditioned skeleton, producing the final 3D mesh. Crucially, the camera head predicts a full-perspective translation conditioned on the input intrinsics rather than the weak-perspective crop-frame camera typical of prior MANO fitting pipelines, which is what makes outputs directly consumable in metric-robotics downstream stacks. The backbone is initialized from Meta’s SAM 3D Body DINOv3 checkpoint, inheriting the full-body geometry prior that SAM 3D Body: Robust Full-Body Human Mesh Recovery established on the body side and transferring it to hand-only crops.

Training data is heavily tilted toward synthetic fidelity: ~5M synthetic frames come from NVIDIA’s in-house RigPlay/DigitalHuman renderer with exact ground-truth 3D keypoints + MHR/SOMA parameters + camera pose, ~9M in-the-wild and motion-capture RGB images from third-party vendors, plus ~16 h of marker-based mocap at two formats. Supervision stacks three loss terms in the crop frame — 2D keypoints, 3D keypoints, and MHR/SOMA parameter targets.

Results

The model card discloses evaluation substrates (HOT3D 227K frames, H2O 114K frames) and qualitative-only testing on EgoDex (829 h / 338K task demonstrations / ~90M frames of Apple Vision Pro egocentric manipulation), but the Benchmark Score is explicitly Undisclosed. No MPJPE, F-score, or AUC numbers are published at filing time. The two reference repositories — NVIDIA/IsaacCapture for the inference pipeline and NVlabs/GEM-X for finetuning — are the only evidence of the integration pattern NVIDIA intends; the 25,053 Hugging Face downloads at filing suggest meaningful external uptake despite the absence of headline numbers.

Why it’s interesting

EgoHand productizes the upstream side of several retargeting-heavy recipes already on the wiki: EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale, The Will to Act — Natura-1 Natura-1, and Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2 all depend on a per-frame 3D hand pose pipeline of exactly this shape to turn raw egocentric RGB into either action targets or shape-aware kinematic retargeting — Natura-1 in particular reports its in-house pose pipeline cuts MPJPE 41% vs open baselines, and EgoHand is the first filed NVIDIA open-weights model aimed at closing that same gap for the ecosystem. It sits adjacent to SAM 3D Body: Robust Full-Body Human Mesh Recovery (which shares the DINOv3 + MHR substrate at full-body resolution) and complements the egocentric capture hardware line (Ego-OSCAR: Egocentric Open source Stereo CAptuRe System Ego-OSCAR, HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding HOT3D) — hardware supplies the raw bytes, EgoHand supplies the hand-side 3D structure. Unlike Introducing Perceptron Egocentric API Perceptron Egocentric (closed-source VLM-with-hand-pose-sidecar) and Grounded API launch — hand tracking, depth, and SLAM from one RoboCap recording Grounded API (hosted), EgoHand is openly released under OpenMDW 1.1 + Apache 2.0, making it a candidate drop-in substrate for every “convert human video into hand supervision” pipeline this wiki tracks.