Skip to content
Bud's Wiki

EgoHand: Monocular 3D Hand Reconstruction from Egocentric Video

TL;DR

Section titled “TL;DR”

EgoHand is NVIDIA’s open-weights monocular 3D hand reconstruction model targeting the egocentric-video substrate most human-to-robot retargeting pipelines depend on. A transformer encoder takes 2D hand crops with camera-cue conditioning and an auxiliary 2D branch to output SOMA-Hand parameters; training sits on a curated 14M-image corpus mixing synthetic data, in-the-wild video, and multi-camera motion capture to trade off diversity against supervision quality. The model reports SOTA zero-shot joint-position, root-translation, and acceleration errors on HOT3D, ARCTIC, and H2O at ~9 ms per frame, with a blinded user study preferring EgoHand reconstructions over Apple EgoDex and Ropedia Xperience-10M. Weights, inference pipelines, and training code are released under the OpenMDW license (free commercial use).

Key claims

Section titled “Key claims”
  • A 14M-image training corpus combining synthetic data, in-the-wild video, and multi-camera motion capture balances scene diversity against supervision quality, serving as the central methodological lever [§Abstract].
  • The architecture uses the SOMA-Hand parametric representation with a transformer encoder over 2D hand crops, augmented by camera cues and an auxiliary 2D branch for geometric consistency and image alignment [§Abstract].
  • Zero-shot evaluation on HOT3D, ARCTIC, and H2O reports lower joint-position, root-translation, and acceleration errors than competing baselines at ~9.01 ms inference latency [§Abstract].
  • Blinded user preference study shows participants favor EgoHand’s reconstructed hand poses over Apple EgoDex and Ropedia Xperience-10M [§Abstract].
  • Model + source code released under OpenMDW license allowing free commercial use [§Abstract].

Method

Section titled “Method”

EgoHand takes 2D hand crops from monocular egocentric video and runs them through a transformer encoder that outputs SOMA-Hand parametric parameters. Two explicit design choices are highlighted: (a) camera cues are fed as input to the encoder, letting the model condition on egocentric-headset intrinsics rather than assuming a canonical pinhole model; (b) an auxiliary 2D branch supervises image-aligned 2D keypoint predictions alongside the parametric 3D output, which the paper argues drives geometric consistency and improves alignment to the input pixels.

The training corpus is the central engineering lever: 14M images assembled from three sources with distinct trade-offs — synthetic data for diversity and perfect supervision, in-the-wild video for scene coverage, and multi-camera motion capture for supervision quality on real hands. Evaluation is zero-shot on three standard HOI benchmarks (HOT3D, ARCTIC, H2O), reporting lower joint-position, root-translation, and acceleration errors than unspecified baselines at 9.01 ms per frame (Blackwell GPU per the companion post). A blinded user-preference study against Apple EgoDex and Ropedia Xperience-10M is used as a perceptual check beyond numeric benchmarks.

Results

Section titled “Results”
  • Inference latency ~9.01 ms per frame on a Blackwell GPU [§Abstract].
  • Reports SOTA joint-position, root-translation, and acceleration errors on HOT3D, ARCTIC, and H2O under zero-shot evaluation [§Abstract].
  • Blind user study: participants prefer EgoHand reconstructions over Apple EgoDex and Ropedia Xperience-10M [§Abstract].
  • Companion LinkedIn post reports 25,000+ Hugging Face downloads within two weeks of release [pointer note].

Why it’s interesting

Section titled “Why it’s interesting”

EgoHand lands in a load-bearing slot for the retargeting pipeline this team already tracks: multiple recent VLA/WAM substrates — HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos HumanEgo (which explicitly argues upstream hand-tracking quality bounds downstream policy success), The Will to Act — Natura-1 Natura-1 (whose 41% MPJPE reduction over open-source baselines is the gating claim for its 1M-hour corpus), and EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning EgoInfinity — all depend on a monocular-RGB-to-3D-hand front-end whose quality they treat as the bottleneck. EgoHand is the first open-weights NVIDIA entry in that slot, sitting next to Introducing Perceptron Mk1 Perceptron Mk1 (closed-source hand-pose-as-API) and complementing the data-collection-rig entries (Delta Intelligence launches Δ₀ — whole-body humanoid model with latent world-action brain, 69-DoF controller, and 10,000+ h egocentric panoramic pretraining Delta D1, Ego-OSCAR: Egocentric Open source Stereo CAptuRe System Ego-OSCAR).

Two things make it structurally distinct from prior filings: (a) OpenMDW licensing allows free commercial use of weights, which neither Apple EgoDex nor Ropedia Xperience-10M offer; (b) the 14M-image synthetic+in-wild+mocap mixture is the explicit lever, in a space where most entries (LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World LAMP, MolmoMotion: Language-guided 3D motion forecasting MolmoMotion) focus on architectural novelty. The SOMA-Hand parametric choice echoes the SAM 3D Body: Robust Full-Body Human Mesh Recovery SAM 3D Body / MHR move away from SMPL-X — signaling NVIDIA is standardizing on its own parametric hand model across the stack.

See also

Section titled “See also”