Fast and Accurate Hand Estimation in Pixels, Interaction and World Space

TL;DR

FAH (Fast and Accurate Hand Estimation) is a family of monocular hand-estimation models built by fine-tuning and distilling SAM 3D Body’s hand branch. Instead of regressing MANO/MHR parameters and projecting the mesh to get 2D keypoints — the dominant recipe — FAH predicts corresponding 2D and 3D joints and surface points directly, then fits MANO or MHR-Hand to them. Downstream heads optionally estimate anatomical parts and hand-object contact (without object geometry at inference), and a camera-estimation stage places the hand in world space. The default ViT-L runs at 65 fps with mesh fitting on an RTX 4070 Ti SUPER; the smallest ViT-S reaches 125 fps without mesh fitting. Code is released.

Key claims

Directly predicted 3D joints agree better with directly predicted 2D joints than joints obtained by projecting a parameter-fit mesh; fitting a mesh to the predicted points reduces mesh error relative to end-to-end parameter regression [Abstract].
A single architecture handles 2D/3D keypoints, surface points, mesh (MANO or MHR-Hand), anatomical part segmentation, and hand-object contact with one shared backbone — contact is estimated without object geometry at inference time [Abstract].
Causal temporal refinement reduces acceleration error, and camera estimation places the hand in world space rather than only in the crop-local or weak-perspective frame [Abstract].
The approach reports improvements in image-space alignment, hand geometry, contact estimation, and world-space tracking over the evaluated baselines [Abstract].
The ViT-L default runs at 65 fps with mesh fitting on an RTX 4070 Ti SUPER (49 fps with parts and contact enabled); ViT-S reaches 125 fps without mesh fitting, parts, or contact [Abstract].

Method

FAH is built on SAM 3D Body’s hand branch via fine-tuning and feature distillation. The model emits corresponding 2D and 3D joints plus 3D surface points, then fits a parametric hand model (MANO or MHR-Hand) to the predicted points as a post-process rather than regressing model parameters end-to-end. Optional heads predict anatomical part segmentation and per-point hand-object contact without requiring object geometry at inference. A camera-estimation stage lifts the per-frame prediction into world coordinates; a causal refinement module reduces jitter and acceleration error across frames. The model family spans ViT-S to ViT-L, with the smaller sizes trading accuracy for throughput.

Results

Default ViT-L: 65 fps with mesh fitting on an RTX 4070 Ti SUPER; 49 fps with parts and contact enabled [Abstract].
Smallest ViT-S: 125 fps without mesh fitting, parts, or contact [Abstract].
Author’s companion tweet reports ViT-S reaches 30 FPS on a MacBook with its built-in webcam (tweet pointer).
Improvements over baselines claimed on image-space alignment, hand geometry, contact estimation, and world-space tracking [Abstract]; specific baseline numbers not in the abstract.

Why it’s interesting

FAH is the second filed datapoint in two days (after EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics EgoHand v1.0) to productize monocular hand pose + mesh + camera-in-world at foundation-model scale as reusable infrastructure for robot learning, but it stakes a different position on the fit-vs-predict axis: EgoHand regresses MHR/SOMA parameters and camera translation end-to-end from a DINOv3 backbone, while FAH predicts keypoints + surface points and fits the parametric model to them. Author’s argument is that the parameter-regression recipe makes 2D and 3D predictions inconsistent, and that fit-to-points inverts the error budget favorably. This is the same architectural choice MoCapAnything (MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos) makes one level up — predict 3D keypoints then IK — now applied specifically to hand mesh fitting.

Paired with the real-time budget (125 fps ViT-S / 65 fps ViT-L with mesh fitting), FAH is a candidate for the “upstream hand perception” slot in the retargeting substrates on Human-to-Robot Retargeting (e.g. the kinematic-reconstruction branch of Do as I Do: Dexterous Manipulation Data from Everyday Human Videos Do as I Do, C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video C2Dex’s object-side contact inference) where prior perception stacks were either latency-prohibitive or didn’t emit contact + world-frame camera together. Contact-without-object-geometry at inference is the specific feature that would let a downstream retargeting pipeline consume contact flags without the object-tracking stack usually required to compute them.