SAM 3D Body: Robust Full-Body Human Mesh Recovery

TL;DR

SAM 3D Body (3DB) is Meta’s promptable single-image human mesh recovery (HMR) model — body, hands, and feet from one RGB image, optionally guided by 2D keypoints or segmentation masks like the rest of the SAM family. Two things make it land: a new parametric body representation called the Meta Momentum Human Rig (MHR) that decouples skeletal structure from soft-tissue shape (unlike SMPL/SMPL-X, where they’re entangled in a single shape space), and a scaled data pipeline — billions of source images mined for unusual poses and rare conditions, distilled down to ~8M training images via an automated data engine. Reported as SOTA on multiple HMR benchmarks. Released as part of the SAM 3D family alongside SAM 3D: 3Dfy Anything in Images under a permissive commercial license, with MHR open-sourced separately.

Key claims

The task: promptable single-image full-body 3D human mesh recovery — predict body, hand, and foot pose + shape from one RGB image, optionally conditioned on 2D keypoint and mask prompts in the SAM-family style [Abstract; §1].
Momentum Human Rig (MHR) is the first parametric human body representation in HMR that decouples skeletal structure from soft-tissue shape, unlike SMPL/SMPL-X which intertwine bone lengths and surface shape in a single shape space [§3, Fig. 2; MHR project page].
This decoupling improves interpretability — MHR parameters map directly to bone lengths — and controllability for downstream animation rigs [§3].
The architecture is a promptable transformer encoder-decoder: a multi-input image encoder captures high-resolution body-part detail, and a separate mesh decoder predicts MHR pose/shape parameters with prompt-based conditioning on 2D keypoints or masks [§3.2, Fig. 2].
A separate hand pose decoder is used; SAM 3D Body follows the recent full-body HMR paradigm rather than older body-only methods like HMR 2.0 [§2, §3].
Training data is built by mining a billion-image source pool with an automated data engine that selects for unusual poses, occlusions, and rare imaging conditions, distilling down to ~8M high-quality training images [Meta blog; §4].
Annotations come from a multi-stage pipeline mixing manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection [Abstract; §4].
Training uses prompt-based guidance and multi-step refinement, improving 2D alignment with visual evidence [Meta blog; §3.3].
Reported as SOTA on multiple 3D human benchmarks, with robustness to occlusions, unusual postures, and diverse clothing [Abstract; §5].
Known limitations: processes each person separately (no multi-person or human–object interaction reasoning); hand pose accuracy does not surpass specialized hand-only methods despite gains as part of full-body estimation [§Limitations / Meta blog].
Release package: weights and inference code under a permissive commercial license; the MHR parametric model is separately open-sourced (facebookresearch/MHR) and is also used in Meta’s Codec Avatars [Meta blog].

Method

The model takes one RGB image plus optional prompts (2D keypoints or a person mask) and outputs MHR parameters describing the subject’s skeleton (bone lengths and joint angles) and soft-tissue shape independently. The image encoder is a multi-input design that crops and processes body parts at higher resolution alongside the global image — relevant because hand and face detail is what specialized HMR methods previously had to handle with separate models. The mesh decoder is split into a body decoder and a hand decoder, conditioned by prompt embeddings when prompts are supplied. MHR itself sits at the output: a parametric rig that combines the decoupled-skeleton-and-shape paradigm of ATLAS with a flexible rig + pose corrective system inspired by the Momentum library, supporting both linear and non-linear pose correctives. The training data engine is the throughput-critical piece: from a billion-image pool, an automated selector preferentially mines uncommon poses, occlusions, and rare clothing/imaging conditions; manual keypoint annotation feeds differentiable optimization to lift those 2D labels into 3D MHR fits; multi-view geometry and dense keypoint detection provide additional supervision sources. The final training set is ~8M images, considerably richer than prior HMR corpora. Training is prompt-aware (so the same checkpoint serves both unprompted inference and SAM-style mask/keypoint-guided refinement) and uses multi-step refinement to tighten 2D alignment.

Results

SOTA on multiple 3D HMR benchmarks; outperforms prior models on body, hand, and foot estimation tasks [Abstract; §5].
Robust to occlusions, unusual postures, and diverse clothing — the explicit targets of the data-engine mining policy [Meta blog].
Released model variants include a DINOv3-encoder configuration (facebook/sam-3d-body-dinov3 on HuggingFace).
Downstream uptake at filing time already includes SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning-style integrations into video tracking, motion retargeting to humanoids (arxiv 2512.21573), training-free 4D HMR from video (SAM-Body4D), and Gaussian-avatar pipelines where MHR replaces SMPL as the body model (arxiv 2604.01447, which reports higher PSNR on PeopleSnapshot and ZJU-MoCap with a minimal pipeline).
Limitations on hand accuracy vs specialized hand-only methods and on multi-person/contact reasoning are explicitly called out [§Limitations].

Why it’s interesting

The MHR-vs-SMPL decision is the methodologically important part. SMPL/SMPL-X conflate bone lengths and soft-tissue shape into a single PCA-style shape space, which is why downstream avatar pipelines have to layer learned deformations and pose-dependent correctives on top to get visual fidelity — and is why the Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy / ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion line of work keeps hitting a representational ceiling on humans. The early external evidence (arxiv 2604.01447 “Better Rigs, Not Bigger Networks”) is exactly the prediction: with MHR + SAM 3D Body, a minimal avatar pipeline gets the highest reported PSNR on standard avatar benchmarks. For Luma’s character / avatar pipelines this matters concretely — MHR is open-source and commercially licensed, so it can be the body-model layer under any Gaussian-splat or mesh-based avatar stack without the usual SMPL/SMPL-X licensing complications. The sibling release SAM 3D: 3Dfy Anything in Images handles generic objects in the same SAM 3D family, while the upstream segmentation foundation is SAM 3: Segment Anything with Concepts — together they form a stack where a single mask from SAM 3 can prompt either object or body 3D reconstruction. Worth contrasting against EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation and the Audio-Driven Character Animation family, which currently rely on SMPL-X-style rigs; MHR may be the better target.