SAM 3D: 3Dfy Anything in Images

TL;DR

SAM 3D Objects is Meta’s generative foundation model for single-image visually-grounded 3D object reconstruction — given an RGB image and an object mask, it predicts full 3D geometry, texture, and scene layout in a few seconds. The key contribution is breaking the 3D data barrier: a model-in-the-loop annotation engine in which human raters score/verify meshes generated by a stable of models, routing hard examples to expert 3D artists, was used to label ~1M natural images with ~3.14M meshes — orders of magnitude beyond existing 3D corpora. The model is trained in an LLM-style two-stage recipe (synthetic-asset pre-training, then real-image post-training on the engine’s output), and beats prior single-image 3D methods at ≥5:1 win rate in human preference tests. Released alongside SAM 3D Body and the new SA-3DAO benchmark for in-the-wild 3D reconstruction.

Key claims

The task is visually grounded single-image 3D reconstruction: predict full 3D shape (not 2.5D), texture, and pose relative to the camera, including occluded back-surfaces, for an object selected by a 2D mask in a natural image [§1, Fig. 1].
The data bottleneck for 3D is structural — high-quality 3D ground truth requires expert 3D artists, multiple orders of magnitude slower to produce than text/image labels — and the team’s key insight is that verifying/ranking meshes is a more accessible skill than authoring them, which is what makes a model-in-the-loop data engine viable [§Approach / Meta blog].
The data engine annotates ~1M distinct physical-world images with shape/texture/layout, producing ~3.14M model-in-the-loop meshes; expert 3D artists fill in blindspots routed by the rater pipeline [§Data Engine].
Training is recast as an LLM-style two-stage recipe: synthetic pre-training on isolated 3D assets, then real-image post-training (alignment) on data-engine-generated meshes from natural images, with a positive feedback loop where model improvements yield better data and vice versa [§Training].
SAM 3D Objects beats prior single-image 3D methods at ≥5:1 win rate in head-to-head human preference tests on real-world objects and scenes [Abstract; §Results].
Full textured reconstructions return within a few seconds via diffusion shortcuts and engineering optimizations, enabling near real-time uses (e.g. robotics perception modules) [§Results / Meta blog].
A new evaluation benchmark — SAM 3D Artist Objects (SA-3DAO) — pairs natural images with artist-authored object meshes; significantly more challenging than prior staged-/synthetic-asset benchmarks like CO3D, OmniObject3D [§Benchmark].
Known limitations: moderate output resolution loses detail on complex objects (e.g. full-person reconstructions distort); the model predicts one object at a time without joint reasoning about contact/interpenetration between objects [§Limitations].
Release package: weights, inference code, online demo (Segment Anything Playground), and the SA-3DAO benchmark; the encoder is DINOv2-based and is being added to HuggingFace Transformers [GitHub README; HF model card].

Method

The pipeline takes an RGB image plus an object mask (typically obtained from SAM 3) and outputs a textured 3D mesh plus the object’s layout relative to the camera. The backbone is a DINOv2-based vision encoder; the head is a generative network that predicts geometry, texture, and pose. The training recipe has two stages. Pre-training uses a large pool of isolated synthetic 3D assets — the standard regime that prior single-image 3D models lived in — and gives the model a starting point on shape and texture priors. Post-training (alignment) is the new step: a corpus of natural images annotated by the data engine, where each image+mask pair has a verified or expert-authored mesh, is used to close the sim-to-real gap. The data engine works as follows: a stable of candidate 3D models proposes multiple meshes for each image, human raters pick or score the best, and uncertain or out-of-distribution cases get routed to expert 3D artists. As model quality improves, the engine produces better candidates, which produces better post-training data, which produces a better model — the explicit positive feedback loop the paper calls breaking the 3D data barrier. At inference time, diffusion shortcuts (skipping/reusing computation in the generator) keep latency in the few-seconds regime.

Results

Human preference: ≥5:1 win rate over prior single-image 3D reconstruction methods on real-world objects and scenes [Abstract; §Results].
Inference latency: full textured 3D reconstruction returns within a few seconds on a single GPU [Meta blog].
New benchmark SA-3DAO is positioned as substantially harder than existing 3D-reconstruction benchmarks because images are natural (cluttered, occluded) and meshes are artist-authored rather than synthetic [§Benchmark, Meta blog].
Encoder variants supported: ViT-S/14 (384), ViT-B/14 (768), ViT-L/14 (1024), ViT-G/14 — selectable via the HuggingFace integration [GitHub Issue #42464].
Limitations quantified qualitatively only at filing time: distortion on full-body reconstructions, no joint multi-object physical reasoning [§Limitations].

Why it’s interesting

This is the clearest existence proof so far that the LLM playbook — pre-train on cheap synthetic, post-train on expensive but high-value real annotations, harvest more real data with the model in the loop — transfers to 3D. The leverage point isn’t a new architecture; it’s framing 3D annotation as a verification/ranking task rather than an authoring task, which is what makes the data engine work. That recipe complements SAM 3: Segment Anything with Concepts (which uses essentially the same hybrid AI+human data engine for 4M concept labels in 2D) and inherits the same release pattern flagged in Open foundation-model releases — gated weights + open code + open benchmark + paper, with the segmentation/3D models composed into a single playground demo. For Luma the immediate read is on the upstream perception stack: single-image, mask-conditioned 3D meshes that return in seconds are the right primitive for asset-creation pipelines and for the “Reshoot” target WonderJourney: Going from Anywhere to Everywhere sketches — though the moderate resolution and one-object-at-a-time limitations mean it isn’t yet a drop-in for scene-level work. Worth watching: how does SAM 3D Objects compare against Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material head-to-head, since both target single-image → textured 3D mesh from very different design points (Hunyuan3D 2.1 is asset-style + PBR materials; SAM 3D Objects is natural-image + visually grounded).