SAM 3: Segment Anything with Concepts

TL;DR

SAM 3 is Meta’s third Segment Anything foundation model, an 848M-parameter unified detector + memory-based video tracker that introduces Promptable Concept Segmentation (PCS): given a short noun phrase (e.g. “yellow school bus”), an image exemplar, or a combination, the model returns instance masks with stable IDs for every matching object across an image or short (≤30 s) video. SAM 3 roughly doubles cgF1 over open-vocabulary baselines (OWLv2, DINO-X, Gemini 2.5) on the new SA-Co benchmark — which contains 270K unique concepts, ~50× prior open-vocab benchmarks — while preserving SAM 2’s promptable-visual-segmentation capability. The training data is bootstrapped by a hybrid AI+human data engine that produced 4M unique concept labels (including hard negatives). The follow-on SAM 3.1 release (already filed in the wiki) ships an “Object Multiplex” tracker that delivers a ~2× FPS speedup at the same checkpoint scale.

Key claims

PCS is formalized as: given an image or ≤30-second video and a concept prompt (short noun phrase, image exemplars, or both), detect/segment/track every matching instance with stable identities across frames [§2].
SAM 3 doubles the accuracy of existing systems on both image and video PCS while also improving SAM 2’s prior promptable visual segmentation (PVS) capabilities [Abstract].
The architecture is a single shared backbone (Meta Perception Encoder) feeding a DETR-style detector and a SAM 2-derived memory-based tracker, totaling 848M parameters [GitHub README §Model].
A “presence head” decouples recognition (does this concept appear?) from localization (where?), which is what produces the detection-accuracy gain over single-head baselines [Abstract].
The data engine produced 4M unique concept labels including hard negatives — concepts that don’t appear in an image — by combining SAM 3 itself, a Llama-based captioner, and a Llama-3.2v-based AI annotator with humans in the loop [Abstract; §Data Engine].
SA-Co (Segment Anything with Concepts) is released as a benchmark with 270K unique concepts across SA-Co/Gold (image), SA-Co/Silver (image), and SA-Co/VEval (video) splits [GitHub README §SA-Co Dataset].
On SA-Co/Gold image cgF1, SAM 3 reaches 54.1 vs OWLv2 24.6, DINO-X 21.3, Gemini 2.5 13.0; human is 72.8 [Table, GitHub README §Image Results].
On LVIS instance segmentation, SAM 3 hits cgF1 37.2 / AP 48.5; on COCO it hits AP 56.4 [Table, GitHub README §Image Results].
On SA-V video test, SAM 3 reaches cgF1 30.3 / pHOTA 58.0 (human: 53.1 / 70.5); on YT-Temporal-1B test, cgF1 50.8 / pHOTA 69.9 (human 71.2 / 78.4) [Table, GitHub README §Video Results].
Inference cost: ~30 ms per image with 100+ detected objects on an H200 GPU; video latency scales with object count and sustains near real-time for small N [Abstract / §6].
Noun-phrase prompts are global across all video frames, while image-exemplar prompts can be given per-frame as positive/negative boxes to iteratively refine the mask set [§2, Fig. 3].
Concepts are restricted to those expressible as simple noun phrases (noun + optional modifiers); long descriptive sentences are out of scope by design [§2].
Weights are released gated (Hugging Face access request); the SA-Co benchmark and inference code are open-source under the SAM license [GitHub README §Getting Started, §License].

Method

The model has two heads sharing a single Perception-Encoder vision backbone. The detector is a DETR-style transformer conditioned on the prompt — text encoded by a text tower, geometry (boxes/points/masks) as in SAM 2, or image exemplars marked as positive/negative bounding boxes. It is trained for open-vocabulary detection and instance segmentation jointly. The key architectural novelty is the presence head: a learned token that predicts whether the prompted concept is present in the image at all, separately from per-region localization scores. This decoupling reduces the false-positive rate on hard negatives (concepts that don’t appear) and is what most directly produces the doubled cgF1.

The tracker is a memory-based video segmenter inheriting SAM 2’s transformer encoder-decoder. For each detected instance, it maintains a memory across frames and propagates the mask while assigning a stable identity. In the original SAM 3, each tracked object is processed in a separate forward pass; SAM 3.1 (Mar 2026, already in the wiki as SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning) replaces this with Object Multiplex, which jointly processes all tracked objects through shared memory in a single pass for a ~2× FPS gain at the same accuracy.

The data engine is the throughput-critical piece. The team mines candidate concepts from web image+caption pairs using a Llama-based captioner, lets a Llama-3.2v AI annotator propose masks for both positive and hard-negative concepts, and routes uncertain cases to humans. The system is reported as ~5× faster than humans on negative prompts and ~36% faster on positive prompts (see SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning). The result is a training set with 4M unique concepts that includes the long tail well past what hand-labeled benchmarks like LVIS cover.

Results

Image PCS (SA-Co/Gold cgF1): SAM 3 54.1 vs OWLv2 24.6, DINO-X 21.3, Gemini 2.5 13.0; human ceiling 72.8 [GitHub README §Image Results].
Box detection on SA-Co/Gold cgF1: SAM 3 55.7 vs OWLv2 24.5, DINO-X 22.5, Gemini 2.5 14.4 [GitHub README §Image Results].
LVIS instance segmentation: cgF1 37.2 / AP 48.5; COCO box detection: AP 56.4 [GitHub README §Image Results].
Video PCS (SA-V test): cgF1 30.3 / pHOTA 58.0 vs human 53.1 / 70.5 — ~57% of human cgF1 and ~82% of human pHOTA [GitHub README §Video Results].
YT-Temporal-1B test: cgF1 50.8 / pHOTA 69.9 vs human 71.2 / 78.4 [GitHub README §Video Results].
SmartGlasses test: cgF1 36.4 / pHOTA 63.6 (human 58.5 / 72.3) [GitHub README §Video Results].
LVVIS mAP 36.3; BURST HOTA 44.5 [GitHub README §Video Results].
SAM 3 Agent (an MLLM that wraps SAM 3 as a perception tool, iteratively refining noun-phrase queries) beats prior work on ReasonSeg and OmniLabel zero-shot, without any referring-expression or reasoning-segmentation training data [§Results, GitHub README].
Latency: ~30 ms per image with 100+ objects on H200; video sustains near real-time for small object counts in SAM 3, raised to ~32 FPS at 16 objects (H100) by SAM 3.1’s Object Multiplex [Abstract, §6].
Human preference: users prefer SAM 3 outputs over OWLv2 ~3:1 [§Results].

Why it’s interesting

SAM 3 reframes open-vocabulary segmentation as concept prompting — explicit noun-phrase or exemplar prompts that produce all instances with IDs, not the single-instance-per-prompt outputs of SAM 1/2 or the bounding-box detections of OWLv2/DINO-X. For Luma, this matters because the upstream perception primitives consumed by character-animation and mask-driven editing pipelines (Wan-Animate: Unified Character Animation and Replacement with Holistic Replication, OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models, VideoMaMa: Mask-Guided Video Matting via Generative Prior, Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos) are now natively open-vocabulary + tracked + identity-stable, which compresses what used to be a multi-model pipeline. It is also the clearest recent datapoint that the Open foundation-model releases pattern is consolidating to a gated weights + open code + open benchmark + paper stack rather than fully open weights; that constraint matters for downstream finetuning. Finally, the presence-head trick — splitting “does the concept exist here” from “where is it” — is the kind of architectural decoupling that often transfers to other multi-task heads (recognition vs localization, classification vs counting), and is a candidate idea for Unified Multimodal Models designs that currently overload a single head with both.