SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning

TL;DR

SAM 3.1 is a drop-in update to Meta’s SAM 3 (the open-vocabulary detection+segmentation+tracking foundation model with text/exemplar/visual prompts) that replaces SAM 3’s per-object tracking pass with Object Multiplex — a shared-memory approach that processes all tracked objects in a single forward pass. The change preserves accuracy while delivering ~2× throughput at medium object counts (16→32 FPS for 16 objects on H100) and up to ~7× faster inference at 128 objects. Architecture, training data, prompt interface, and checkpoint scale (848M parameters, detector + tracker sharing a Perception-Encoder vision backbone) are unchanged from SAM 3 — the gain comes from removing a concurrency inefficiency that was baked into the original design. Weights are gated on Hugging Face; the SAM 3 paper (arXiv:2511.16719) documents Object Multiplex in Appendix H.

Key claims

SAM 3.1 is a drop-in replacement for SAM 3, preserving the same API and reusing the same model architecture and checkpoint scale — the change is a tracking-time engineering improvement, not a new model [§Update March 27].
Object Multiplex processes all tracked objects together in one forward pass via shared memory, replacing SAM 3’s per-object dedicated pass and eliminating redundant computation and memory bottlenecks [§Update March 27].
Throughput on a single H100 GPU rises from 16 to 32 FPS at medium object counts (≤16 objects) with no reported accuracy loss [§Update March 27].
The underlying SAM 3 model is 848M parameters: a DETR-based detector conditioned on text/geometry/exemplar prompts and a SAM 2-derived tracker, both sharing a Meta Perception Encoder vision backbone [GitHub README §Model].
SAM 3 doubles cgF1 over prior systems on the SA-Co Gold image benchmark and the SA-Co Video benchmark, with the image numbers covering open-vocabulary concept segmentation against OWLv2, DINO-X, Gemini 2.5, and others [GitHub README §Image Results, §Video Results].
SAM 3’s detector uses a “presence token” that improves discrimination between closely related text prompts (e.g. “a player in white” vs “a player in red”) and a decoupled detector–tracker design that minimizes task interference [GitHub README §intro].
The SA-Co benchmark contains 270K unique concepts (>50× prior benchmarks), and the data engine that produced training masks for >4M unique concepts uses a hybrid AI+human annotation loop with Llama-based captioners and Llama-3.2v-based AI annotators (§Building a Novel Data Engine).
SAM 3 Agent: an MLLM that uses SAM 3 as a perception tool, iteratively proposing noun-phrase queries and analyzing returned masks, beats prior work on ReasonSeg and OmniLabel zero-shot — without training on any referring-expression or reasoning-segmentation data [§Results].
Weights are gated (Hugging Face access request required) — not fully open-weight; no native HF Transformers from_pretrained integration, all code lives in facebookresearch/sam3 [GitHub README §Getting Started].

Method

Object Multiplex. The relevant SAM 3 inefficiency is in video: SAM 3 tracks every object with a SAM 2-style masklet, processing each object separately and sharing only the per-frame embeddings. When tracking N objects, the tracker runs N forward passes per frame, each redoing work that doesn’t depend on which object is being tracked. Object Multiplex restructures the tracker so that all N objects share memory and are processed jointly in one pass, with object-level information mixed via the shared representation. The change is described in Appendix H of the SAM 3 paper.

SAM 3 (unchanged in 3.1). A unified DETR-style detector + SAM 2-style tracker share a Meta Perception Encoder backbone (848M parameters total). The detector takes text, geometry (boxes/points/masks), or image exemplars as prompts. A learned “presence token” disambiguates same-category instances under different attribute phrases. The decoupled detector–tracker design is motivated by the task conflict between concept detection (which wants intra-class feature similarity) and re-detection/tracking (which wants intra-class feature distinctiveness).

Data engine. SAM 3, a Llama-based captioner, and Llama-3.2v-based AI annotators jointly mine and verify masks; the system is ~5× faster than humans on negative prompts (concepts not present) and ~36% faster on positive prompts. Together they covered >4M unique concepts across the training set, with a Wikipedia-derived concept ontology used to expand coverage of rare concepts.

Results

Video throughput on H100, single GPU: 16 → 32 FPS at 16 objects (≈2×); up to ~7× speedup at 128 objects per third-party benchmarks of the same checkpoint [external coverage of release; consistent with §Update].
Image accuracy (unchanged from SAM 3): SA-Co/Gold cgF1 = 54.1 (vs OWLv2 24.6, DINO-X 21.3, Gemini 2.5 13.0); LVIS instance-seg cgF1 = 37.2, AP = 48.5; SA-Co/Gold box-detection cgF1 = 55.7; COCO AP = 56.4 [GitHub README §Image Results].
Video accuracy (unchanged from SAM 3): SA-V test cgF1 = 30.3 / pHOTA = 58.0 (human 53.1 / 70.5); YT-Temporal-1B test cgF1 = 50.8 / pHOTA = 69.9 (human 71.2 / 78.4); SmartGlasses cgF1 = 36.4 / pHOTA = 63.6; LVVIS mAP = 36.3; BURST HOTA = 44.5 [GitHub README §Video Results].
Latency: ~30ms per image with >100 objects on an H200; video sustains near real-time for ~5 concurrent objects under SAM 3, raised substantially by Object Multiplex [§Results].
User preference: users prefer SAM 3 outputs over OWLv2 (the strongest prior baseline) ~3:1 [§Results].

Why it’s interesting

The pattern of shipping a “drop-in” architectural fix as a versioned model release — same checkpoint scale, same API, no retraining, but a substantial production-relevant speedup — is itself notable, and connects this release to Open foundation-model releases (it’s a coordinated release: HF weights + GitHub code + paper appendix + Segment Anything Playground demo, with the update shipped as a labeled checkpoint rather than as a code-only patch). Compared to other open-foundation releases tracked at Luma, SAM 3.1 sits in the partially-open category: weights are gated, no native HF Transformers integration, which contrasts with the fully-open Qwen3-VL-Embedding-8B and DeepSeek-OCR 2: Visual Causal Flow releases. For Luma’s video work, the relevant primitive is that SAM 3 + multiplexed tracking is now a credible upstream perception tool for character/instance-mask pipelines — the same upstream role that motivates Wan-Animate: Unified Character Animation and Replacement with Holistic Replication‘s use of segmentation, OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models‘s mask-driven omnimatte, and VideoMaMa: Mask-Guided Video Matting via Generative Prior‘s mask-conditioned matting — all of which currently depend on SAM-family masks as input.