sam3-onnx — ONNX export of Meta's SAM 3 for offline CUDA and CoreML inference

TL;DR

sam3-onnx is an MIT-licensed community tool that exports Meta’s SAM 3 (filed as SAM 3: Segment Anything with Concepts) to ONNX so it runs offline on NVIDIA CUDA or Apple CoreML, with a sibling CLI (osam) that packages installation and inference behind one pip install. The repo splits SAM 3 into four ONNX graphs — image encoder, language encoder, decoder, point decoder — and reports a text-prompt latency of 0.37 s on an M5 MacBook Pro (CoreML) and 0.08 s on an RTX 3080 Ti versus 1.43 s on M5 CPU, i.e. a 3.8× CoreML speedup over CPU on the same chip. It also integrates SAM3-LiteText, a MobileCLIP-distilled language encoder variant for lighter deployment. The weights keep Meta’s gated SAM license; the code and export scripts are MIT.

Key claims

Four exported ONNX graphs implement SAM 3 end-to-end: image encoder, language encoder, decoder, and a separate point decoder, with a Hugging Face mirror (wkentaro/sam3-onnx-models-v0.5.0) for pre-exported weights [README §Pre-exported ONNX models].
Prompt modes supported: text noun phrase, box prompt (positive or negative, repeatable), click/point prompt (positive or negative, repeatable), and combinations — e.g. text “dog” plus a negative box to exclude one instance [README §Usage].
Text-prompt latency, image already encoded (median of 5 runs, no model-load): M5 CPU 1.43 s, M5 CoreML 0.37 s, RTX 3080 Ti CUDA 0.08 s — a 3.8× CoreML-over-CPU speedup on the same Apple chip [tweet / README §Inference with onnx on Apple GPU].
Image encoding on CoreML runs ~5× faster than CPU on an M5 (3 s vs 15 s); the language encoder, decoder, and point decoder stay on CPU and together take <2 s [README §Inference with onnx on Apple GPU].
SAM3-LiteText swaps the SAM 3 language tower for MobileCLIP-S0/S1/L distilled encoders; the LiteText language-encoder input is a 1x16 int64 token tensor, and the decoder is unchanged so existing exports stay compatible via zero-padded 32-position outputs [README §SAM3-LiteText].
onnxruntime-gpu is installed into a separate uv environment (UV_PROJECT_ENVIRONMENT=.venv-gpu) to avoid file collisions with onnxruntime and with Torch’s cuDNN [README §Inference with onnx on GPU].
A parity test (test_text_parity.py) checks that PIL preprocessing, image/language features, and decoder scores stay within 0.01 of PyTorch on the same prompts, isolating decoder errors by feeding PyTorch features to the ONNX decoder [README §Check text-prompt scores against PyTorch on CPU].
The companion runner osam (also by the author) packages the full pipeline as pip install 'osam[cpu]' / osam[gpu] with osam run sam3 --image … --text … --device coreml as the one-liner [tweet / README of sibling repo].
Weights are 3.6 GB; downloaded once from Hugging Face then run offline. Code is MIT; weights remain under Meta’s gated SAM license [tweet, README §License-equivalent note].
MobileCLIP weights used by SAM3-LiteText count as derivatives of Apple’s ml-mobileclip model license, which restricts to non-commercial research — flagged in the README [README §SAM3-LiteText].

Method

sam3-onnx re-expresses the SAM 3 inference graph (released by Meta as SAM 3: Segment Anything with Concepts) as four decoupled ONNX subgraphs so each can be routed to the best execution provider. export_onnx.py traces the PyTorch SAM 3 modules — Perception-Encoder image backbone, text tower, DETR-style detector, and SAM 2-derived point/mask decoder — individually, producing sam3_image_encoder.onnx, sam3_language_encoder.onnx, sam3_decoder.onnx, and sam3_point_decoder.onnx. At inference time the image encoder is run once per image (and can be sent to CoreML/CUDA), the language encoder is run once per text prompt, and the decoder is run per prompt against the cached image features — matching the prompt-reuse pattern SAM 3 was designed for. Box and point prompts bypass the language encoder entirely and go through the point decoder.

The SAM3-LiteText variant adds a distilled MobileCLIP language encoder as a drop-in replacement (sam3_litetext_<s0|s1|l>_language_encoder.onnx). To keep decoder compatibility with the full-model export, LiteText truncates to 16 CLIP tokens on input, then zero-pads its 32-position memory/embeds outputs and masks the padded tail — so decoders already exported for the full language encoder consume the LiteText output without re-export. The repo’s benchmark harness (docs/benchmark.md) is what produced the quoted 0.37 s / 0.08 s text-prompt numbers; latency is reported with the image already encoded, which is the realistic case when prompting multiple concepts against the same image.

Results

Text prompt, image pre-encoded, median of 5 runs: M5 CPU 1.43 s, M5 CoreML 0.37 s, RTX 3080 Ti CUDA 0.08 s [tweet, README §benchmark link].
Image encoder on M5: CoreML ~3 s vs CPU ~15 s (~5× speedup) [README §Inference with onnx on Apple GPU].
Weights footprint: 3.6 GB total for the four ONNX graphs; downloaded once, reusable offline [tweet].
Prompt types validated: single text, single click, text + negative box, multiple exemplar boxes with per-box positive/negative labels, box selection via a GUI picker [README §Examples / §Usage].
Parity test: ONNX and PyTorch pipelines return the same detection count and sorted scores within 0.01 on the apple test image and bus.jpg with “person”; a one-level uint8 difference from PIL vs PyTorch resize is documented as the only remaining source of drift [README §Check text-prompt scores against PyTorch on CPU].

Why it’s interesting

This is the first viable offline-on-laptop deployment path for SAM 3 (SAM 3: Segment Anything with Concepts) and complements SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning on the opposite axis: SAM 3.1’s Object Multiplex optimizes server-side video FPS, while sam3-onnx hits the Apple-silicon / consumer-GPU inference loop that annotation tools and interactive apps need. For Luma, the pattern matters because mask-driven editing and character-animation pipelines (Wan-Animate: Unified Character Animation and Replacement with Holistic Replication, OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models, VideoMaMa: Mask-Guided Video Matting via Generative Prior) already assume concept-prompted, identity-stable masks as an upstream primitive — sam3-onnx removes the cloud round-trip from that primitive, with sub-100 ms per-prompt latency on a 3080 Ti. It is also a natural companion to EdgeTAM: On-Device Track Anything Model, which pursues the same offline-on-device goal for SAM 2 by attacking the memory-attention cost rather than exporting to ONNX — a useful contrast as the SAM family bifurcates between “export the stock graph” and “redesign for edge.” The LiteText branch (MobileCLIP-distilled language encoder) is also a concrete sub-threshold data point for a distilled-encoder pattern worth watching separately from SAM-specific releases.