Introducing SAM Audio: The First Unified Multimodal Model for Audio Separation

TL;DR

SAM Audio extends Meta’s Segment Anything family to audio: a flow-matching diffusion transformer that takes an audio mixture plus one or more prompts (natural-language text, a click on a visual object in a paired video, or a time-span marker) and emits two waveforms — target (the isolated sound) and residual (everything else). It’s released alongside Perception Encoder Audiovisual (PE-AV), a contrastive video–audio encoder trained on 100M+ videos that powers both SAM Audio’s visual prompting and the SAM Audio Judge reference-free quality model; plus SAM Audio-Bench, an in-the-wild benchmark with human-drawn masks, time markers, and text descriptions per 10s sample. Three model sizes ship (small/base/large, 500M–3B params), all faster than real-time (RTF ≈ 0.7 on A100), with the largest hitting Instr(pro) subjective score 4.49/5. The “industry-first” span-prompt mode is the main novelty over text/visual-prompted separation tools.

Key claims

SAM Audio is positioned as the first unified model supporting text, visual, and span prompts (alone or combined) for audio separation across speech, music, and general sound [§A Unified, Multimodal Prompting Model For Segmenting Audio].
The generator is a flow-matching diffusion transformer operating on DAC-VAE latent audio tokens; the prompt encoders (audio mixture, text, span, and visual-from-video-plus-mask) are concatenated into time-aligned features that condition the DiT via self- and cross-attention [§Model Architecture].
The data engine combines real audio mixtures, advanced synthetic-mixing strategies, automated multimodal prompt generation, and a pseudo-labeling pipeline to produce large-scale training data — explicitly framed as solving the supervised-separation data shortage [§Model Architecture].
PE-AV is built on the open-source Perception Encoder (April 2025 release) and extends it to audio by aligning frame-level video features with audio representations at precise timestamps; trained on 100M+ videos with multimodal contrastive learning [§A Unified, Multimodal Prompting Model For Segmenting Audio].
SAM Audio Judge is a reference-free perceptual quality metric trained against nine human-defined dimensions (recall, precision, faithfulness, overall quality, …) on a 5-point scale, using a transformer over advanced audio + text encoders and a pre-training strategy for text–audio prompt matching [§SAM Audio Judge].
SAM Audio-Bench provides 10-second samples spanning speech, music, and general sound effects, each annotated with text descriptions, human-drawn visual masks, and time spans — supporting reference-free evaluation via SAM Audio Judge + human listening tests [§SAM Audio-Bench].
Subjective scores (MOS-style, across General / SFX / Speech / Speaker / Music / Instr(wild) / Instr(pro)) include 3.62 General for sam-audio-small, 3.28 base, 3.50 large; Instr(pro) reaches 4.49 for sam-audio-large [reported in Meta blog, see external coverage].
Mixed-modality prompting (e.g. text + span) outperforms any single-modality prompt — explicit “stronger outcomes than single-modality approaches” claim [§Results].
Inference operates faster than real-time (RTF ≈ 0.7 on A100) across the 500M–3B parameter range [§Results].
Stated limitations: audio-as-prompt is not supported; complete audio separation without any prompt is out of scope; separation between highly similar audio events (one singer from a chorus, one instrument from an orchestra) remains a challenge [§Results].
Release package: SAM Audio + PE-AV + SAM Audio Judge + SAM Audio-Bench, with two accompanying research papers, weights gated on Hugging Face, and an in-browser demo at the Segment Anything Playground [§Looking Forward].
Accessibility partnerships are named at launch — Starkey (largest US hearing-aid manufacturer) and 2gether-International — explicitly framed as exploring SAM Audio integration into hearing-aid pipelines [§Looking Forward].

Method

SAM Audio is a generative separation model rather than a discriminative one. A pretrained DAC-VAE encoder compresses the input audio mixture to a sequence of latent tokens. Prompt-side, the system runs one encoder per modality: an audio encoder over the mixture, a text encoder over the natural-language prompt, a span encoder over time-range markers, and a visual encoder that consumes video frames plus an object mask (or click-derived mask). Encoded prompt streams are concatenated into a single time-aligned feature stream that conditions a flow-matching diffusion transformer; the DiT runs self-attention over the time-aligned representation and cross-attention to the textual feature, then the DAC-VAE decoder reconstructs the target waveform. The residual waveform is the difference between the input mixture and the target.

PE-AV is the visual-and-audio backbone used at two points: as the visual prompt encoder inside SAM Audio (extracting time-stamped per-frame features from video) and as the perception model inside SAM Audio Judge. It’s built by extending Meta’s April-2025 Perception Encoder — the same backbone repurposed for object detection in SAM 3 — with audio alignment. Training uses large-scale multimodal contrastive learning over 100M+ videos, with PyTorchVideo for video processing and FAISS for semantic-search-based data curation.

SAM Audio Judge replaces traditional reference-based separation metrics (SI-SDR vs. clean reference stem) with a reference-free perceptual model. The team defined nine perceptual dimensions, collected human ratings on a 5-point scale, and trained a transformer over audio + text encoders with a pre-training objective that teaches it to score whether the separated output matches the text prompt. This is what makes SAM Audio-Bench possible in-the-wild — most real recordings have no clean reference stem to compare against.

Results

Subjective scores (MOS-style, 5-point, reported per category): sam-audio-large reaches 4.49 on Instr(pro) [music-instrument professional]; General scores 3.62 (small), 3.28 (base), 3.50 (large). The full table covers seven categories: General, SFX, Speech, Speaker, Music, Instr(wild), Instr(pro) [results reported in Meta blog; see external coverage].
Inference: RTF ≈ 0.7 on a single A100, across model sizes from 500M to 3B parameters [§Results].
Universal vs domain-specific baselines: SAM Audio is claimed to match the best domain-specific models across speech, music, and general sound while being a single unified model [§Results].
Prompt-mode ablation: mixed-modality prompts (e.g. text + span) deliver “stronger outcomes than single-modality approaches” — the blog does not give the delta but flags it as a headline finding [§Results].
Reference-free judge: SAM Audio Judge is used as the primary metric on SAM Audio-Bench in addition to human listening tests; the page argues this is necessary because most in-the-wild recordings lack clean reference stems [§SAM Audio-Bench].
Coverage: SAM Audio-Bench combines real audio/video from “a variety of high-quality sources,” each 10s sample annotated with text, visual mask, and span [§SAM Audio-Bench].

Why it’s interesting

SAM Audio is the audio counterpart to SAM 3: Segment Anything with Concepts — same release pattern (gated weights + open code + open benchmark + paper), same prompt-driven framing (“segment anything you can point at, describe, or mark in time”), now extended into the time-domain audio modality. For Luma it’s relevant on two fronts: as a generic audio-cleanup primitive that could plug into a video-generation pipeline downstream of synthesis, and as a target/residual interface that’s directly compatible with the kind of operations (remove a barking dog from a podcast, extract a guitar stem) that production pipelines actually need. It complements LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space (a flow-matching DiT for waveform-domain text-to-speech) in showing that flow-matching DiTs on DAC-VAE latents are now the default architecture for serious audio work — separation here, synthesis there. It also sits alongside UniVerse-1: Unified Audio-Video Generation via Stitching of Experts and Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation (joint A/V generation) as the analytical counterpart: where those generate audio conditioned on video, SAM Audio decomposes a mixed audio track conditioned on the video. The PE-AV backbone — Perception Encoder reused across SAM 3 (vision) and SAM Audio (audio) — is the more interesting datapoint for the Unified Multimodal Models thread: a single contrastive encoder is now load-bearing across two SAM releases in two modalities.