Stable Audio 3

TL;DR

Stable Audio 3 is a family of latent diffusion models (small, medium, large) for variable-length text-to-audio generation and editing from Stability AI. The architecture sits on top of a novel semantic-acoustic autoencoder that projects audio into a compact latent that preserves both fidelity and semantic structure, with adversarial post-training applied to the diffusion stage to reduce sampling steps while improving quality. The models generate music and sound effects of variable length (the medium and large variants exceed six minutes per the companion launch blog), train only on licensed and Creative Commons data, and reach <2 s generation on an H200 and a few seconds on a MacBook Pro M4. The small and medium checkpoints are released under open weights.

Key claims

Stable Audio 3 is shipped as a family of fast latent diffusion models — small, medium, large — for variable-length audio generation and editing [Abstract].
Variable-length generation is the central design driver: producing audio of a target duration rather than always producing full-length and cropping avoids wasted compute on short sounds [Abstract].
The models support inpainting, which the authors frame both as targeted audio editing and as continuation of short recordings [Abstract].
A novel semantic-acoustic autoencoder underlies the system; the latent is compact for diffusion efficiency while preserving fidelity and encouraging semantic structure in the latent space [Abstract].
Adversarial post-training is used to both accelerate inference and improve generation quality — reducing inference steps while improving fidelity and prompt adherence [Abstract].
Training data is restricted to licensed and Creative Commons sources [Abstract].
Generation latency is under 2 s for music/sounds on an H200 GPU, and a few seconds on a MacBook Pro M4 [Abstract].
The small and medium variant weights are released alongside the training and inference pipeline; large is held back as a hosted/enterprise tier per the companion launch blog [Abstract].

Method

The architecture is latent diffusion in audio: a semantic-acoustic autoencoder compresses raw audio (24 kHz / 44.1 kHz class) into a compact latent that is both acoustically reconstructive and semantically organized, and a diffusion backbone then generates in this latent space, with the decoder mapping back to waveform. The “semantic” claim is the defining latent-design choice — most prior latent audio diffusion models train the autoencoder for reconstruction fidelity only, which makes the latent harder to condition on text and harder to organize for inpainting. The Stable Audio 3 autoencoder is explicitly trained to balance compactness, fidelity, and semantic structure, with the goal of producing a latent that supports diffusion-based generation with good text adherence.

Variable-length generation is built into the model rather than emulated at inference by post-hoc cropping or stitching — the launch blog says generations can be produced at per-second granularity, with small reaching ~2 minutes, medium/large reaching beyond six minutes (Meet Stable Audio 3.0 — the model family for artistic experimentation with open-weight models). Inpainting is supported as a single capability that subsumes both interior segment editing and causal continuation of an existing clip.

Post-training is adversarial: a discriminator-driven refinement of the pretrained diffusion model that simultaneously reduces the number of inference steps required and improves both fidelity and prompt adherence. This is the same headline mechanism that diffusion-distillation methods on the image/video side use, applied to audio — the abstract is explicit that adversarial post-training does both jobs at once (acceleration + quality).

Results

The abstract reports two operational metrics directly:

Latency: under 2 s to generate music or sounds on an H200 GPU [Abstract].
Latency on consumer hardware: a few seconds on a MacBook Pro M4 [Abstract].

The companion launch blog (Meet Stable Audio 3.0 — the model family for artistic experimentation with open-weight models) reports duration capability: small → up to ~2 minutes (vs 11 s for Stable Audio Open Small and 47 s for Stable Audio Open), medium and large → beyond ~6:20. No headline benchmark comparison vs Suno / Udio / ACE-Step is in the abstract itself; the technical report is the source for quantitative quality numbers.

The release scopes commercial deployment by license tier: open-weights small + medium under Stability AI Community License (free commercial use under a revenue threshold), large via the Stability API and enterprise licensing per the launch blog (Meet Stable Audio 3.0 — the model family for artistic experimentation with open-weight models).

Why it’s interesting

For Luma this completes a triple in the wiki’s audio-generation coverage — Stable Audio 3 joins ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation (open music generation, planner-LM + DiT pipeline) and LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space (waveform-latent diffusion TTS) — and the three designs disagree informatively. ACE-Step factors generation into an LM planner + DiT renderer with a 5 Hz latent token rate. LongCat-AudioDiT collapses to a single waveform Wav-VAE + diffusion backbone, no intermediate mel. Stable Audio 3 sits in between: a single semantic-acoustic autoencoder + latent diffusion, but the autoencoder is deliberately trained to be semantic (not just reconstructive), which is the design lever neither sibling pulls explicitly. The “compact + fidelity + semantic” three-way constraint on the autoencoder is the part most worth pulling out for general latent-design discussions; it’s the same trilemma latent image AEs face (What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion), and Stable Audio 3 is one of the first audio-side answers shipped with open weights.

Adversarial post-training as a combined accelerator + quality fix is also worth noting alongside the broader Diffusion Distillation thread: most distillation methods on the wiki frame step-reduction and quality as competing objectives (DP-DMD, RMD, DiagDistill), while Stable Audio 3 claims its adversarial pass moves both axes in the same direction. Whether that holds against a careful DMD baseline is an open question, but it’s a useful counterexample to file against the “fewer-NFE always costs quality” prior.