ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation

TL;DR

ACE-Step v1.5 is an open-weights (MIT) music foundation model from ACE Studio + StepFun that runs a full song in under 2 s on an A100 and under 10 s on an RTX 3090 (≤4 GB VRAM), trained on a deliberately license-clean corpus (licensed + royalty-free + MIDI-synthesized audio). The architecture is a two-stage LM-then-DiT pipeline: a Qwen3-derived LM (0.6B / 1.7B / 4B) acts as a planner that turns short user queries into a Chain-of-Thought “song blueprint” of metadata + lyrics + captions, which then conditions a 2B-parameter DiT decoder (XL variant: 4B) with sliding-window attention and separate encoders for reference audio, lyrics, and instructions. A claimed novelty is intrinsic reinforcement learning — alignment is driven purely from internal model signals rather than an external reward model or human preference data — and the released ecosystem includes base / SFT / turbo (8-step) variants in both 2B and 4B sizes, a Gradio + REST + CLI + VST3 surface, and one-click LoRA training on consumer GPUs.

Key claims

ACE-Step v1.5 generates a full song in under 2 s on an A100 and under 10 s on an RTX 3090, running locally in under 4 GB of VRAM [Abstract].
Reported quality “beyond most commercial music models” on common evaluation metrics — SongEval Coherence 4.72 (XL: 4.79) vs Suno v5 4.72, AudioBox Content-Enjoyment 7.42 (XL: 7.76) vs Suno v5 7.69, and the XL variant tops Suno v5 on Style Alignment (47.9 vs 46.8) and Lyric Alignment (35.8 vs 34.2) [Project page benchmark table].
Hybrid LM+DiT architecture: a planner LM (Qwen3-0.6B/1.7B/4B base) emits metadata + lyrics + captions via Chain-of-Thought before the DiT decoder synthesizes audio [Abstract].
Intrinsic RL alignment that relies solely on the model’s internal mechanisms, framed as eliminating biases inherent in external reward models or human-preference labels [Abstract].
Versatile editing surface from the same checkpoint: cover generation, repainting, vocal-to-BGM conversion, track separation, multi-track “Add Layer” generation, audio understanding (BPM / key / scale / time signature / caption), LRC timestamp generation [GitHub Features].
50+ language coverage for lyric prompts with claimed strict prompt-adherence in both style and lyrics [Abstract].
Lightweight personalization: LoRA training from ~8 songs in ~1 hour on a 3090 with 12 GB VRAM, surfaced as a one-click flow inside the shipped Gradio UI [GitHub Train, GitHub Features].
Training data is deliberately license-clean — licensed catalogues + royalty-free / public-domain music + synthetic MIDI-to-audio data — and the authors explicitly position outputs as commercial-use-safe [HF model card §Safe & Robust Training Data].
Release surface: 3 base DiT variants (acestep-v15-base, -sft, -turbo) at 2B + 3 XL variants at 4B, 3 LM sizes (0.6B/1.7B/4B), MIT license, weights on HF + ModelScope, project page / Gradio / REST API / CLI / VST3 plugin / ComfyUI integration [GitHub Model Zoo, GitHub Documentation].

Method

The system factors music generation into a planner and a renderer. The planner is a Qwen3-derived autoregressive LM (0.6B, 1.7B, or 4B) that takes a short user query (free-text intent, optional tags, optional reference audio understanding) and expands it via Chain-of-Thought into a structured song blueprint: metadata (BPM, key/scale, time signature, duration), full lyrics with section structure, and a sequence of textual captions describing the intended timbre, instrumentation, and mood at song-level granularity. The blueprint then conditions a Diffusion Transformer that produces the actual audio latents.

The DiT decoder operates with sliding-window attention and ingests three separate condition streams: a reference-audio encoder (for cover / repaint / vocal-to-BGM), a lyrics encoder, and an instruction/caption encoder. Acoustic representation has moved beyond v1.0’s mel-spectrogram bottleneck — v1.5 produces audio at 5 Hz token rate (per the acestep-5Hz-lm-* naming convention used for the LM family), enabling both fast generation and longer (up to 10-minute) compositions. Three pretraining/post-training tracks are shipped per backbone size: a pretrain-only base, an SFT-tuned sft, and an 8-step CFG-free turbo distilled variant. The XL family duplicates this structure with a 4B-parameter DiT decoder (~9 GB bf16) for higher audio quality at the cost of ≥20 GB VRAM without offload.

Post-training uses an “intrinsic reinforcement learning” objective: the abstract explicitly contrasts it with external reward models and human-preference data, claiming alignment is driven by signals internal to the model itself (mechanics not detailed in the abstract; full description is in the arXiv technical report).

The release ships VST3 plugin, ComfyUI nodes, MLX backend for Apple Silicon, and a community LoRA-training pipeline (Side-Step) alongside the official one-click Gradio path.

Results

Headline benchmark numbers (from the project page comparison table covering AudioBox sub-axes, SongEval sub-axes, Style Alignment, Lyric Alignment): ACE-Step 1.5 vs Suno v5 — AudioBox-CU 8.09 vs 7.87, SongEval Coherence 4.72 vs 4.72, SongEval Musicality 4.67 vs 4.62, SongEval Memorability 4.72 vs 4.71, SongEval Clarity 4.66 vs 4.63, SongEval Naturalness 4.59 vs 4.56. The XL variant pushes further: AudioBox-CE 7.76 vs 7.69 and Style/Lyric Alignment 47.9/35.8 vs 46.8/34.2 [project page table].
Generation speed: <2 s for a full song on A100, <10 s on RTX 3090; a third-party summary reports a 10–120× speed advantage over comparable open music models [GitHub Features; rits.shanghai.nyu.edu summary].
Memory footprint: runs under 4 GB VRAM in the base 2B configuration with INT8 + CPU offload, scaling up to ≥20 GB for the XL (4B DiT) variant without offload [GitHub GPU-tier table].
Editing capabilities released and benchmarked at parity with synthesis: cover generation, selective repaint, track separation/extraction, vocal-to-BGM (“Lego” mode in the v15-base checkpoint), Complete (continuation) [GitHub Model Zoo capability matrix].
LoRA personalization: ~8 reference songs, ~1 hour of training, 12 GB VRAM, integrated one-click flow [GitHub Train].
Ecosystem: 68 derived models and 62 derived Spaces on Hugging Face citing the paper at the time of filing [HF papers page].

Why it’s interesting

For Luma this is the first OSS music-generation model that lands clearly inside the “commercial-grade on consumer hardware” envelope, and several design choices are worth pulling apart independently of audio:

(1) DiT + sliding-window attention + separate encoders per condition stream (reference audio / lyrics / instructions) is a clean three-way conditioning recipe that maps directly onto problems Luma has in joint-modality generation — it’s a different decomposition than LTX-2’s two-stream DiT (LTX-2: Efficient Joint Audio-Visual Foundation Model) and worth comparing as a conditioning topology even outside audio.

(2) The LM-as-planner + DiT-as-renderer split, with the LM emitting a structured CoT blueprint (metadata + lyrics + captions) that the DiT consumes, is essentially the same “automatic caption / planner” pattern HunyuanImage 3.0 (HunyuanImage 3.0 Technical Report) deploys for image generation — but here the planner is a separately released, separately-scalable artifact (0.6B / 1.7B / 4B), not fused into the generator. That separation makes it a natural reference for any captioning / prompt-rewriting pipeline.

(3) Intrinsic RL is the most genuinely novel claim and the part that demands reading the technical report carefully — if alignment really does come from internal model signals only (no external RM, no preference data), it’s a recipe that could transfer to other generative-model post-training where preference data is expensive or impossible to collect. The abstract claim alone is not enough; the mechanism is the open question.

(4) The release packaging is at the high end of “open foundation release” — base/SFT/turbo × 2B/4B × LM-sizes, MIT license, multi-backend (CUDA/ROCm/MPS/Intel XPU/CPU + vLLM + pt), Gradio + REST + CLI + VST3 + ComfyUI, one-click LoRA training, license-clean training data with explicit commercial-use claim — a useful datapoint for the wiki’s open-foundation-releases tracking.