Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing

TL;DR

Audio-Omni is an end-to-end unified framework that handles audio understanding, generation, and editing across all three audio sub-domains — general sound, music, and speech — within a single model. The architecture decouples a frozen Qwen2.5-Omni-3B MLLM (reasoning) from a trainable 3.05B-parameter DiT (synthesis), bridged by a hybrid conditioning split: high-level semantic features (MLLM hidden states + transcript) via cross-attention, and low-level signal features (mel-spectrogram + video-sync via Synchformer) via channel concatenation. The team also releases AudioEdit, a >1M-pair instruction-guided audio editing dataset built by a hybrid real-mining + programmatic-synthesis pipeline. Audio-Omni matches or beats specialist models on T2A / T2M / V2A / V2M / TTS / editing and inherits multilingual zero-shot generation, in-context generation, and knowledge-augmented generation from the frozen MLLM.

Key claims

Existing “unified” audio models cover only sub-domains (speech-only FunAudioLLM / Ming-Omni, music-only MuMu-LLaMA) or rely on tool-orchestration (AudioGPT) that lacks end-to-end optimization — no prior model unifies understanding + generation + editing across general sound, music, and speech simultaneously [§1, §2].
The architecture is a frozen-MLLM + trainable-DiT decouple: Qwen2.5-Omni-3B is kept frozen as the reasoning core, and a 36-block / 2048-dim / 32-head DiT with ~7.9B total / 3.05B trainable parameters does the actual synthesis under a Rectified Flow objective [§4.2, §5.1].
The penultimate (second-to-last) hidden state of the frozen MLLM is used as the conditioning feature for the DiT, empirically chosen over the final hidden state for being a more generalizable representation for downstream generation [§4.2].
Conditioning is split into two streams with different injection mechanisms: High-Level Semantic Features (MLLM hidden state ⊕ transcript-encoder output) enter via cross-attention; Low-Level Signal Features (mel-spectrogram ⊕ Synchformer video-sync features) are added to the time embedding and then channel-concatenated with the noisy VAE latent input [§4.2, Eqs. 3-4].
AudioEdit dataset: >1.1M training samples + 2K test samples covering add / remove / extract / style-transfer tasks, built via a hybrid pipeline — real-data branch (Gemini 2.5 Pro for category labeling + SAM-Audio for source separation + VAD + CLAP filtering on VGGSound, yielding 50K pairs at ~9.2% retention with ~83% human agreement) and synthesis branch (Scaper toolkit mixing ESC-50 foreground events into AudioCaps backgrounds with randomized SNR / pitch / time-stretch) [§3, Table 1].
For style-transfer specifically, Gemini is used to expand each filtered target with semantically-related-but-different keywords, then ZETA performs the style transformation while preserving temporal structure and pitch, then the result is remixed with the residual track — yielding ~500K real-data style-transfer pairs [§3].
Generation results match or beat specialists: T2A FAD 1.86 (= AudioX), T2M FAD 1.94 (beats Stable-Audio-Open 3.23 and MusicGen 3.94), V2A FAD 1.71 (beats MMAudio 2.04 and VATT 2.55), V2M FAD 1.58 (beats AudioX 2.12 and VidMuse 2.46), TTS WER 1.77 (beats F5-TTS 1.83, CosyVoice3 2.46, and Ming-Omni 4.31) [Table 3].
Audio editing: Audio-Omni beats ZETA / SDEdit / MMEDIT across the board on the AudioEdit test set — FAD 3.27, LSD 2.27, CLAP 0.32 vs ZETA 3.81 / 3.80 / 0.30, SDEdit 3.51 / 4.40 / 0.22, MMEDIT 3.95 / 4.05 / 0.15 [Table 4].
Understanding inherited from the frozen MLLM matches or approaches dedicated specialists: 56.83 on MMSU and 63.30 on MMAU, beating all other unified models (Ming-Omni 47.53 / 70.80, Unified-IO2 30.74 / 3.20, MuMu-LLaMA 6.32 / 12.87) [Table 2].
Mixed real + synthetic editing data outperforms either alone (KL 1.30 / FAD 2.48 / LSD 1.82 vs Only-Real KL 1.27 / FAD 2.67 / LSD 1.84 vs Only-Synthesis KL 1.93 / FAD 3.80 / LSD 5.17) — synthetic provides broad operation coverage, real provides acoustic fidelity [Table 5].
Conditioning-strategy ablation: the best configuration is mm + transcript as cross-attention Context, with sync + mel as channel concatenation — putting everything as context yields the worst result (T2A FID 40.91 vs 28.90 for the optimal split) and putting only mm as context with the rest concatenated is also worse (T2A FID 60.58) [Table 6].
Zero-shot cross-lingual T2A: Audio-Omni trained only on English maintains strong T2A quality on CN / ES / DE / FR / JP prompts via the multilingual understanding inherited from the frozen MLLM, comparable to English-only specialists [§5.3.2, Appendix Table A2].
Inherited emergent abilities: (a) knowledge-augmented generation — given “music using the instrument Jimi Hendrix played in a happy mood” it infers electric guitar; (b) in-context generation — given a piano recording + “dramatic cinematic chord progression” it transfers piano timbre to the new piece; (c) the speech-prompt masking strategy (20-75% mel-spectrogram masking during training, after F5-TTS) yields zero-shot voice conversion and speech editing at inference time without task-specific supervision [§5.3.3].
Training scale: ~80K steps at batch size 5120, AdamW with lr 5e-5 / β₁=0.9 / β₂=0.95 / wd 1e-3; inference uses 100 ODE solver steps with CFG scale 6.0 [§5.1].

Method

The framework follows a decoupled understanding-and-generation design. A frozen Qwen2.5-Omni-3B MLLM ingests textual instructions, audio waveforms (16 kHz), and video (5 fps) tokenized by their respective encoders; for understanding tasks it emits text directly, and for generative tasks its second-to-last hidden state is exposed as a conditioning signal f_mm for the DiT. A trainable DiT backbone (36 blocks, hidden 2048, 32 heads, ~3.05B trainable parameters out of ~7.9B total) is trained from a Rectified Flow objective — linear interpolation x_t = (1-t)x₀ + t·noise and an MSE loss on the predicted velocity field — and produces a VAE-latent audio output decoded by the pretrained Stable-Audio VAE.

The novel conditioning split is the architectural core. High-level features (MLLM hidden state ⊕ a ConvNeXtV2 transcript encoder) enter via cross-attention as flexible context the model can re-query at every step; low-level features (mel-spectrogram via a 100-dim mel encoder for editing/voice-conversion references ⊕ Synchformer features extracted from 25-fps video) are added to the time embedding and then channel-concatenated with the noisy VAE latent — giving frame-by-frame precise alignment for editing and audio-visual sync tasks. The ablation in Table 6 directly demonstrates that this split is the right one: putting everything as context, or moving the transcript out of context, both significantly hurt T2A / V2A / TTS / AE metrics.

The AudioEdit dataset is the secondary contribution and the prerequisite for the editing capability. The pipeline mixes (a) a Real Data Branch that mines VGGSound — Gemini 2.5 Pro labels primary sound-emitting object categories, SAM-Audio separates them into target + residual tracks, then VAD + CLAP filtering keeps ~9.2% of candidates as 50K high-fidelity source-target pairs per task; style-transfer is then expanded via Gemini-generated keyword variants + ZETA-based transformation to ~500K pairs — and (b) a Synthesis Data Branch that uses Scaper to programmatically mix ESC-50 foreground events into AudioCaps backgrounds with randomized onset / SNR / pitch / time-stretch, yielding 450K synthetic pairs. The mixed-data ablation (Table 5) shows the real branch dominates fidelity (LSD 1.84 vs 5.17 for synthesis-only) while the synthesis branch broadens operation coverage; using both yields the best of both.

Results

Across understanding, generation, and editing benchmarks Audio-Omni is the only unified model that competes with dedicated specialists in every domain. On generation it ties or beats specialists on T2A (FAD 1.86), V2A (FAD 1.71 vs MMAudio 2.04), V2M (FAD 1.58 vs VidMuse 2.46), and TTS (WER 1.77 vs F5-TTS 1.83); on T2M (FAD 1.94) it beats Stable-Audio-Open (3.23) and MusicGen (3.94) but trails the specialist AudioX (1.53) [Table 3]. On editing it sweeps all three metrics — FAD 3.27 / LSD 2.27 / CLAP 0.32 — vs the next-best ZETA at 3.81 / 3.80 / 0.30 [Table 4]. On understanding it scores MMSU 56.83 / MMAU 63.30, matching the score of its own frozen-MLLM backbone Qwen2.5-Omni-3B (53.27 → 56.83 / 57.40 → 63.30 — i.e. Audio-Omni equals the backbone in this evaluation, confirming the frozen-MLLM design doesn’t degrade understanding) [Table 2]. Cross-lingual zero-shot T2A on CN / ES / DE / FR / JP is reported as comparable to English-only specialists despite English-only training data [Appendix Table A2].

Why it’s interesting

Audio-Omni is the audio counterpart to the visual UMM lineage tracked under Unified Multimodal Models — it adopts the same Decoupled recipe (frozen MLLM + trainable DiT-with-flow-matching, the same family as DuoGen / OmniWeaving / OpenUni / MetaQuery) but applied to audio, and is the first filed paper that unifies three audio domains (sound + music + speech) under one model rather than the speech-only or music-only sub-unifications that previously dominated the audio side. The penultimate-layer-feature choice and the two-stream conditioning split (high-level via cross-attention, low-level via channel-concat) are a clean engineering recipe that complements DuoGen: Towards General Purpose Interleaved Multimodal Generation‘s “freeze MLLM, train only connector + DiT” finding from the image-output side. It also intersects Joint audio-video generation as a V2A model (FAD 1.71 vs MMAudio 2.04) — though, unlike LTX-2 / MOVA / SkyReels-V4, it is not a joint A+V generator, treating V2A as one of many conditional-generation tasks rather than learning a joint A+V distribution. The AudioEdit dataset is the field’s first 1M-scale instruction-guided audio editing corpus and the most likely artifact to be reused beyond this paper.