Ming-Omni: A Unified Multimodal Model for Perception and Generation

TL;DR

Ming-Omni is Ant Group’s open-source any-to-any multimodal model that takes image / text / audio / video as input and produces text, speech, and image as output. The trunk is Ling, an MoE LLM with per-modality routers so visual, audio, and text tokens are routed to specialized expert subsets, with a two-stage balancing schedule to handle the convergence-rate mismatch across modalities. Generation is bolted on through a frozen-MLLM bridging design: an audio decoder with a BPE-compressed token stream (~36% length reduction) for context-aware speech, and a multi-scale-learnable-token + DiT bridge to a Ming-Lite-Uni-style image generator. Released as the first open-source omni model claimed to match GPT-4o’s modality coverage, with the Ming-Lite-Omni 2.8B-active variant comparable to Qwen2.5-VL-7B on image perception, ahead of Qwen2.5-Omni and Kimi-Audio on speech understanding, and GenEval 0.64 / FID 4.85 on T2I.

Key claims

A single language model with per-modality routers in the MoE layers (one router for visual tokens, one for audio, etc.) is sufficient to route modality-specific information to specialized experts without per-modality model copies [§2.1].
A two-stage cross-modal balancing schedule — step-wise balance in pre-training, dynamic adaptive balance (loss-weights driven by per-modality convergence rates) in instruction tuning — mitigates convergence-rate mismatch across modalities and removes the need for hand-tuned per-task loss weights [§2.1, §2.4].
BPE applied to the discrete audio-tokenizer output compresses the audio token stream from 50 Hz to ~32 Hz (≈36% reduction) reversibly, improving training/inference efficiency without quality loss and adds prosodic combinatorial signal similar to text BPE [§2.2].
Audio generation is trained with the MLLM frozen: only the audio decoder is updated on TTS + multi-modal context-aware triplet data, preserving multimodal understanding while still letting the decoder consume the MLLM’s hidden state to capture paralinguistic context (emotion, ambient cues) [§2.2].
Predicting a language identifier before downstream speech tasks (ASR etc.) significantly improves overall performance, especially on dialect ASR — Chinese-language ASR scales but plateaus once dialect data exceeds ~2% of the audio corpus [§2.2, §3.3].
For image generation Ming-Omni keeps the MLLM frozen and bridges to a DiT via multi-scale learnable query tokens with scale-specific positional encodings and explicit start/end boundary tokens per scale, plus a representation-alignment loss (MSE between DiT intermediate hidden states and the final semantic representations) [§2.3].
Training proceeds in two macro stages — perception (pretrain + instruction-tune + alignment-tune, sub-divided into three sub-stages each, following M2-omni) then generation (frozen MLLM, train only audio decoder + image connector + multi-scale queries + DiT) — so adding generation does not regress understanding [§2.4].
The 2.8B-active Ming-Lite-Omni matches Qwen2.5-VL-7B on image perception (e.g. MMBench-TEST-V11 80.8 vs 82.8; OCRBench 88.4 vs 87.8; MathVista 71.6 vs 68.1) while using a fraction of the active parameters [Table image-perception].
Ming-Lite-Omni reports SOTA-or-near-SOTA on visual grounding among Omni models (RefCOCO val 90.6, RefCOCOg test 87.5; average 87.3) and beats Qwen2.5-VL-7B and InternVL2.5-8B-MPO on GUI / Android-task benchmarks (ScreenSpot 82.1 vs 78.9; AITZ-EM 66.6 vs 57.6) [Tables grounding, GUI].
On InfoSeek (knowledge-intensive VQA) Ming-Lite-Omni reports H-mean 27.7 vs PaLI-X-32B 22.1 and Qwen2.5-VL-7B 19.4 — an information-seeking gap the paper attributes to the iterative caption-refinement pipeline + structured/encyclopedia data [Table InfoSeek].
On image generation Ming-Omni reports GenEval AVG 0.64 (above SDXL) and FID 4.85, claimed as new SOTA for the unified-perception+generation setting [results figure; Sec. 1].

Method

The architecture has three layers: (1) modality encoders — Qwen2.5-VL vision backbone (~675M params) for image/video at native resolution, Whisper for audio + a linear + conv-downsample projection into Ling’s hidden size; (2) the Ling MoE LLM trunk with per-modality routers, an active-param count of 2.8B for the released Ming-Lite-Omni variant; (3) generation heads added in a second training stage. The audio decoder follows CosyVoice — an AR architecture predicting BPE-compressed discrete audio tokens conditioned on the MLLM hidden state — and is trained alone with the rest of the model frozen. The image generator bolts a DiT on via a connector: per-scale learnable query tokens (e.g. scales corresponding to global layout, mid-level structure, fine texture) are inserted into the LLM input with scale-specific positional encodings and start/end boundary tokens, the LLM produces hidden states for those queries, the connector feeds them to the DiT, and an MSE loss aligns DiT intermediate hidden states to the LLM’s final semantic representations — a representation-level alignment objective in the spirit of REPA but used for generation rather than understanding.

Training data spans an iterative self-evolving caption-refinement pipeline (DiverseEvol / ASK-LLM-style), a structured-data synthesis pipeline that builds an entity knowledge base from images, an 8-domain encyclopedia corpus, GUI data (AitW / GUICourse / AndroidControl), Chain-of-Thought reasoning data (R1-Onevision), and a 41-subcategory preference corpus for alignment tuning. Audio data combines open-source ASR corpora with a custom web-crawled corpus filtered through VAD + an iteratively trained audio labeler that tags scenario / ambience metadata, which is then optionally injected into the speech-task prompt.

Results

Image perception: Ming-Lite-Omni vs Qwen2.5-VL-7B-Instruct — AI2D 83.1 vs 84.4, HallusionBench 55.0 vs 55.8, MMBench-TEST-V11 80.8 vs 82.8, MMMU 56.3 vs 56.6, MMStar 64.7 vs 65.3, MMVet 71.3 vs 71.6, MathVista 71.6 vs 68.1, OCRBench 88.4 vs 87.8 — i.e. near-parity at a fraction of active params [Table image-perception].
Visual grounding: average of RefCOCO/+/g splits 87.3, matching Qwen2.5-Omni-7B (87.7) and InternVL2.5-8B (87.6), beating Grounding-DINO-Large (86.6) and Qwen2.5-VL-7B (86.6) [Table grounding].
GUI: ScreenSpot 82.1 (vs InternVL3-8B 79.5, Qwen2.5-VL-7B 78.9); ScreenSpot-V2 84.1 (vs InternVL3-8B 81.4); AITZ-EM 66.6 (vs Qwen2.5-VL-7B 57.6) [Table GUI].
Information-seeking VQA (InfoSeek): H-mean 27.7 vs PaLI-X-32B 22.1 and Qwen2.5-VL 19.4; Unseen-question 30.4 vs 23.5 / 20.6; Unseen-entity 25.4 vs 20.8 / 18.3 [Table InfoSeek].
Image generation: GenEval 0.64 AVG (above SDXL on the metric); FID 4.85 (paper claims new SOTA for unified perception+generation models) [Sec. 1, results figure].
Speech understanding & instruction following: outperforms Qwen2.5-Omni and Kimi-Audio (qualitative claim in §1; exact tables in the full PDF beyond the truncated body).

Why it’s interesting

Ming-Omni is the earliest end-to-end open-source datapoint for Ant’s Ming series — the foundation that the later Ming-flash-omni 2.0 (100B-total / 6B-active, MIT-licensed, continuous-AR + DiT-head audio decoder) builds on. The combination of per-modality routers in an MoE LLM + frozen-MLLM bridging to per-modality generation heads is now a recognizable recipe: contrast with NEO-unify: Building Native Multimodal Unified Models End to End (encoder-free MoT, no VE / no VAE), with the AR+Diffusion DuoGen (DuoGen: Towards General Purpose Interleaved Multimodal Generation), and with the from-scratch discrete-MDM trunk of The Design Space of Tri-Modal Masked Diffusion Models — Ming-Omni sits in the “MoE-with-per-modality-routers + frozen-trunk-for-generation” camp and was the first open instance of it. The per-modality-router idea is also a direct entry into the MoE Routing Design cluster: where most MoE-routing papers vary routing primitives across the token dimension, Ming-Omni varies routers along the modality dimension — a routing axis the existing cluster does not otherwise cover.