Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

TL;DR

Ming-Flash-Omni is the Inclusion AI / Ant Group tech report for the Preview release behind Ming-flash-omni 2.0 — a sparse-MoE any-to-any omni-MLLM with 100B total / 6.1B active parameters, built on the Ling-Flash-2.0 backbone. It upgrades Ming-Omni: A Unified Multimodal Model for Perception and Generation along three axes: VideoRoPE positional encoding for long video, context-aware ASR (state of the art on all 12 ContextASR-Bench metrics), and a “Generative Segmentation as an Editing Task” paradigm that recasts segmentation as a semantics-preserving color-edit so understanding and generation share one objective. Image generation reports GenEval 0.90, beating leading non-RL methods; speech generation switches from discrete tokens to continuous acoustic latents to avoid quantization artifacts.

Key claims

The model is built on Ling-Flash-2.0, a sparse MoE LM with 100B total parameters and only 6.1B active per token, with a dual balancing scheme (auxiliary load-balancing loss inherited from Ming-Omni + per-router bias updates from Ling-Flash-2.0) for stable training at high sparsity [§2, §2.1].
VideoRoPE replaces standard RoPE on the perception side to capture temporal dynamics in long video sequences, retaining the Qwen2.5 vision encoder and Whisper audio encoder from Ming-Omni [§2.1].
A context-aware ASR training paradigm conditions decoding on preceding text + a hot-word list during training; reports SOTA on all 12 ContextASR-Bench metrics [§2.2, §1].
Speech generation uses continuous acoustic latents instead of discrete tokens — a frozen Qwen2.5-0.5B audio head takes LLM text tokens and downsampled VAE latents and autoregressively predicts conditioning signals for a flow-matching head, eliminating quantization-induced artifacts [§2.2].
The continuous speech tokenizer is a VAE-GAN trained with multi-scale mel reconstruction + adversarial + feature-matching + KL loss [§2.2].
Generative Segmentation as an Editing Task — instead of producing a binary mask, the model performs a semantics-preserving edit (e.g. “color the banana purple”), forcing understanding (precise contour) to become a prerequisite for editing and giving the edit’s pixel quality as a direct supervisory signal for comprehension [§2.3].
This single training paradigm yields GenEval 0.90, surpassing leading non-RL methods; the paper claims this is evidence that spatio-semantic control trained via segmentation-as-editing generalizes to pure text-to-image compositionality [§2.3].
Identity preservation uses a VAE-encoded identity vector plus a composite global-semantic + local-pixel loss; the segmentation-trained boundary perception is what isolates non-edited regions cleanly [§2.3].
High-fidelity text rendering integrates a Glyph-byT5 text encoder so the model places text using its learned pixel-level control [§2.3].
Multi-image fusion (e.g. ID from image A, background from image B) is supported via distinct Concept Vectors decoded by the same delineation skill from generative segmentation [§2.3].
Training is a two-stage pipeline — perception (pretrain → instruction → alignment → coherent RL with Dynamic-GRPO followed by Unified-DPO) then generation (frozen MLLM, train only the image generator + audio generator from Canxiang et al. 2025) [§2.4].
Coherent RL uses on-policy GRPO with balanced sampling and task-specific verifiable rewards (D-GRPO), then a U-DPO step that adds an instruction-tuning loss term on chosen samples plus interleaved replay of instruction-stage multimodal data [§2.4].
The infrastructure is an enhanced Megatron-LM with sequence packing (dense packing of variable-length multimodal sequences) and flexible encoder sharding across DP/PP/TP to eliminate pipeline bubbles caused by the lightweight modality encoders; reports >2× training throughput vs baseline Megatron-LM [§2.5].
Image-understanding parity with peer MoE-omni models at much smaller active params: MMStar 68.3, AI2D 85.2, MathVista_MINI 81.9, OCRBench 879, TextVQA 82.6, MIA-Bench 93.8 — comparable to Qwen3-Omni 30B-A3B and Qwen3-VL 30B-A3B across most metrics [Table image-perception].
1.5M multimodal long-CoT samples (max 16K tokens) for reasoning, distilled from Gemini and filtered by accuracy/quality; constructed around math, spatial reasoning, and GUI reasoning [§3.1.1].
Synthesized 3M Chinese+English <audio, text, context, entity_list> samples for ContextASR training, with negative samples generated to enhance keyword discrimination [§3.2].
8.2M filtered videos → 1.2M streaming multi-turn conversation samples averaging 5 minutes each, balanced by Qwen3-Embedding clustering to suppress high-frequency data while preserving long-tail content [§3.3].

Method

The architecture retains Ming-Omni’s two-stage pipeline (perception + generation) but swaps the LLM trunk to Ling-Flash-2.0 — a 100B-total / 6.1B-active sparse MoE with distinct routers per modality. Vision and audio encoders (Qwen2.5 + Whisper) project into the LLM’s token stream alongside text. On the understanding side, VideoRoPE replaces standard RoPE for long-context video temporal modeling, and the ASR head is trained with surrounding linguistic context + hot-word lists for context-aware decoding.

The speech generation side abandons discrete tokens in favor of continuous acoustic latents. A frozen pre-trained Qwen2.5-0.5B audio head consumes LLM-generated text tokens plus downsampled VAE latents and autoregressively predicts conditioning signals for a flow-matching head, following Canxiang et al. 2025. The continuous speech tokenizer is a separately trained VAE-GAN.

The most distinctive idea is on the image side: rather than training segmentation as a separate “produce a binary mask” task, Ming-Flash-Omni reframes it as a semantics-preserving edit (color the banana purple). This forces precise object-boundary understanding to be a prerequisite for successful generation, and the edit’s pixel quality becomes a direct supervisory signal back into comprehension. The model is bootstrapped on RefCOCO/+/g and COCO-Panoptic, with each class / instance assigned a unique color via a predefined colormap to create the (pre-edited, edited) training pairs.

For identity-preserving editing the model uses a VAE-encoded identity vector and a composite global-semantic + local-pixel loss; text rendering uses Glyph-byT5 embeddings as additional conditioning; multi-image fusion is enabled by distinct per-source Concept Vectors. Training proceeds as perception (pretrain → instruction → alignment → D-GRPO RL → U-DPO preference) then generation with the perception MLLM frozen — first flow-matching pretraining of the image generator with multi-scale learnable queries, then editing fine-tuning where VAE-encoded reference images are concatenated with the noisy latent and ByteT5 word-level captions enrich text conditioning.

The infrastructure side documents two Megatron-LM extensions important for sparse-MoE multimodal training: sequence packing (variable-length multimodal sequences packed into fixed-length batches to reduce GPU memory fragmentation) and flexible encoder sharding (fine-grained partitioning of modality encoders across DP/PP/TP to avoid pipeline bubbles from heterogeneous encoder shapes). Together these claim >2× throughput over baseline Megatron-LM.

Results

ContextASR-Bench: new SOTA on all 12 metrics within a single unified architecture [§1, §2].
Image perception (vs Qwen3-Omni 30B-A3B / Qwen3-VL 30B-A3B / InternVL3.5 30B-A3B): MMStar 68.3 / 68.5 / 72.1 / 72; AI2D 85.2 / 85.2 / 86.9 / 86.8; HallusionBench 61.1 / 59.7 / 61.5 / 53.8; CV-bench 81.6 (peers not reported) / – / – / 77.3; MathVista_MINI 81.9 / 80.0 / 81.9 / 80.9 — competitive with peer omni models at smaller active params [Table image-perception].
OCR: ChartQA 88.4 / 87.5 / 87.1 / 87.4; DocVQA 94.8 / 95.0 / 95.2 / 94.2; OCRBench 879 / 860 / 903 / 880; TextVQA 82.6 / 81.7 / 81.7 / 80.5 [Table image-perception].
Complex instruction: MIA-Bench 93.8 vs Qwen3-Omni 94.5 [Table image-perception].
Multi-image: MMTBench_val_mi 68.0 vs Qwen3-VL 65.7; Llava_interleave_Bench 63.3 vs Qwen3-VL 51.1 [Table image-perception].
Video: MVBench 74.6 vs Qwen3-VL 72.3 / InternVL3.5 72.1; VideoMME 70.9 vs Qwen3-Omni 70.5 / Qwen3-VL 74.5 / InternVL3.5 68.7; VideoMME_w_subtitle 73.0 (peers not reported); LongVideoBench 61.7 vs InternVL3.5 63.8 [Table image-perception].
Text-to-image (GenEval AVG, vs prior models): Ming-Flash-Omni 0.90, claimed to surpass leading non-RL methods; in the same table SDv1.5 0.43 / SDXL 0.55 / DALL-E 3 0.67 / SD3-Medium 0.74 / Emu3-Gen 0.54 / LWM 0.47 / SEED-X 0.49 / Show-o 0.53 / TokenFlow-XL 0.55 / Janus 0.61 [Table GenEval].
Infrastructure: >2× training throughput vs baseline Megatron-LM after sequence-packing + flexible encoder sharding [§2.5].

Why it’s interesting

This is the tech report behind the Preview release that the already-filed 2.0 announcement (Ming-flash-omni 2.0) cites as its closest published source — it’s the place Luma’s audio + multimodal teams should look for the actual architecture details (continuous-AR + flow-matching audio head, VideoRoPE for long video, the Ling-Flash-2.0 sparse-MoE trunk) rather than the 2.0 blog’s marketing language. Relative to the original Ming-Omni: A Unified Multimodal Model for Perception and Generation, the headline architectural change is the Generative Segmentation as an Editing Task paradigm — a non-trivial unification idea worth contrasting with the wider Unified Multimodal Models cluster: OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning uses thinking-mode reasoning traces to bridge understanding and generation, Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG) uses auxiliary depth/segmentation as understanding targets, and Ming-Flash-Omni does the reverse — uses segmentation as a generation target whose precision implicitly demands understanding. Three independent recipes for the same UMM bottleneck.

For the MoE Routing Design cluster this is also the cleanest open data point on running per-modality routing at 100B/6B-active scale; for Joint audio-video generation the continuous-AR + flow-matching speech-gen head is a concrete alternative to discrete-token TTS that’s now in production weights via the 2.0 release.