Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer

TL;DR

Ming-UniVision (Ant Group / inclusionAI, Ling-lite 2.8B-active backbone) is a unified autoregressive vision-language model whose visual tokenizer, MingTok, produces a continuous latent space that serves both image understanding and image generation through a single next-token-prediction objective. MingTok is a three-stage transformer (low-level encoder → causal semantic decoder → pixel decoder) trained with masked feature prediction so that compact 32-channel latents are autoregressive-friendly while the semantic decoder expands them into CLIP-aligned high-dimensional features for VQA. Because the LLM always consumes the semantic-decoder output regardless of whether it is encoding a real image or expanding its own freshly emitted latent, multi-round in-context editing runs entirely in latent space — no VAE decode + re-encode between rounds, ~66% fewer input tokens than dual-tokenizer baselines like BAGEL.

Key claims

Discrete-tokenizer unified models (VILA-U, TokenFlow, UniTok, Harmon, TokLIP) quantization-bottleneck both understanding and generation, motivating a continuous-latent shared tokenizer instead [§1, §3.2].
MingTok uses three sequential transformer stages — low-level encoder with full attention, semantic decoder with causal attention (per-token AR support), pixel decoder with full attention + pixel-unshuffle — and is trained end-to-end with masked image modeling [§2.1, Table 1].
Three training objectives are imposed jointly on MingTok: latent-space DINOv2-target masked-feature prediction (regularizes the compact latents), semantic-decoder CLIP-target masked-feature prediction (gives text-aligned high-dim semantics), and reconstruction supervision on the pixel decoder under both masked and unmasked inputs (mimics AR-style partial-information decoding) [§2.2].
Architectural improvements to the per-token vision head: a rectified-flow prediction objective replaces the diffusion denoising head used by UniFluid, and the MLP block is replaced by a SwiGLU FFN; both reportedly improve latent prediction accuracy at matched parameters [§3.1].
Multi-round editing argument: DiT-based heads (FLUX.1-class) are structurally incapable of multi-round in-context generation because they are trained for a fixed number of reference images per forward pass; AR-Diffusion hybrids (BAGEL, LMFusion) carry triple representations per image (semantic / noisy latent / clean latent) and require unconventional masking, while unified-AR but separate-tokenizer designs (UniFluid) still double the effective sequence length [§3.2].
The shared semantic-space input means after a generation step, the resulting F_t is reused directly as the conditional input for round t+1 — no pixel-space round trip, no cumulative re-encoding drift [§3.2, Fig. 4].
Pre-training is staged: Stage 1 only trains the LLM↔MingTok MLP and the rectified-flow vision head with both backbones frozen (~30% understanding / 70% generation); Stage 2 unfreezes the LLM and unfreezes only the semantic decoder of MingTok (low-level encoder stays frozen) and uses mixed-resolution training — 1024×1024 for understanding, 512×512 for generation [§3.3.1].
The final SFT stage explicitly trains on multi-round editing instruction chains (55% of data), with the rest split across single-turn generation, understanding, and NLP [§3.3.2].
On understanding benchmarks at 2.8B active params: MMB 78.5 / MMS 63.7 / MathV 66.6 / AI2D 82.8 / MM-Vet 64.2 — competitive with separate-representation unified models (Janus-Pro-7B, Show-o2-7B, BAGEL) and substantially above prior unified-representation discrete-tokenizer models (VILA-U, TokenFlow-XL, UniTok, Harmon-1.5B, TokLIP) [Table 2].
GenEval overall and per-category numbers and DPG-Bench are reported in Table 3 (table truncated mid-result in the fetched preprint); the paper positions Ming-UniVision as state-of-the-art among unified-representation models on T2I [§4.2, Table 3].

Method

MingTok is the central artifact. The input image (512×512, patch size 32) is mapped by the low-level encoder (12 layers, dim 768, full attention) into 16×16 = 256 latent tokens whose output dimension is compressed to 32 channels via a linear projection with a channel-averaging shortcut — these compact 32-d tokens are what the LLM autoregressively emits. The semantic decoder (24 layers, dim 1024, causal attention) takes these compact latents in and expands them back to 1024-d semantic features one token at a time, with a channel-repeating input layer; this is the representation the LLM always sees on input. The pixel decoder (24 layers, dim 1024, full attention, pixel-unshuffle to halve the effective patch size to 16) reconstructs RGB from the 1024-d semantic features.

Training is masked-image-modeling on all three stages jointly: at the latent layer, masked tokens predict DINOv2 features at the same locations; at the semantic-decoder output, masked tokens predict CLIP features; the pixel decoder reconstructs the full image under both masked and unmasked conditions, forcing it to be robust to partial latents (which is exactly what it sees during AR generation).

Ming-UniVision then plugs MingTok into a Ling-lite MoE LLM (2.8B active params): images on input are encoded by MingTok’s full stack and the LLM consumes the semantic features; for generation, the LLM emits compact 32-d latents through a per-token vision head (rectified-flow + SwiGLU FFN), each emitted latent is immediately expanded by the causal semantic decoder back to 1024-d before being fed to the LLM as context for the next token. After generation, the resulting semantic feature sequence is already in the form the LLM expects for the next round of editing — no decode-encode round trip.

Results

Headline understanding numbers at 2.8B active (Ming-UniVision-16B-A3B in Table 2): MMB 78.5, MMS 63.7, MMMU 40.3, MathV 66.6, HallusionBench 47.8, AI2D 82.8, MM-Vet 64.2, OCRBench 724, MME 2023. Within the “unified model, unified representation” row, this beats the prior best (TokenFlow-XL 76.8 MMB, 43.2 MMMU, 48.2 MM-Vet) on every reported metric and brings unified-representation performance to within striking distance of separate-representation unified models like BAGEL (85.0 MMB, 55.3 MMMU, 67.2 MM-Vet) — at a fraction of the active parameters [Table 2].

On the generation side, Table 3 reports GenEval per-category and DPG-Bench numbers; the row for Ming-UniVision is truncated in the fetched preprint but the surrounding “Unified model, Separate representation” row puts Janus-Pro-1B at 0.98 single-obj / 0.82 two-obj / 0.51 counting / 0.89 colors. The paper’s quantitative claim is “state-of-the-art level performance across both domains.”

The qualitative results (Figs. 7–8) show iterative super-resolution and segment-then-edit chains executed entirely in latent space, which is the architectural payoff: up to 66% fewer input tokens than the dual-tokenizer baselines for the same multi-round editing trajectory [§1].

Why it’s interesting

This is the cleanest published recipe so far for the “unified representation” branch of the Unified Multimodal Models taxonomy — explicitly defined in UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? as the regime where the same tokenizer feeds both understanding and generation. Prior unified-representation entries on the wiki (VILA-U, TokenFlow, UniTok, Harmon, TokLIP) all used discrete codebooks and paid a heavy penalty on understanding; Ming-UniVision is the first filed paper to show that a continuous latent with a causal semantic decoder closes most of the gap to separate-representation systems while keeping the in-context editing latency win.

Contrasts worth tracing: NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify) takes the opposite tack — removes the VAE and the vision encoder entirely and works in pixel space through a Mixture-of-Transformer with separate understanding and generation pathways — making Ming-UniVision and NEO-unify the two opposing answers to “what should the visual interface to a UMM be?” Meanwhile Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG) argued that auxiliary generation of intrinsic representations (depth, segmentation) improves understanding; Ming-UniVision’s design is consistent with this — the semantic decoder is trained against CLIP features under a generation-style masked objective — but the paper does not report a UniMRG-style ablation isolating the contribution.