LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

TL;DR

LatentUM is a unified multimodal model (UMM) that represents text and images in a single shared semantic latent space so the model can reason over its own generated visual tokens without ever decoding to pixels. The recipe: discretize CLIP features via Model-Behavior-Aligned Quantization (MBAQ), wire understanding and generation through a Mixture-of-Modal-Experts (MoME) transformer with shared self-attention, and bolt on a separate diffusion decoder only when pixels are needed for display. The base model (built on InternVL3.5-4B, trained on 32M BLIP3o pairs) matches BLIP3o-8B on understanding and beats all open UMMs on GenEval (0.85); GRPO with a self-reward derived from the model’s own understanding head (“generate-then-reflect”) pushes GenEval to 0.87 (0.92 with external pixel reward) and yields SOTA on the Visual Spatial Planning benchmark. The same shared-latent substrate is used to predict future visual states for world modeling.

Key claims

Existing UMMs use disjoint visual representations for understanding vs generation, forcing pixel-space mediation between visual reasoning steps and introducing codec bias + cross-modal misalignment [§1, §3].
MBAQ (Model-Behavior-Aligned Quantization) discretizes CLIP features by minimizing the KL between a VLM’s output distribution on the original vs quantized features — preserving VLM behavior rather than pixel fidelity [§3.1.1, Eq. 3.1].
The MoME architecture maintains two parallel per-layer branches (Φu for understanding, Φg for generation), sharing self-attention but with separate FFNs and projections; routing is triggered by a <BOI> special token [§3.1.2].
Multi-codebook quantization (8 chunks × 2048 codes each → effective vocab 2048⁸) discretizes each CLIP token; a 3-layer causal transformer “vision head” autoregresses code indices sequentially within a token [§3.1.1, §3.1.3, §4.1].
A separate dual-stream diffusion decoder (adapted from Stable Diffusion 3.5 Medium) is trained for optional pixel visualization with the quantized features as conditioning — the main model never optimizes for pixel reconstruction [§3.1.4, Eq. 3.2].
LatentUMBase, initialized from InternVL3.5-4B’s understanding branch and trained on 32M BLIP3o image-text pairs over 4 days on 64 H100s, hits MME-P 1654 / SEED 76.3 / MMMU-val 54.6 — beating BLIP3o-4B and approaching BLIP3o-8B (1683 / 77.5 / 50.6) on understanding [§3.2, Table 1].
Quantized-feature setting degrades understanding only modestly vs continuous features (MME-P 1654 → 1638; MMMU 54.6 → 52.3), and in some benchmarks LatentUM with quantized features outperforms baselines using original continuous features [§4.2.1, Table 1].
LatentUMBase scores 0.85 overall on GenEval — best among unified models despite the smallest training data (matching BLIP3o-4B) [§4.2.2, Table 2].
“Generate-then-reflect” via GRPO with a self-reward (the model’s own understanding head answers verification multiple-choice questions about its generations) drives GenEval to 0.87 (LatentUMVis-Gen); swapping in the external GenEval pixel-reward model pushes it to 0.92 [§3.3, §4.3.1, Table 2].
On GenEval2 (MLLM-as-judge), LatentUMVis-Gen scores 31.3/72.9 (GM/AM) vs BLIP3o-8B 13.3/59.4 and Bagel+CoT 23.1/70.9 — a large margin from purely self-derived reward [§4.3.1, Table 3].
The model can predict future visual states as semantic tokens in the shared latent space for world modeling, yielding semantically coherent (rather than pixel-faithful) anticipation of environment dynamics [§1, §2].
LatentUMVis-Plan supports two interleaved planning paradigms — coarse-grained (analyze maze → emit full visual plan → derive solution) and fine-grained (alternating textual reasoning with visual state updates, each visual state feeding the next action) — and beats prior visual reasoning models on Visual Spatial Planning under both [§4.3.2].

Method

The pipeline has three components. (i) Visual tokenizer via MBAQ: take CLIP features from InternVL3.5-4B’s encoder, apply multi-codebook quantization (8 chunks × 2048-entry codebooks), and train the quantizer so a downstream VLM’s output distribution on quantized features matches its distribution on the originals (KL loss) — i.e. preserve behavior, not pixels. (ii) MoME transformer: each layer has parallel understanding (Φu, initialized from InternVL3.5-4B and frozen) and generation (Φg, trained from scratch) branches sharing self-attention. A <BOI> token routes subsequent positions to Φg; otherwise tokens go to Φu. A 3-layer causal transformer “vision head” predicts the 8 code indices within each visual position autoregressively. (iii) Pixel decoder: a fine-tuned MMDiT (SD3.5 Medium) with text conditioning replaced by quantized visual features — trained independently for display purposes only.

Post-training has two modes. SFT on interleaved multimodal datasets uses a parallel-sequence trick: visual tokens flow through both Φu (to build the understanding KV cache for downstream tokens) and Φg (to compute generation loss), with a custom attention mask preserving each branch’s causal structure for single-pass training. GRPO with cross-modal rollouts and a self-reward derived from the model’s own understanding head — multiple-choice verification questions whose ground-truth options come from the prompt specification — drives both visual-generation quality (LatentUMVis-Gen) and visual-spatial-planning skill (LatentUMVis-Plan).

Results

Understanding (Table 1): MME-P 1654 / POPE 88.9 / SEED 76.3 / MMBench 80.3 / MMMU-val 54.6, beating all unified models except BLIP3o-8B at most metrics (BLIP3o-8B is ~2× the size).
Generation, GenEval overall (Table 2): LatentUMBase 0.85, LatentUMVis-Gen (self-reward GRPO) 0.87, LatentUMVis-Gen + pixel-reward 0.92 — vs Bagel 0.82, Janus Pro 0.80, BLIP3o-8B 0.84; competitive with Qwen-Image 0.87 (a specialized T2I model).
GenEval2 (Table 3): LatentUMVis-Gen TIFA-GM 31.3 / TIFA-AM 72.9, beating Bagel+CoT (23.1 / 70.9), Janus Pro (14.5 / 58.3), BLIP3o-8B (13.3 / 59.4) by large margins.
Visual Spatial Planning: LatentUMVis-Plan achieves “near-perfect accuracy” on the fine-grained variant (full numbers truncated in the fetched excerpt) and is reported as SOTA among unified models.
World modeling: qualitative — the model produces “semantically coherent anticipation of environment dynamics” as semantic tokens, not pixel video [§1, §4].
Compute: pretrain 4 days × 64 H100s; decoder fine-tune 2 days × 32 H100s.

Why it’s interesting

LatentUM cuts directly at the open question the Unified Multimodal Models page tracks under “Does generation supervision improve understanding?” — by unifying the representation space itself so the same tokens are read by Φu and produced by Φg, the model can run cross-modal CoT without the codec round-trip UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? flagged as a key cause of G2U failure. This also reframes the Thinking with Modalities substrate taxonomy: LatentUM is neither the diffusion-native parallel-exploration substrate of DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models nor the external-tool loop of Introducing Agentic Vision in Gemini 3 Flash — it’s a fourth position, “tagged autoregressive on shared semantic tokens”, and it’s the first instance where the visual reasoning trace never leaves the latent space. The world-modeling experiments connect the same shared-latent substrate to World Foundation Models as predictive-latent rather than pixel-rollout — closer in spirit to the V-JEPA family than to Genie 3. The MBAQ trick (quantize to preserve VLM behavior, not pixels) is the kind of “what is the right loss for the tokenizer in a UMM” question that several recent papers (NEO-unify: Building Native Multimodal Unified Models End to End, Unified Latents (UL): How to train your latents) circle around without converging.