Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

TL;DR

UniMRG is an architecture-agnostic post-training recipe for Unified Multimodal Models (UMMs) that asks the model to generate depth maps, segmentation maps, and pixel reconstructions of input images alongside standard image-understanding supervision — explicitly inverting the usual direction where understanding signals are used to improve generation. The hypothesis: forcing the UMM to emit intrinsic geometric (depth) and structural (segmentation) representations makes it internalize spatial relations and object boundaries that transfer to downstream perception. Across three UMM paradigms (Show-o for AR, Harmon for AR+MAR, OpenUni for AR+Diffusion), the recipe improves fine-grained perception (MMVP), hallucination (HallusionBench), and spatial understanding (VSR, RealWorldQA), while also preserving or improving generation metrics (GenEval, DPG). On OpenUni-3.6B the gains are +3.0 MMVP, +3.7 HallusionBench, +7.2 VSR, putting it at SOTA among ~2B unified models.

Key claims

The reverse direction — using generation to improve understanding — is largely unexplored in UMM post-training, in contrast to RecA, SRUM, UniCorn which all use understanding to improve generation [§1, §2].
Off-the-shelf UMMs cannot generate plausible depth or segmentation maps even when prompted; they default to RGB-like reconstructions, suggesting intrinsic geometric/structural cues are not internally represented [§1, Fig. 1, Fig. 2].
Pixel-level reconstruction alone is dominated by texture/color and provides only weak supervision on intrinsic factors (geometry, structure); explicit auxiliary generation of depth and segmentation is needed to induce spatial and object-structural understanding [§3.2].
UniMRG trains four losses jointly with equal weights: visual understanding (cross-entropy on VQA), image reconstruction, image-to-depth (target from Depth Anything V2), and image-to-segmentation (target from SAM automatic mask generation); no architectural changes, no inference overhead [§3.1, Eq. 5].
The VQ-VAE and text encoder/decoder are frozen during training; the understanding encoder is updated only for UMMs with a shared understanding/generation encoder (Harmon); all other components are trainable [§3.1].
SFT with only the understanding loss catastrophically destroys generation: on Harmon, GenEval drops from 71.37 → 0.30 and DPGBench from 80.52 → 2.85, motivating the need for an auxiliary generation objective [Table 1, Table 3].
On Harmon-1.5B, UniMRG yields +1.80 MMBench, +2.67 MMVP, +2.63 HallusionBench, +5.23 RealWorldQA, +0.33 VSR, +13.89 GenEval, +4.75 DPG over the base model [Table 1].
On OpenUni-3.6B, UniMRG yields +3.00 MMVP, +3.68 HallusionBench, +7.21 VSR, +6.87 GenEval over the base, beating Janus-Pro-7B and UniLIP-2B at similar/smaller scale [Table 1, Table 2].
Ablation: adding losses incrementally (SFT → +pixel → +depth → +segmentation) shows understanding gains only kick in once depth supervision is added (VSR 59.00 → 60.39), with segmentation contributing the final boost on Hallusion (48.26 → 49.32). Pixel-only is sufficient for restoring generation quality but provides no understanding gains [Table 3].
Pixel reconstruction is necessary to keep generated images natural: with only depth or only segmentation auxiliary generation, the generator’s output distribution drifts toward depth-/segmentation-map appearance [§4.3, Fig. 5].
Out-of-distribution depth-generation evaluation on synthetic MidjourneyV6 images shows UniMRG’s depth quality (1−MAE) jumps from 0.623 → 0.822 (Harmon) and 0.617 → 0.834 (OpenUni), evidence that the model is internalizing depth structure rather than memorizing the training distribution [§4.4, Fig. 6].
Show-o gains less from UniMRG (0.637 → 0.664 depth quality, small understanding lifts) because its 4,096-token VQ codebook acts as a representational bottleneck when generating multiple intrinsic representations — same capacity ceiling RecA also hit on Show-o [§4.4].
Training is cheap: ~3 hours on 8×H20 for OpenUni, ~5h for Harmon, ~8h for Show-o, using LLaVA Mix-665K + LLaVA-Next-Data [§4.1].

Method

UniMRG augments a UMM’s standard understanding training with three auxiliary generation tasks, sharing a single multi-task cross-entropy / diffusion loss. The four tasks are: (1) image understanding — cross-entropy on VQA-style (question, image, answer) triples; (2) image reconstruction — generate the input image given prompt “Generate original image for the input image”; (3) image-to-depth — generate the Depth Anything V2 depth map given prompt “Generate depth map for the input image”; (4) image-to-segmentation — generate the SAM automatic-mask segmentation map given prompt “Generate segmentation map for the input image”. Generation-side loss is diffusion loss (OpenUni) or cross-entropy on discrete image tokens (Show-o, Harmon). All four losses are summed with weight 1.

The recipe is architecture-agnostic and assumes only that the UMM exposes (a) an understanding encoder producing visual embeddings for VQA and (b) a generation head that can produce images. VQ-VAE and text encoder/decoder are frozen; the understanding encoder is updated only when the UMM shares it with the generator (Harmon). The depth and segmentation targets are precomputed once with Depth Anything V2 and SAM, so no third model needs to live in the training loop. Inference is identical to a vanilla UMM — the auxiliary heads only matter at training time.

The design discussion (§3.2) frames the choice of depth + segmentation in terms of two criteria: (i) the representation must capture intrinsic factors weakly constrained by pixel reconstruction, and (ii) it must give dense supervision aligned with common failure modes — spatial reasoning errors (→ depth) and hallucinated objects (→ segmentation, which forces object-boundary awareness).

Results

On Harmon-1.5B, UniMRG raises MMBench 50.43 → 52.23, MMVP 60.00 → 62.67, HallusionBench 46.69 → 49.32, RealWorldQA 46.67 → 51.90, VSR 60.88 → 61.21, while GenEval rises 71.37 → 85.26 and DPGBench 80.52 → 85.27. RecA matches the generation gain (GenEval 83.86) but provides no understanding lift (Hallusion 47.11 vs UniMRG 49.32, MMBench 49.50 vs 52.23). On OpenUni-3.6B the same pattern holds with larger absolute spatial gains: VSR 66.69 → 73.90, MMVP 71.67 → 74.67, Hallusion 60.88 → 64.56. At 2B params UniMRG outperforms unified models 3–5× larger on the understanding benchmarks: 64.56 Hallusion vs Janus-Pro-7B 60.15, 73.90 VSR vs Janus-Pro-7B 71.03, while also reaching 74.67 MMVP vs Qwen2.5-VL-3B’s 64.67 and InternVL3-2B’s 71.67 (Table 2).

The Show-o result is a clean negative control: a UMM with a 4,096-token VQ codebook does not have the representational capacity to host depth + segmentation + pixel + understanding jointly, and UniMRG’s depth-quality lift (0.637 → 0.664) and understanding gains (~+1 to +3 across the board) are correspondingly small. This is consistent with RecA’s earlier finding on the same model.

Ablation on Harmon (Table 3) is the cleanest evidence that depth and segmentation are separately doing work: from + Pixel Generation (the RecA-equivalent setting) to + Depth Generation, MMVP rises 61.00 → 62.00, Hallusion 47.11 → 48.26, RealWorldQA 49.54 → 50.46, VSR 59.00 → 60.39; adding segmentation on top then lifts Hallusion further to 49.32 and MMBench 50.60 → 52.23. The OOD MidjourneyV6 depth-generation check (Fig. 6) is the key generalization argument — the model is not just regurgitating Depth Anything V2 on the training distribution.

Why it’s interesting

This is a direct rebuttal of the assumption that, in a unified model, only understanding supervision can improve understanding. The Feynman epigraph in §1 (“what I cannot create, I do not understand”) becomes an actually-tested architectural prescription rather than a slogan. The recipe matters for any team running a UMM stack: it’s three extra cross-entropy / diffusion terms with pre-computed targets, no architecture change, no inference cost, and the deltas on spatial understanding (VSR +7.2 on OpenUni) and hallucination (Hallusion +3.7) are the kind of numbers that usually require either much more data or a dedicated module like SpatialRGPT / DepthVLA.

This complements DuoGen: Towards General Purpose Interleaved Multimodal Generation, which makes the related architectural bet that a video DiT is the right image-generation head for a unified text+image system — DuoGen exploits frame-consistency priors to get visual consistency, UniMRG exploits depth/segmentation generation to get spatial understanding; both arguments are that the choice of generation target shapes what the model implicitly learns. It also contrasts with The Design Space of Tri-Modal Masked Diffusion Models which explores the design space of which modalities a unified masked-diffusion model should jointly model — UniMRG’s claim is downstream: once you’ve decided on the joint set, adding more generation targets (even synthetic ones from off-the-shelf perception models) keeps paying. The Show-o codebook bottleneck finding is also a concrete data point for Parametric memory-adjacent discussions: representational capacity (codebook size, hidden width) sets a hard ceiling on how many intrinsic representations the model can host simultaneously.