UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?

TL;DR

UniG2U-Bench is a 3,000-sample, 30-subtask, 7-regime benchmark built specifically to test whether unified multimodal models (UMMs) actually use generation to improve understanding — the so-called G2U direction. The protocol strictly pairs each UMM against its purely discriminative base VLM under matched budgets and evaluates two inference modes (Direct vs. Generate-then-Answer). Across 30+ models (E2E like Bagel/UniPic2, Decoupled like OneCAT/Bagel-two-stage, Agentic like GPT-4o + image tool), the headline finding is negative: unified models generally underperform their base VLMs, and explicitly generating intermediate images before answering usually makes things worse. Gains do appear on spatial-intelligence, visual-illusion, and multi-round-reasoning subtasks where externalizing transformations is intrinsically useful.

Key claims

Unified models generally underperform their corresponding base VLMs on standard understanding tasks; the Generate-then-Answer protocol typically degrades performance relative to direct inference [§1, Findings].
Consistent G2U gains appear in spatial intelligence, visual-illusion, and multi-round reasoning subtasks where enhanced spatial/shape perception and multi-step intermediate states are intrinsically useful [§1, Findings].
G2U effects cluster structurally: perception tasks correlate with perception, logic/reasoning with logic/reasoning; UMMs sharing the same base model show strong behavioral correlation, while UMMs sharing only architecture do not — generation-understanding coupling induces class-consistent inductive biases driven mainly by pretraining data [§1].
The G2U gain is rigorously defined as Δ(UMM, base-VLM) under matched compute budget, isolating it from scale/architecture confounders [§3.1, Eq. 3].
The overall G2U shift decomposes into ΔUMM (capability shift from unified training, no explicit generation) and ΔGtA (effect of explicitly externalizing visual reasoning) [§3.3, Eq. 6].
Two new metrics — Reasoning-Alignment (RA) and Answer-Alignment (AL) — quantify intermediate-image quality during GtA, attributing GtA failures to inaccurate/implausible/misaligned generated artifacts [§1, §3.3].
UMMs are taxonomized into End-to-End (parameter-coupled, e.g. Bagel, UniPic2-Metaquery-9B), Decoupled (module-separated, e.g. Bagel two-stage, OneCAT), and Agentic (tool-use, e.g. GPT-4o + image tool); only E2E and Decoupled support strict base-VLM pairing [§3.2].
Benchmark composition: 200 each (Real-world Apps, Geometry, Physics), 537 Puzzles & Games, 100 Chart/Table, 500 Spatial Intelligence, 1263 Perception Reasoning — 3000 total, the largest unified-model evaluation testbed at the time of writing [Table 1, Table 2].
Compared to MME-Unify (1964 samples, 3 subtasks), Uni-MMMU (524, 4), ROVER (404, 6), RealUnify (400, 4), UniG2U is ~5–7× larger in subtask coverage [Table 1].

Method

UniG2U fixes the central confound in prior UMM benchmarking — that “unified model X beats VLM Y” usually conflates scale, pretraining data, and architecture — by always pairing a UMM with its own discriminative base VLM under a matched budget, then attributing any Δ to G2U synergy. Two inference protocols are run on every model: Direct (single forward pass to answer) and Generate-then-Answer (model first emits an intermediate visual rationale, then re-consumes it to derive the final answer). For each unified system the overall shift Δ_G2U is decomposed into a unified-training component Δ_UMM and an externalization component Δ_GtA. RA (Reasoning-Alignment) and AL (Answer-Alignment) metrics score the intermediate images so failures can be attributed to bad artifacts rather than to the protocol itself.

The 3000 instances are organized into 7 categories chosen because each category is theoretically G2U-relevant — i.e., a domain where externalizing a visual transformation (auxiliary line, simulated state, re-expressed chart, etc.) could plausibly help. Sources include Geometry3K, AuxSolidMath, PhyX, Uni-MMMU, RealUnify, BabyVision, ChartQA, MMSI-Bench, IllusionBench, Visual Puzzles, and VSP-style benchmarks. Evaluation runs under lmms-eval whenever possible, with native stacks as fallback for proprietary models.

Results

Headline (negative) result: across >30 models, unified models generally underperform their base VLMs on standard understanding tasks, and the GtA protocol typically degrades accuracy versus Direct — visual errors in the intermediate artifact propagate forward and mislead the answerer [§1, Findings].
Headline (positive) result: G2U is not uniformly negative. Spatial-intelligence subtasks, illusion-sensitive subtasks, and multi-round reasoning subtasks show consistent improvements where externalized visual transformations function as a genuine reasoning aid [§1, Findings].
Structural correlations: G2U gains cluster by cognitive demand (perception-with-perception, logic-with-logic). UMMs built on the same base model behave alike; UMMs sharing only architecture do not — pretraining data dominates over architectural choices [§1, Findings].
Scale: at the time of writing, UniG2U is the largest unified-model evaluation by sample count and subtask diversity (3000 / 30 vs. nearest competitor MME-Unify’s 1964 / 3) [Table 1].

Why it’s interesting

This is a direct empirical answer to the open question raised on Unified Multimodal Models — does generation supervision actually transfer to understanding? — and the answer is “mostly no, except in specific regimes.” It complements Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (UniMRG), which showed that auxiliary generation of intrinsic representations (depth, segmentation) during training improves understanding; UniG2U evaluates the inference-time G2U variant (Generate-then-Answer) and finds it usually fails. The two together sharpen the picture: G2U synergy works when the generated artifact is an intrinsic structural representation baked in during training, not a free-form intermediate image emitted at test time. The taxonomy (E2E vs. Decoupled vs. Agentic) and the strict base-VLM pairing protocol are a useful template for future UMM evaluation — including for video-side UMMs like DuoGen: Towards General Purpose Interleaved Multimodal Generation and tri-modal MDMs like The Design Space of Tri-Modal Masked Diffusion Models, which currently have no comparable diagnostic harness.