ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation

TL;DR

ROVER is a human-annotated benchmark of 1,312 tasks grounded in 1,876 images that explicitly targets reciprocal cross-modal reasoning in unified multimodal models (UMMs) — using one modality to guide, verify, or refine outputs in the other. It splits into two complementary settings: verbally-augmented reasoning for visual generation (does language-side CoT guide faithful image synthesis) and visually-augmented reasoning for verbal generation (do intermediate visualizations strengthen QA reasoning). Across 17 evaluated unified models, two headline findings emerge: cross-modal reasoning quality correlates with interleaved-generation architecture (and combining strong unimodal models fails to match), and models show a clean dissociation between physical and symbolic visual reasoning — they interpret perceptual concepts literally but fail to construct visual abstractions for symbolic tasks. Relevant to Luma because it sharpens the gap between “image-generation as decoration” and “image-generation as a reasoning substrate,” and provides a verification-ready evaluation harness for omnimodal model development.

Key claims

ROVER is the first benchmark targeting reciprocal cross-modal reasoning specifically, with 1,312 tasks over 1,876 images split into two settings (verbally-augmented visual generation; visually-augmented verbal generation) [§1, Fig. 1].
Cross-modal reasoning capability largely determines reasoning-dependent visual generation quality, with interleaved image-text generation models significantly outperforming non-interleaved counterparts [§1 findings].
Combining strong unimodal models (a frontier LLM + a frontier T2I model) fails to match the cross-modal reasoning of a true interleaved UMM on ROVER — independent optimization of constituent modalities is insufficient [§1 findings].
Models show a striking dissociation between physical and symbolic visual reasoning: they succeed at literal interpretation of perceptual/physical concepts but fail to construct visual abstractions for symbolic tasks, where faulty intermediate reasoning degrades rather than improves downstream performance [§1 findings].
The benchmark provides a principled task taxonomy (natural science, culture & art; temporal, spatial, causal, imaginative, quantitative axes) and a verification-ready instance design with process targets and visual artifacts, scored along coherence, alignment, and step-correctness dimensions [§1 contributions].
17 unified multimodal models are evaluated across 23 diverse task types [§1; project page].

Method

ROVER’s design factors omnimodal evaluation into two complementary tasks. Verbally-augmented reasoning for visual generation prompts a UMM with a problem requiring world-knowledge or symbolic reasoning, then evaluates the generated image against ground-truth visual artifacts and process targets (e.g., a chemistry-textbook diagram of water’s molecular structure with accurate bond angles). Visually-augmented reasoning for verbal generation evaluates whether a UMM benefits from emitting an intermediate visualization (a sketch, plot, or diagram) during its CoT before producing a textual answer — i.e. whether the model can use its own generation capability as a scratchpad. Instances are human-annotated with process targets and verification-ready visual artifacts so that intermediate-step correctness can be scored, not just final-answer accuracy. The multi-dimensional protocol scores coherence between modalities, alignment with the prompt’s reasoning intent, and step-level correctness.

Results

Headline architecture finding: interleaved image-text generation models significantly outperform non-interleaved unified models on cross-modal reasoning; the gap is not closeable by pairing a strong unimodal LLM with a strong unimodal T2I model [§1].
Dissociation finding: across the two settings, models that interpret visual elements literally (perceptual / physical world concepts) succeed, but the same models fail when the task requires constructing visual abstractions as symbolic representations; in the visually-augmented verbal-generation setting, poor intermediate visualizations actively hurt downstream QA accuracy [§1].
Scope: 17 unified models × 23 task types; explicit comparison categories cover natural science, culture & art, temporal, spatial, causal, imaginative (Imagin.), and quantitative (Quant.) reasoning [Fig. 1 panel labels].

Why it’s interesting

ROVER is the first benchmark to directly stress-test the central premise of the unified-multimodal-models program — that generation and understanding should mutually reinforce through reciprocal cross-modal reasoning. The interleaved-beats-decoupled finding is consistent with the page’s current architectural taxonomy (Unified Multimodal Models) and gives an evaluation harness orthogonal to GenEval/MMMU. It also directly engages the “thinking with images” thesis surveyed in Thinking with Modalities: ROVER’s visually-augmented verbal generation setting is the same substrate as DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models and Introducing Agentic Vision in Gemini 3 Flash, but evaluated end-to-end on a UMM rather than retrofitted via fine-tuning or tool use. The symbolic-vs-physical dissociation finding sharpens the negative result already on the wiki — Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs reports that pure-text CoT fails on binding-heavy visual tasks; ROVER reports that self-generated visual reasoning also fails when the visual abstraction is symbolic. The benchmark itself sits in the VLM-as-Evaluator frame: it uses human-annotated process targets and a multi-dimensional scoring protocol rather than rule-based or LMM-only judges.