R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

TL;DR

R2I-Bench is a reasoning-focused T2I benchmark of 3,068 prompts across 7 primary reasoning categories (commonsense, compositional, logical, numerical, mathematical, causal, concept mixing) and 32 fine-grained subcategories, each built via a GPT-4o-plus-human-expert iterative pipeline. Each prompt is paired with per-instance, human-validated diagnostic questions across three dimensions (text-image alignment, reasoning accuracy, image quality), aggregated by a QA-style metric (R2I-Score) using GPT-4o as the judge. 16 representative T2I models are evaluated: all open-source models score under 45 overall, mathematical reasoning is near-zero across the board, and reasoning-augmented variants (Show-o+PARM/ORM/DPO) yield only marginal gains; gpt-image-1 leads at 0.77 overall and a GPT-4o→SD3-medium prompt-rewrite pipeline lifts SD3-medium by +0.13 to 0.58.

Key claims

R2I-Bench spans 7 primary reasoning categories — commonsense, compositional, logical, numerical, mathematical, causal, concept mixing — and is the only listed T2I benchmark covering all seven; closest competitors (OK-VQA, Winoground, HEIM, GeckoNum, GenEval, GenAI-Bench, ConceptMix, Commonsense-T2I, WISE) each cover one or two of these axes [Table 1, §1].
R2I-Score is a per-instance QA metric: each prompt ships with a set of GPT-4o-generated, human-validated evaluation questions tagged with one of three dimensions (text-image alignment, reasoning accuracy, image quality), an importance weight, and a scoring criterion; the final score is a weighted aggregate of GPT-4o’s per-question scores [§3, eq. 1].
Weight ranges are pinned per dimension to enforce a reasoning-centric metric: reasoning accuracy gets the largest range, then text-image alignment, then image quality — so R2I-Score does not collapse into a visual-quality proxy [§3].
All open-source T2I models score below 0.45 overall on R2I-Score; the best is SD3-medium at 0.45, with Lumina-Image 2.0, Sana-1.5, Omnigen, LLM4GEN at 0.40–0.42 [Table 2].
Mathematical reasoning is a persistent failure mode across all open-source and reasoning-augmented models — best open-source is SD3-medium at 0.19, several models (LlamaGen, ELLA, LLM4GEN) at 0.07; even gpt-image-1 only reaches 0.58 [Table 2, §4.2].
Reasoning-enhanced variants of Show-o (ORM / DPO / PARM) deliver only marginal improvements over base Show-o (0.36 → 0.38 overall with PARM) and continue to score poorly on mathematical and causal reasoning [Table 2, §4.2].
Closed-source models dominate: DALL-E-3 0.71 overall, gpt-image-1 0.77, with gpt-image-1 leading every reasoning category [Table 2, §4.2].
A prompt-rewrite pipeline (GPT-4o reasons over the prompt → SD3-medium renders) lifts SD3-medium from 0.45 to 0.58 overall (+0.13); causal reasoning gains the most (+0.58, 0.18 → 0.76) and commonsense gains +0.21, but compositional/numerical/mathematical gains are modest (+0.11 / +0.13 / +0.03), and the pipeline still fails on compositional cases like “a surreal waterfall cascades upwards” where SD3-medium renders downward flow despite a correct GPT-4o rewrite [Table 2, §4.2, Fig. 4].
R2I-Score outperforms CLIPScore, DSGScore, VIEScore, and VQAScore on alignment with human pairwise preferences — Pairwise Accuracy 0.713, Kendall’s τ 0.747, Spearman 0.694 vs the second-best metric (VIEScore) at 0.694 / 0.494 / 0.451 [Table 3, §4.3].
Failure-mode breakdown across Emu3, SD3-medium, Show-o+PARM, gpt-image-1: reasoning errors dominate (>80% of errors) for every model; Show-o+PARM additionally has more basic-element errors; gpt-image-1 has the lowest rates of both basic-element and image-quality errors [§4.4, Fig. 5].
The authors hypothesize the open-source failure mode (e.g. “either a spoon or a bowl” → both rendered) is rooted in CLIP-based conditioning’s bag-of-words encoding, but explicitly leave a formal investigation to future work [§4.2].

Method

R2I-Bench is built via a three-stage human-in-the-loop pipeline. (1) Data collection: five domain experts identify the seven primary reasoning categories from prior T2I-reasoning work, then refine them into 32 fine-grained subcategories; for each subcategory GPT-4o is prompted in-context (three positive + three negative human-authored exemplars) to generate (T2I prompt, reference caption, explanation description) triples under two guardrails — prompts may not directly describe visual features, and the visual elements must be uniquely identifiable. (2) Filtering and refinement: manual inspection retains ~800 of an initial 3,200 instances as seeds; experts then iteratively prompt GPT-4o for new candidates and review/feedback each round until each subcategory reaches ~100 validated instances, totalling 3,068. (3) Evaluation-criteria generation: for every instance, GPT-4o produces a set of evaluation questions, each with a dimension tag (text-image alignment / reasoning accuracy / image quality), an importance weight (constrained to per-dimension ranges that prioritize reasoning), and a scoring criterion; human experts validate and refine all three. At evaluation time R2I-Score feeds the generated image plus each question and its scoring criterion to GPT-4o, which emits a per-question score; the aggregate is a weight-normalized sum. The accompanying prompt-rewrite framework (gpt-4o+SD3-medium) is the strongest probe of the upper bound: GPT-4o rewrites the original prompt into a detailed structured description, which SD3-medium renders.

Results

16 models evaluated across four families — diffusion (SD3-medium, Lumina-Image 2.0, Sana-1.5, Lumina-T2I, Omnigen, LLM4GEN, ELLA), autoregressive (EMU3, Janus-Pro-7B, LlamaGen, Show-o), reasoning-enhanced (Show-o+ORM/DPO/PARM), closed-source (DALL-E-3, gpt-image-1) [§4.1].
Top-line scores: gpt-image-1 0.77 > DALL-E-3 0.71 > gpt-4o+SD3-medium pipeline 0.58 > SD3-medium 0.45 > Lumina-Image 2.0 0.42 > Sana-1.5 0.41 > EMU3 0.41 > Omnigen / LLM4GEN 0.40 > Janus-Pro-7B 0.38 > Show-o+PARM 0.38 > Show-o 0.36 > Show-o+DPO 0.36 > Show-o+ORM 0.34 > Lumina-T2I 0.33 > ELLA 0.31 > LlamaGen 0.29 [Table 2].
Concept-mixing is the easiest category for most models — top open-source 0.66 (Sana-1.5); mathematical is the hardest — best open-source 0.19 (SD3-medium), gpt-image-1 0.58 [Table 2].
gpt-4o+SD3-medium pipeline gains by category (vs SD3-medium): commonsense +0.21, causal +0.58, compositional +0.11, concept-mix +0.18, logical +0.10, numerical +0.13, mathematical +0.03 [Table 2].
R2I-Score vs human alignment: Pairwise Accuracy 0.713, Kendall’s τ 0.747, Spearman 0.694; next best (VIEScore) 0.694 / 0.494 / 0.451; CLIPScore lowest [Table 3].

Why it’s interesting

R2I-Bench is the third filed example in 2026 of the “benchmark-time MLLM-as-judge with per-sample structured questions” shape, alongside UniREditBench: A Unified Reasoning-based Image Editing Benchmark (image editing, dual text+GT-image reference) and IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation (infographic reliability, atomic yes/no questions). Where IGenBench uses prompt-decomposed atomic questions and UniREditBench adds a GT image reference, R2I-Bench’s contribution is the explicit reasoning-dimension weight constraint on the rubric: reasoning accuracy is weighted higher than text-image alignment, which is higher than image quality. This is the first filed benchmark to make the weighting itself a design lever for forcing a reasoning-centric score rather than a visual-quality proxy — useful because the wiki has documented (see VLM-as-Evaluator §“Conflict”) that the design space of MLLM-judge rubrics is wide open and the right shape per task is unsettled. The diagnosis that pipeline-based decoupling (GPT-4o → SD3-medium) wins on causal/commonsense but is nearly powerless on mathematical reasoning also matches a pattern from UniREditBench: A Unified Reasoning-based Image Editing Benchmark (UniREdit-Bagel posts +17 over GPT-4o on game-world reasoning specifically because the training pipeline can verify game-world edits exactly): when the verifier or rewriter can render the reasoning step faithfully into text/code, downstream gains are large; when reasoning requires symbolic/structural cognition the model itself does not encode (geometry, number theory), neither rewriting nor PARM-style RL post-training helps.