IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation

TL;DR

IGenBench is the first benchmark for text-to-infographic generation — 600 curated prompts spanning 30 infographic types across 6 categories, with an automated evaluation framework that decomposes per-image reliability into atomic yes/no verification questions over a taxonomy of 10 question types (Title, Chart Type, Decorations, Annotations, Axes, Legend, Data Marks, Data Completeness, Data Ordering, Data Encoding). Each generated image is scored at two aggregation levels — Q-ACC (fraction of satisfied questions) and I-ACC (fraction of fully-correct infographics) — by an MLLM judge (Gemini-2.5-Pro). Across 10 SOTA T2I models, the authors find a three-tier hierarchy: Nanobanana-Pro (Q-ACC 0.90 / I-ACC 0.49), a middle tier of Seedream-4.5 + GPT-Image-1.5 (Q-ACC ~0.55–0.61, I-ACC 0.06–0.12), and a long tail (FLUX.1-dev, HiDream-I1, Image-01) at Q-ACC ≤ 0.13 and I-ACC ≈ 0. Data-related dimensions (Completeness 0.21, Encoding 0.26, Ordering 0.27 averaged) are the universal bottleneck — current T2I models render decorations and titles competently but cannot reliably encode numerical values into faithful visual marks.

Key claims

IGenBench contains 600 prompts spanning 30 infographic types across 6 high-level categories (Composition, Trend & Evolution, Categorical Comparison, Deviation & Gap, Correlation & Flow, Bonus / multi-panel), built by clustering and stratified sampling from a 42,315-infographic pool drawn from Statista, Visual Capitalist, and the real-world subset of ChartGalaxy [§3.1].
The evaluation framework decomposes each prompt + image into atomic yes/no questions over a taxonomy of 10 question types, sourced from (i) prompt-derived constraints and (ii) expert-informed seed dimensions covering Data Completeness, Data Ordering, and Data Encoding, yielding 5,259 verification questions in total (~7–11 per case) [§4.1, Fig. 3].
The verifier is an MLLM applied to each (question, image) pair with a strict binary score; ambiguity, partial satisfaction, or missing visual evidence all score 0 [§4.1].
Across 10 evaluated T2I models, three tiers emerge: Nanobanana-Pro at Q-ACC 0.90, Seedream-4.5 / GPT-Image-1.5 at 0.61 / 0.55, and the remaining 7 models at Q-ACC ≤ 0.48 [Table in §5.2].
High Q-ACC does not imply reliable infographics: the best model (Nanobanana-Pro) attains only 0.49 I-ACC, and second-tier models collapse to 0.06–0.12 I-ACC despite Q-ACC > 0.55 — a “long-tail” failure mode where one or two wrong dimensions invalidate the whole image [§5.2].
Data-encoding dimensions are the universal bottleneck across all 10 models: Data Completeness averages 0.21, Data Encoding 0.26, Data Ordering 0.27, versus 0.66 for Decorative elements and 0.54 for Title [Table in §5.2].
Gemini-2.5-Pro is selected as the MLLM evaluator after a study against 12 candidate judges (9 open-source + 3 closed); it is the only model exceeding a 0.8 Pearson correlation with human judgments (r = 0.90), with GPT-5-mini (0.70) and GLM-4.5v (0.75) trailing [§5.4, Fig. 6].
Per-subset (n=25) bootstrapping of automatic vs human scores yields Pearson r > 0.9 (p < 0.001), validating that the MLLM judge tracks human reliability ratings at aggregate level [§5.3, Fig. 4a].
IGenBench rankings correlate with but diverge from LMArena (Spearman r ≈ moderate-to-strong, p < 0.05 on 7 shared models): Seedream-4.5 climbs from LMArena 4th to IGenBench 2nd, GPT-Image-1.5 drops from LMArena 1st to IGenBench 3rd — natural-image quality is necessary but not sufficient for infographic reliability [§5.3].
Prompt token lengths in IGenBench span tens to several thousand tokens, 1–2 orders of magnitude longer than typical T2I benchmarks (OneIGBench, EvalMuse), reflecting the in-prompt data-table specification needed for faithful infographic generation [§4.3, Fig. 3a].

Method

The pipeline has three stages. (1) Dataset construction: collect 42,315 real-world infographics from Statista, Visual Capitalist, and ChartGalaxy; assign each to one of 30 chart types under a 6-category taxonomy via MLLM classification; cluster intra-type and apply cosine-similarity-based stratified sampling to remove redundancy; manually verify to land at 600 cases. (2) Prompt synthesis: for each reference infographic, an MLLM extracts a structural design description (“Create an infographic that…”) and the underlying data table separately, both subject to manual verification; the final prompt embeds the data table verbatim at the end (“The given data is: {data}.”). (3) Evaluation: for each (prompt, generated image) pair, the system generates a set of atomic yes/no verification questions — half decomposed directly from prompt constraints and half instantiated from the three expert-informed seed dimensions (Data Completeness, Ordering, Encoding) — then Gemini-2.5-Pro scores each binarily, aggregated into Q-ACC (per-question accuracy) and I-ACC (fraction of images that pass all their questions).

Results

Per-model overall: Nanobanana-Pro 0.90 Q-ACC / 0.49 I-ACC; Seedream-4.5 0.61 / 0.06; GPT-Image-1.5 0.55 / 0.12; Nanobanana 0.48 / 0.02; Qwen-Image 0.36 / 0.01; Z-Image-Turbo 0.35 / 0.00; P-Image 0.34 / 0.00; Image-01 0.13 / 0.00; HiDream-I1 0.11 / 0.00; FLUX.1-dev 0.10 / 0.00 [Table in §5.2].
Per-question-type averages across all models: Data Completeness 0.21, Data Encoding 0.26, Data Ordering 0.27, Data Marks 0.35, Annotations 0.40, Axes 0.40, Legend 0.46, Chart Type 0.49, Title 0.54, Decorative 0.66 [Table in §5.2].
Judge-vs-human agreement: Pearson r = 0.90 between Gemini-2.5-Pro scores and expert human scores over 100 bootstrap subsets of 25 samples (p < 0.001) [§5.3, Fig. 4a].
MLLM-judge calibration: only Gemini-2.5-Pro clears the 0.8 correlation threshold; GLM-4.5v 0.75, GPT-5-mini 0.70, with most open-source MLLMs below 0.5 [§5.4, Fig. 6].
Cross-benchmark: Spearman correlation against LMArena ranks on 7 shared models is moderate-to-strong and significant, but Seedream-4.5 and GPT-Image-1.5 swap positions, showing infographic reliability is a partially-orthogonal axis to natural-image generation quality [§5.3].

Why it’s interesting

For a team working on image and video generation, IGenBench is two useful things at once. First, it is a stress test for the structured-rendering capability gap — T2I models that look great on photographs collapse on infographic tasks specifically because they cannot bind numerical values to visual mark sizes, and the I-ACC numbers quantify how far that gap runs (even the best closed model gets <50% of its infographics fully right). Second, the methodology is a textbook instance of the VLM-as-Evaluator paradigm: rather than a fixed scalar scorer like CLIP or HPSv3, the rubric is prompt-derived (atomic questions decomposed from the input specification) plus seed-augmented (three expert-defined dimensions instantiated per-image) and the judge is a frozen MLLM — directly analogous to the per-sample question-answering shape that RISE-Video: Can Video Generators Decode Implicit World Rules? uses on the video side. The judge-selection study (only Gemini-2.5-Pro reliably correlates with humans here) is a useful data point for anyone choosing an MLLM-as-judge backbone.