GenExam: A Multidisciplinary Text-to-Image Exam

TL;DR

GenExam is a 1,000-prompt benchmark that frames text-to-image generation as a multidisciplinary drawing exam — 10 subjects (math, physics, chem, bio, geo, CS, eng, econ, music, history), four-level ISCED-aligned taxonomy, every prompt paired with a ground-truth image and a per-sample set of atomic yes/no scoring points (avg 6.9 per prompt) that sum to 1. Even GPT-Image-1 hits only 12.1% strict score and most open-source models score 0% strict — the benchmark is deliberately designed so a single wrong atom or mislabeled axis fails the whole image. Notable as a per-sample-rubric benchmark in the IGenBench / RISE-Video shape but on multidisciplinary still images, and as a likely target eval for Luma’s image-generation work.

Key claims

Multidisciplinary text-to-image generation is an unsolved capability: GPT-Image-1 leads at 12.1% strict score, Seedream 4.0 at 7.2%, Imagen-4-Ultra at 6.9%, Gemini-2.5-Flash-Image at 3.9%, and every other tested model (FLUX.1 Kontext max, Seedream 3.0, Qwen-Image, HiDream-I1-Full, FLUX.1 dev/Krea, SD3.5 Large, BAGEL ±thinking, Show-o2-7B/1.5B-HQ, BLIP3o-8B/NEXT-GRPO-3B, Janus-Pro, Emu3) scores at or near 0% strict [Table 2].
The relaxed score (0.7·semantic + 0.1·spelling + 0.1·logic + 0.1·readability) opens up the gap between models that strict scores collapse: GPT-Image-1 62.6, Gemini-2.5-Flash-Image 55.9, Imagen-4-Ultra 53.4, Seedream 4.0 53.0, FLUX.1 Kontext max 28.5, Qwen-Image 27.5 [Table 2].
Each prompt has its own rubric — atomic yes/no scoring points (avg 6.9, min 3, max 14) co-authored by GPT-5 and three PhD annotators against a ground-truth image, with score weights summing to 1; semantic correctness is then the sum of scores on questions the judge answers “Yes” [§3.1, §3.3].
Visual plausibility is scored along three 0-2 sub-dimensions — Spelling, Logical Consistency, Readability — orthogonal to semantic correctness, so a model can fail strict on layout/spelling even when all scoring points pass [§3.3].
Evaluator is GPT-5 (gpt-5-2025-08-07, “low” reasoning effort) with both the generated image and the ground-truth image in context; judge MAE vs. five-expert mean is 0.10 on semantic correctness and 0.11/0.28/0.20 on the plausibility dimensions, with relaxed-score Kendall 0.67 / Spearman 0.82 / Pearson 0.84 against human overall ratings [Table 3].
GPT-5-mini at low reasoning effort retains most of GPT-5’s calibration (Kendall ≈ 0.61 vs 0.67) at roughly one-seventh the API cost; minimal reasoning effort drops Kendall to 0.54 [Table on Evaluator ablation, §4.4].
Relaxed score correlates with human raters far better than legacy automatic metrics — Pearson 0.84 (relaxed) vs 0.18 (VQA score) vs 0.16 (CLIP score) — empirically establishing that older CLIP/VQA scorers fail on multidisciplinary images [Table 3].
Data pipeline: ~40K candidate images from web search + MMMU + ScienceQA + TextbookQA + MAVIS + ChEBI + HLE + R-Bench, filtered by GPT-5 on text richness / image domain / complexity / subject-knowledge density down to 6.5K, then PhD-annotator-curated to the final 1K [§3.2].
A 250-sample subset is released alongside the full 1K for cheaper evaluation runs [§3.1].

Method

GenExam treats T2I as a drawing exam in three layers: (i) a hierarchical four-level taxonomy (10 subjects → 40 level-2 → 132 level-3 → 236 level-4 categories) seeded by GPT-5 against ISCED-F fields and refined by PhD annotators; (ii) exam-style prompts (avg 74.8 words) drafted by GPT-5 from ground-truth images and original captions, then re-written by humans to remove imprecise / irrelevant content; (iii) per-sample scoring points — short atomic yes/no questions like “Does the molecule contain exactly 8 carbon atoms?” — generated by GPT-5 with few-shot examples, scored by difficulty so the weights sum to 1.

Evaluation runs MLLM-as-judge with GPT-5: the judge gets the generated image plus the ground-truth image and either (a) answers each scoring point yes/no for semantic correctness, or (b) emits a 0/1/2 rating for each of Spelling, Logical Consistency, Readability. The final per-sample strict score is 1 iff semantic correctness == 1 and all three plausibility dimensions == 2. The relaxed score is the human-aligned weighted average (0.7 / 0.1 / 0.1 / 0.1).

The human-alignment study has five experts label 250 GPT-Image-1 generations along the same protocol; the relaxed score’s Pearson 0.84 against human overall ratings is the headline correlation.

Results

Strict scores (Table 2, overall column): closed-source models cluster around 4–12%, open-source T2I clusters at 0–0.3%, open-source unified MLLMs at 0%. Per-subject, strict scores are highest on Biology (GPT-Image-1 22.8) and lowest on History (GPT-Image-1 2.4), suggesting biology drawings are more pattern-like and harder-to-fake history diagrams expose model weaknesses.

Relaxed scores spread further: GPT-Image-1 62.6 > Gemini-2.5-Flash-Image 55.9 ≈ Imagen-4-Ultra 53.4 ≈ Seedream 4.0 53.0 >> FLUX.1 Kontext max 28.5 ≈ Qwen-Image 27.5 > rest. The 23-point relaxed-score gap between GPT-Image-1 and the best open-source model (Qwen-Image, which uses a Qwen2.5-VL feature extractor) is the cleanest current headline number for the open-vs-closed gap on knowledge-intensive image generation.

Human-alignment table validates the relaxed score against five-expert mean overall ratings on 250 GPT-Image-1 generations: Kendall 0.6746, Spearman 0.8217, Pearson 0.8444 — strong relative to VQA score (Pearson 0.1786) and CLIP score (Pearson 0.1647).

Evaluator ablation: GPT-5 at low reasoning effort is the quality/cost sweet spot at ~ $0.024/image, ~72s/image; medium and high reasoning effort barely move correlations while doubling-to-tripling cost. GPT-5-mini at low reasoning gives Kendall ≈ 0.61 at ~$ 0.003/image.

Why it’s interesting

Boxiao flagged this as a likely good eval for the team, which is the right read — GenExam is a much harder downstream eval than the in-house prompt-following / aesthetic checks Luma typically runs on image generators. It complements two existing per-sample-rubric benchmarks already on the wiki: IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation, which has the same atomic-yes/no-question rubric shape but for text-to-infographic reliability (and which also pinned Gemini-2.5-Pro as the only judge clearing r=0.8 against humans on infographics), and RISE-Video: Can Video Generators Decode Implicit World Rules?, which carries the same shape over to TI2V. GenExam is the third datapoint in this family and the first to systematically expose the multidisciplinary-knowledge axis.

It also sharpens the open question on VLM-as-Evaluator: GenExam requires both the generated image and the ground-truth image to be fed to the judge, otherwise the judge cannot reliably answer chemistry/music/geometry-style scoring points — an empirical restriction the wiki had not previously documented. Contrast with Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (PhyGenBench) (PhyGenBench) which evaluates video physics without a ground-truth reference video, relying on physics-law decomposition instead.