GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation

TL;DR

GenEval, one of the most-used T2I benchmarks, has silently drifted from human judgment as T2I models have improved — the static object-detector-based judge now disagrees with humans by up to 17.7% absolute error on current models, masking the fact that GenEval is saturated. The paper argues this drift is endemic to any benchmark whose prompts must be challenging to the generator but not the judge, then introduces (a) GenEval 2, 800 compositional prompts spanning objects/attributes/relations/counting at varying compositionality, and (b) Soft-TIFA, a per-atom VQA judge that decomposes the prompt into visual primitives and aggregates per-atom probabilities — claimed to be both better human-aligned (94.5% AUROC vs 92.4% VQAScore, 91.6% TIFA with Qwen3-VL-8B) and inherently more resistant to drift than holistic VQA judges.

Key claims

GenEval has drifted: absolute disagreement between GenEval’s object-detector judge and human raters reaches 17.7% on current T2I models, masking saturation [§1, Fig. 1].
The drift is structural: the failure cases include the COCO-trained detector mislabeling a correctly-placed toothbrush as a bird because of dataset prior mismatch [§1, Fig. 1 caption].
GenEval 2 prompts: 800 prompts with compositionality N=3..10 covering objects, attributes (color/material), spatial relations, and counts; each template prompt optionally rewritten via GPT-4o into longer, context-rich forms to stay realistic [§3].
Soft-TIFA: per-prompt the rubric is decomposed into atoms; for each atom a templated VQA question is posed to a frozen VQA model and a soft probability P_M(a_i | image, q_i) is computed; aggregate via geometric mean over N atoms [§4].
Soft-TIFA with Qwen3-VL-8B reaches 94.5% AUROC on human-judgment prediction on GenEval 2, vs 92.4% for VQAScore and 91.6% for TIFA using the same VQA backbone [§5, Table — abstract numbers].
Soft-TIFA with an open 8B Qwen3-VL judge beats GPT-4o-based VQAScore (91.2% AUROC) — open VQA + per-atom aggregation can outperform a closed holistic judge [§5].
Drift mechanism hypothesis: holistic VQA judges (VQAScore, TIFA) accumulate distribution-shift error as generators improve; per-atom decomposition reduces the surface area of each judge call so the underlying VQA model is less affected by image distribution shift [§5, discussion].

Method

GenEval 2 builds 800 compositional prompts at varying difficulty N=3..10, covering four visual primitive categories (objects / attributes / relations / counting). Each templated prompt can be paraphrased via GPT-4o into longer, context-rich forms to better match how real users prompt T2I systems. Soft-TIFA evaluates a (prompt, image) pair by (i) extracting per-prompt atoms (one per visual primitive), (ii) instantiating a templated yes/no VQA question per atom, (iii) querying a frozen VQA model — Qwen3-VL-8B as the default — for the soft probability of the affirmative answer, and (iv) aggregating per-atom probabilities via geometric mean P_GM = (∏ P_M(a_i))^{1/N}, with per-atom probabilities also exposed for diagnostic use.

Results

Soft-TIFA / Qwen3-VL-8B: 94.5% AUROC vs human judgment on GenEval 2 prompts [§5].
VQAScore (same Qwen3-VL-8B backbone): 92.4% AUROC [§5].
TIFA (same backbone): 91.6% AUROC [§5].
VQAScore with GPT-4o backbone: 91.2% AUROC — beaten by Soft-TIFA with an 8B open judge [§5].
GenEval drift on current T2I models: up to 17.7% absolute error vs human ratings [§1].
Benchmark size and composition: 800 prompts, compositionality range N=3..10, four primitive categories [§3].

Why it’s interesting

Drops directly into the wiki’s growing tension on VLM-as-judge (VLM-as-Evaluator): IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation introduced the “prompt-decomposed atomic yes/no” rubric for infographics and ran the 12-MLLM judge-selection study where Gemini-2.5-Pro is the only judge above r=0.8; Soft-TIFA is essentially the T2I sibling of that pattern with an explicit drift-robustness story and the surprising result that an open 8B VQA model beats GPT-4o-based VQAScore once you aggregate per-atom. Also a useful counterpoint to Unified Personalized Reward Model for Vision Generation, which argues for a single context-adaptive judge with self-generated criteria — Soft-TIFA shows that fixing the rubric to templated atomic questions yields better drift resistance than letting the VLM regenerate criteria per call. Finally, the diagnosis itself — that any benchmark whose prompts are co-selected with a fixed judge will silently saturate — is the kind of evaluation-infrastructure result that retroactively reframes a lot of headline T2I numbers from the past two years (GenExam: A Multidisciplinary Text-to-Image Exam, Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions, reported GenEval/T2I-CompBench scores throughout).