Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

TL;DR

Qwen-Image-Bench is Alibaba Qwen’s open T2I evaluation suite built around Q-Judger, a Qwen3.6-27B model fine-tuned to score generated images along a three-level hierarchy: 5 L1 dimensions (Quality, Aesthetics, Alignment, Real-world Fidelity, Creative Generation), 23 L2 sub-capabilities, and 56 L3 facets, covering 1,000 evaluation prompts. The judge ingests (prompt, image) pairs together with the benchmark’s per-prompt facet metadata, runs deterministic CoT inference (temperature=0, top_k=1, seed=42, enable_thinking=True), and emits structured JSON scores that aggregate bottom-up (L3 → L2 → L1 → Overall). The full package — paper, judge weights, evaluation dataset, scoring code, and an ms-swift inference pipeline — is shipped Apache-2.0. The released leaderboard ranks 18 T2I models, with GPT Image 2 on top (Overall 64.69) followed by Nano Banana 2.0 (59.82), GPT Image 1.5 (59.65), Nano Banana Pro (59.45), and Qwen Image 2.0 Pro (57.84).

Key claims

The evaluation framework is a three-level hierarchy of 5 L1 dimensions × 23 L2 sub-capabilities × 56 L3 facets, with each L3 facet scored on a 0/1/2 scale (Fail → 0, Pass → 60, Excel → 100, with N/A excluded) and aggregated bottom-up [§Evaluation Framework]. The five L1 dimensions explicitly separate Creative Generation (Imagination, Feature Matching, Logical Resolution, Text Rendering, Design Applications, Visual Storytelling) from Real-world Fidelity (Fairness, Safety & Compliance, World Knowledge) — the structural choice the title alludes to (“From Generation to Creation”).
The judge is a fine-tuned Qwen3.6-27B (Q-Judger) released as an open Apache-2.0 model, intended to score arbitrary T2I outputs against the 1,000 benchmark prompts and emit JSON-formatted multi-dimensional scores [README §Quick Start, §Inference Parameters]. Inference is fixed for reproducibility: seed=42, temperature=0, top_k=1, top_p=1.0, repetition_penalty=1.05, max_new_tokens=4096, enable_thinking=True, max_batch_size=24.
Released artifacts span the full reproducibility stack: paper (arXiv 2605.28091), Q-Judger weights (HF Qwen/Qwen-Image-Bench), evaluation dataset (HF datasets/Qwen/Qwen-Image-Bench + ModelScope mirror), evaluation code (judge.py, compute_scores.py), and an ms-swift PtEngine inference backend [README §Project Structure]. The dataset doubles as the source for pre-generated responses, which compute_scores.py can score without re-running any T2I model [README §Compute Scores from Pre-generated Responses].
Top-5 leaderboard from the released results — GPT Image 2 leads at Overall 64.69, followed by Nano Banana 2.0 (59.82), GPT Image 1.5 (59.65), Nano Banana Pro (59.45), and Qwen Image 2.0 Pro (57.84) [README §Top-5 Models]. Per-L1: GPT Image 2 leads on every L1 dimension (Quality 58.65, Aesthetics 67.53, Alignment 65.85, Real-world Fidelity 57.38, Creative Generation 75.23). Full per-model results for all 18 evaluated T2I models are in the paper.
The judge ingestion contract is (ID, prompt, image_path) where the ID indexes into the benchmark’s per-prompt facet metadata (bench_metadata.json), so the rubric instantiated for any one image is determined by the prompt’s pre-tagged facets rather than dynamically inferred at judge time [README §Quick Start]. This is the “static expert-seeded rubric” half of the per-prompt rubric design.

Method

The benchmark consists of 1,000 evaluation prompts, each pre-tagged in bench_metadata.json with the subset of the 56 L3 facets it should be scored against. A T2I model generates one image per prompt; the user produces a JSONL of (ID, prompt, image_path); judge.py runs the Q-Judger model (Qwen3.6-27B with enable_thinking=True) on each row using ms-swift’s PtEngine, with greedy decoding fixed at seed=42, temperature=0, top_k=1 for reproducibility. The judge emits a structured JSON object per L1 dimension containing per-L3 scores (0/1/2 or N/A); these are aggregated bottom-up — L3 mean → L2 → L1 → Overall — with 0/1/2 mapped to 0/60/100.

Three output files are written: a per-row JSONL/CSV with the raw judge output preserved (<input>_judged), an aggregated scores JSON (<input>_bench_scores.json), and an Excel workbook with L1 summary + per-L1 sheets carrying L2 detail. A separate compute_scores.py re-aggregates pre-generated responses from the released HF dataset, allowing reproduction of the leaderboard without re-running the 18 T2I models.

Results

The Top-5 leaderboard from the README, with per-L1 breakdown:

Model	Quality	Aesthetics	Alignment	Real-world Fidelity	Creative Generation	Overall
GPT Image 2	58.65	67.53	65.85	57.38	75.23	64.69
Nano Banana 2.0	54.77	61.08	62.40	54.28	67.05	59.82
GPT Image 1.5	55.14	60.88	61.72	53.95	66.35	59.65
Nano Banana Pro	55.67	60.26	61.25	54.07	66.23	59.45
Qwen Image 2.0 Pro	54.39	58.67	59.28	51.83	64.94	57.84

Two structural observations: (1) the spread on Creative Generation (75.23 vs 64.94 in the top 5, a 10.3-point gap) is roughly twice the spread on Real-world Fidelity (57.38 vs 51.83, a 5.6-point gap), consistent with the design intent of using Creative Generation as the harder discriminator. (2) GPT Image 2 leads on every L1 dimension, not just Overall — there is no L1 where an open or alternative-closed model overtakes it in the Top-5.

Why it’s interesting

Qwen-Image-Bench is the first judge-model-as-release to ship under Apache-2.0 — earlier filed VLM-judge work in the wiki either ships only a benchmark + closed-judge protocol (IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation with Gemini-2.5-Pro as the only judge clearing r=0.8 against humans on infographics) or ships a reward model trained for GRPO post-training (Unified Personalized Reward Model for Vision Generation UnifiedReward-Flex, with per-prompt dynamic rubrics for FLUX-family GRPO). Q-Judger occupies the third slot: a benchmark-time T2I judge with a fixed, pre-tagged per-prompt rubric, released as open weights — the missing reproducible analogue to the IGenBench protocol. It also contrasts cleanly with GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation, which combats benchmark drift via prompt rewriting + open vs closed splits while still leaving the judge model as an external dependency; Qwen-Image-Bench’s response to drift is instead to ship the judge itself.

The 56-facet hierarchy with explicit Creative Generation as a top-level dimension is a structural answer to a problem IDEA-Bench: How Far are Generative Models from Professional Designing? raised about “professional designing” tasks — IDEA-Bench framed the gap as a multi-modal I/O regime problem, while Qwen-Image-Bench frames it as a capability-axis problem with Imagination / Logical Resolution / Visual Storytelling as named facets. Worth tracking how the open Q-Judger correlates with humans relative to the closed Gemini-2.5-Pro baseline that IGenBench identified as the only reliable judge — the paper presumably reports this, but it isn’t in the README excerpts and the arXiv body was rate-limited at filing time.