IDEA-Bench: How Far are Generative Models from Professional Designing?

TL;DR

IDEA-Bench is a benchmark for image-generation models on professional design work — 100 real-world design tasks (rendering, VFX, storyboarding, picture books, fonts, style-based and identity-preserving generation) split across five I/O regimes: text→image (T2I), image→image (I2I), multi-image→image (Is2I), text→multi-image (T2Is), and multi-image→multi-image (Is2Is). Evaluation is hierarchical — 275 test cases × 6 binary questions each = 1,650 atomic criteria, with later questions gated on earlier ones passing. The benchmark exposes how far commercial and open generators are from professional design quality: even the best-performing dedicated model scores 22.48/100, while the best general-purpose model scores only 6.81. An 18-task subset comes with MLLM-based auto-evaluation for fast iteration.

Key claims

100 professional design tasks span five I/O regimes — T2I, I2I, Is2I (multi-image→image), T2Is (text→multi-image), Is2Is (multi-image→multi-image) — chosen to stress modes that ControlNets and LoRAs cannot cover [Abstract, §1].
275 test cases × 6 binary evaluation questions = 1,650 atomic criteria; hierarchical scoring gates Q3–Q6 on Q1 ∧ Q2, and Q5–Q6 on Q3 ∧ Q4, so a case can’t earn detail credit without first passing identity/intent checks [Evaluation §Human Evaluation].
Best dedicated model achieves 22.48/100; best general-purpose model only 6.81/100 — a large headroom even before professional designers are in the loop [Abstract].
An 18-task subset is equipped with MLLM-based auto-evaluation using annotator-refined prompts, so model developers can iterate without running the full human protocol every time [Abstract, §Evaluation MLLM Evaluation].
Final score is the mean of five category scores, each itself the mean of task scores within the category — task types a model cannot support are scored 0, so coverage of all five I/O regimes is rewarded [§Evaluation Hierarchical Scoring].

Method

The benchmark is constructed by surveying real-world design and art workflows across multiple platforms and distilling them into 100 representative tasks, grouped into five major categories defined by I/O cardinality (one/many on each side of text and image). Each task is instantiated with multiple cases (275 total), and each case carries six expert-written binary criteria. The criteria are hierarchical: the first two questions cover identity/intent (e.g. “is the requested subject present, is the prompt followed”); the next two cover structural adherence; the last two cover fine detail. Failing an earlier tier zeroes the later tiers, so the score is a strict conjunction of professional expectations rather than a soft average.

Two evaluation paths are provided. Human evaluation walks the full hierarchy on every case. MLLM evaluation covers an 18-task subset where the per-case binary criteria have been rewritten into prompts that an MLLM can score directly; the prompts were iterated by annotators to track human judgement. The MLLM scripts and prompts are released alongside the dataset in the ali-vilab/IDEA-Bench GitHub repo and on Hugging Face.

Results

Best-performing model: 22.48/100 [Abstract]
Best general-purpose (single-model T2I/I2I) baseline: 6.81/100 [Abstract]
Coverage: tasks that a model architecturally cannot perform (e.g. a pure T2I model cannot do Is2Is) are scored 0, dragging single-modality models below the dedicated multi-task systems [§Evaluation Hierarchical Scoring].
Published at CVPR 2025.

Why it’s interesting

For Luma’s Vespa-style evals (alexm’s question in the Slack post), IDEA-Bench gives a ready-made hierarchical rubric covering the multi-image-input and multi-image-output regimes that single-prompt benchmarks like GenEval and DrawBench miss — and that newer in-context editors like Qwen-Image-Edit-2509 (Qwen-Image-Edit-2509 — multi-image editing and enhanced consistency (Qwen)), Seedream 4.0 (Seedream 4.0 — Unified Image Generation and Editing Model (ByteDance Seed)), and FLUX.1 Kontext (Introducing FLUX.1 Kontext and the BFL Playground) are explicitly built for. It sits alongside EdiVal-Agent: An Object-Centric Framework for Automated, Scalable, Fine-Grained Evaluation of Multi-Turn Editing (multi-turn editing eval) and GenExam: A Multidisciplinary Text-to-Image Exam (text-to-image exam-style eval) as a third axis — professional-design coverage rather than fine-grained editing accuracy or knowledge-grounded factuality. The hierarchical-gating scoring also makes it a natural complement to VLM-as-Evaluator work like Unified Personalized Reward Model for Vision Generation, where the prompt-decomposed atomic-question pattern is the dominant judging shape; IDEA-Bench’s questions are pre-written by experts rather than dynamically generated, so it pins down a fixed reference rubric the dynamic methods can be evaluated against.