UniREditBench: A Unified Reasoning-based Image Editing Benchmark

TL;DR

UniREditBench is a reasoning-based image-editing benchmark (2,700 samples across 8 primary and 18 sub-dimensions) that goes beyond single-object attribute edits by adding multi-object real-world interactions and rule-governed game-world scenarios (Sokoban, Sudoku, Tic-Tac-Toe, Maze, Jewel2, Space Invader, Word Search), and adds a ground-truth image reference alongside the textual reference for dual-reference VLM-as-judge scoring. The accompanying UniREdit-Data-100K supplies 100,421 training samples with stepwise chain-of-thought reasoning traces, generated by a two-pipeline data synthesis stack (text-then-image for real-world, Python-program-then-VLM-rewrite for game-world). Fine-tuning Bagel on UniREdit-Data-100K yields UniREdit-Bagel, which posts an overall 78.15 on UniREditBench — the largest gain coming from game-world scenarios (+17.08 over GPT-4o) — and improves the open-source ceiling on RISEBench by 9.1 points over Bagel-Think. The dataset URL Zhifan posted (maplebb/UniREdit-Data-100K) is the 100K training corpus, not the 2,700-sample benchmark.

Key claims

The benchmark covers 8 primary dimensions × 18 sub-dimensions split across real-world scenarios (Single-Object Transformation: viewpoint, pose, temporal, material; Multi-Object Interaction: structural integrity, motion state, mechanical reaction, medium interaction, spatial arrangement) and game-world scenarios (Long-Horizon Planning: Maze, Sokoban; Logical Puzzle Solving: Sudoku, Tic-Tac-Toe, Word Search; Strategic Reasoning: Pacman, Jewel2, Space Invader; Spatial Intelligence), with 150 human-inspected samples per sub-dimension for a total of 2,700 [§3.2].
Dual-reference evaluation: for each sample the evaluator receives both a textual reference of the edited effect and a ground-truth reference image, with GPT-4.1 used as the VLM judge to produce three 1–5 scores — Instruction Following, Visual Consistency, Visual Quality — aggregated as 0.6 / 0.2 / 0.2 [§3.3].
A motivating failure case shows text-only references producing inflated scores when the edited image introduces a new faulty path on top of the requested edit; adding the reference GT image lets the VLM catch the spurious change [§3, Fig. 2].
Real-world data synthesis is text-then-image: handcraft a small triple of (original-image description, editing instruction, edited-effect text), scale up with Gemini VLM, then use GPT-4o to synthesize the original-and-edited image pair from each triple, then quality-filter with Gemini and have it write the CoT [§3.4 (a)].
Game-world data synthesis is Python-then-natural-language: design game problems, write Python programs that emit paired images + instructions + textual references + programmatic CoT traces, then convert programmatic CoT to natural-language CoT via a VLM and quality-filter [§3.4 (b)].
UniREdit-Data-100K contains 100,421 samples with stepwise CoT, balanced so each reasoning category contains at least 4,000 instances [§4.1].
On UniREditBench, closed-source ranking is GPT-4o (73.39 overall) > Nano Banana (68.26) > Wan 2.5 (61.36) > Seedream 4.0 (55.77) > FLUX-Kontext-Pro (45.77); only GPT-4o and Nano Banana clear 60 average on game-world tasks [Table 3.3].
Among open-source baselines, Qwen-Image-Edit leads on real-world (70.95 average) but collapses on game-world (41.92); DreamOmni2 (52.81 overall), UniWorld-V2 (54.87), Bagel-Think (50.96), Step1X-Edit (50.15), Omnigen2 (43.41) follow [Table 3.3].
UniREdit-Bagel posts 78.15 overall — surpassing GPT-4o by 4.76 points, with a +17.08 gain on game-world scenarios (80.48 vs GPT-4o’s 62.07) [Table 3.3, §5.3].
UniREdit-Bagel improves the Bagel-Think baseline on RISEBench by 9.1 points and beats Gemini-2.0-Flash-exp by 5.0 points on the same OOD benchmark, narrowing but not closing the gap with Nano Banana and GPT-4o [§5.3].
The training objective is a joint flow-matching latent loss for the edited image + negative-log-likelihood on the CoT reasoning text, supervising both the visual edit and the reasoning trace under Bagel’s think-then-edit architecture [§4.2].

Method

The benchmark is constructed in three pieces. (1) Dimension hierarchy: a scenario→category split where real-world tasks decompose into Single-Object Transformation (intrinsic per-object changes) and Multi-Object Interaction (mutual physical/spatial changes), and game-world tasks decompose into four reasoning archetypes (long-horizon planning, logical puzzles, strategic reasoning, spatial intelligence) each instantiated by 1–3 concrete game environments. (2) Two-pipeline data synthesis: real-world uses a text-then-image pipeline — Gemini expands hand-crafted instruction triples → GPT-4o renders the original+edited pair → Gemini filters and writes the CoT; game-world uses Python programs that simulate the game, emit the image pair + ground-truth instruction + programmatic CoT, which is then translated to natural-language CoT by a VLM and quality-filtered. (3) Dual-reference evaluation: GPT-4.1 ingests (original image, instruction, generated image, edited-effect text reference, ground-truth edited reference image) and produces three 1–5 dimension scores — Instruction Following (weight 0.6), Visual Consistency (0.2), Visual Quality (0.2). The 100K-scale training corpus (UniREdit-Data-100K) is built by the same two pipelines without the 150-per-subcategory cap; UniREdit-Bagel is then trained from Bagel (all components except the VAE) for 5,000 iterations with Adam + cosine LR, peak 5e-5, warm-up 500, loss weights λ_text and λ_mse tuned per the paper.

Results

UniREdit-Bagel overall 78.15 on UniREditBench, vs GPT-4o 73.39, Nano Banana 68.26, Qwen-Image-Edit 56.52, Bagel-Think 50.96 [Table 3.3].
Real-world average: UniREdit-Bagel 75.74, GPT-4o 81.01, Nano Banana 75.22, Qwen-Image-Edit 70.95 — UniREdit-Bagel trails GPT-4o on real-world but leads all open-source baselines [Table 3.3].
Game-world average: UniREdit-Bagel 80.48 — a +17.08 lead over GPT-4o (62.07) and +20.09 over Nano Banana (60.39); previous open-source ceiling was DreamOmni2 at 48.98, so UniREdit-Bagel improves the open-source state of the art on game-world reasoning by ~31.5 points [Table 3.3, §5.3].
Per-task within game-world: Spatial Intelligence 84.90 (next best closed: GPT-4o 77.73), Strategic Reasoning 72.83 (GPT-4o 51.44), Long-Horizon Planning 84.88 (Nano Banana 56.83), Logic Puzzle Solving 83.72 (Nano Banana 64.91) [Table 3.3].
On out-of-distribution RISEBench: UniREdit-Bagel posts the strongest open-source results across all four RISEBench categories, +9.1 over Bagel-Think baseline and +5.0 over Gemini-2.0-Flash-exp [§5.3].

Why it’s interesting

Two things make this paper actionable for an image/video-generation team. First, the benchmark design — adding a GT image reference alongside the textual reference for VLM-as-judge scoring — directly addresses the failure mode that VLM-as-Evaluator currently has no clean answer for: a frozen MLLM judge with only a text reference will systematically miss extraneous changes that don’t contradict the instruction text. This is the first filed paper that supplies the second reference rather than asking the judge to re-derive ground truth from the instruction alone. Second, the data side — UniREdit-Data-100K is a textbook instance of Synthetic Training Data with a strong verification gate: the real-world half goes through GPT-4o image synthesis + Gemini quality filtering + Gemini-written CoT; the game-world half is generated by Python programs (logically exact by construction), with a VLM only to translate programmatic CoT into natural language. The +17-point game-world gap UniREdit-Bagel opens over GPT-4o specifically on game scenarios is consistent with the broader pattern on the wiki — when the verification gate is exact (Python program emits a known-correct edit pair + CoT), small-model finetuning beats frontier closed models on the same axis. Contrast with IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation (per-sample atomic yes/no questions + Gemini-2.5-Pro as the only reliable judge) and RISE-Video: Can Video Generators Decode Implicit World Rules? (LMM-as-judge over extracted frames with per-sample human-written questions) — UniREditBench, IGenBench, and RISE-Video together form three points in the design space of “benchmark-time MLLM-as-judge over structured per-sample rubrics”, with UniREditBench uniquely contributing the GT-image second reference.