UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits

TL;DR

UnicEdit-10M is a 10M-scale instruction-based image-editing dataset built by a three-stage pipeline (Qwen2.5-VL-72B writes instructions → FLUX.1-Kontext and Qwen-Image-Edit synthesize edited images → a 7B dual-task expert “Qwen-Verify” both filters failed edits and recaptions instructions to match the actual visual change). The dataset covers 22 fine-grained sub-tasks across Object / Attribute / Scene / Reasoning Editing (3.529M attribute, 3.242M object, 3.063M scene, 1.746M reasoning) and beats prior open datasets (SEED-Data-Edit, ImgEdit, X2Edit, NHR-Edit, GPT-Image-Edit-1.5M, Nano-consistent-150k) on VIEScore overall (8.08) and aesthetics (8.00 source / 7.76 target). The companion benchmark UnicBench (50 cases per sub-task, hybrid VLM-then-human curated) introduces four metrics — Instruction Following (IF), Non-edit Consistency (NC), Visual Quality (VQ), and Reasoning Accuracy (RA) — aggregated as a geometric mean, designed to fix VIEScore’s blind spot for unintended changes in non-edited regions. The paper sits on a familiar fault line: closed-source GPT-Image-1 / Seedream 4.0 / Nano Banana lead overall, but all models — open and closed — drop sharply on RA, pinpointing reasoning-driven edits as the open problem.

Key claims

The data pipeline replaces multi-toolchain pseudo-labeling (SAM, Grounding-DINO, IP2P-style chains in UltraEdit / ImgEdit / Step1X-Edit) with an end-to-end editing model + a unified post-verification stage, avoiding the error-propagation failure mode of toolchains [§1, §3].
Qwen-Verify is a 7B Qwen2.5-VL-7B fine-tuned in two stages: SFT on 200k human-screened Normal + No-Edit triplets, then a Differential DPO (D-DPO) stage on 20k preference pairs that adds Hallucination correction; the policy is conditioned on a “visual differential context” v_diff extracted from the (source, edited) pair rather than treating the two images as static context [§4.1, §4.2, Eq. 1-3].
Qwen-Verify outperforms Qwen2.5-VL-72B as a verifier (Normal Acc 6.32 vs 5.25, No-Edit Acc 9.80 vs 9.60, Hallucination Acc 6.22 vs 6.12) while being ~10× cheaper to run [Table 5].
The pipeline filters ~26% of generated triplets as failed edits, and the final dataset is ~11.6M triplets organized into 22 sub-tasks; the raw post-instruction-generation volume was 22.4M [Table 3].
UnicEdit-10M leads on dataset-quality metrics: VIEScore overall 8.08 (vs Nano-consistent-150k 7.75, GPT-Image-Edit-1.5M 7.74, NHR-Edit 7.78); PQ 8.20 (best); Aesthetics source 8.00 and target 7.76 (both best, with the source advantage attributed to pre-filtering by aesthetic score) [Table 2].
Face-consistency comparison head-to-head with GPT-Image-Edit-1.5M: 0.89 vs 0.30, attributed to the recaptioning stage matching instructions to the actual visual change rather than asking the editor to follow the original prompt [§6.2].
UnicBench introduces four metrics replacing VIEScore’s two — IF (instruction-following, cross-modal VLM alignment), NC (Non-edit Consistency, penalizes unintended changes outside the target region), VQ (instruction-conditioned naturalness), and RA (Reasoning Accuracy, evaluates whether semantic/causal outcomes implied by the instruction were realized); overall score is the geometric mean over the relevant subset (IF·NC·VQ for Basic Edits, IF·NC·VQ·RA for Complex Edits) [§5.2, Eq. 4].
On UnicBench-EN, ranking is GPT-Image-1 8.35 > Seedream 4.0 8.04 > Nano Banana 7.88 > SeedEdit 3.0 7.87 > Qwen-Image-Edit 7.73 > BAGEL 6.98 > Step1X-Edit-v1.1 6.92 > FLUX.1-Kontext 6.80 > NextStep-1 6.41 > OmniGen2 6.12 > UniWorld-v1 5.60 > Instruct-Pix2Pix 2.92 > MagicBrush 2.34 [Table 4].
The RA metric drops sharply across all models — GPT-Image-1 8.34, Seedream 4.0 7.60, Nano Banana 6.87, Qwen-Image-Edit 6.45, BAGEL 5.26 — diagnosing reasoning-driven edits (knowledge/inference-conditioned) as the residual gap even for frontier closed models [Table 4, §6.4].
Editing taxonomy spans 4 categories × 22 sub-tasks: Object Editing (Subject Addition/Removal/Replacement, Portrait Editing, etc.), Attribute Editing (Color Alteration, Material Modification, Texture Editing), Scene Editing (Background Change, Tone Transformation), Reasoning Editing (Spatial Reasoning Edits, Multi-object Coordination, Knowledge-Driven Edits). Tasks requiring spatial awareness or factual knowledge are classified as Complex Edits [§3.1, Appendix A].
Source images are drawn from a “large-scale internal library” pre-filtered for high aesthetic scores; image-editing modules are FLUX.1-Kontext + Qwen-Image-Edit (both leading open-source editors), with center-crop/resize preprocessing and a 20%-crop quality gate [§3.1, §3.2].

Method

The pipeline has three stages. (1) Instruction generation: Qwen2.5-VL-72B ingests a high-aesthetic source image and the 22-class taxonomy, emitting 3–7 instruction candidates per image with a balanced sub-task distribution. (2) End-to-end editing: each ⟨image, instruction⟩ pair is routed to FLUX.1-Kontext or Qwen-Image-Edit, producing a raw triplet. (3) Unified post-verification (Qwen-Verify): a 7B dual-task expert ingests the original + edited images, captions both, decides via CoT whether a valid edit occurred (binary is_changed), and — if valid — recaptions the instruction to precisely describe the actual visual change. The output is a structured JSON object. Qwen-Verify is trained from Qwen2.5-VL-7B: SFT on 200k Normal + No-Edit triplets, then D-DPO on 20k pairs that include a Hallucination split (the target is correct but the action/attribute was misstated; GPT-4o generates candidate corrections, human experts review). D-DPO conditions the policy on a learned visual-differential embedding v_diff = f_φ(I_src, I_edit) extracted by a visual encoder, then maximizes a Bradley-Terry preference margin via the standard DPO re-parameterization — distinct from vanilla DPO in that the conditioning is the transformation between images rather than the images themselves. UnicBench-construction is hybrid: Qwen2.5-VL writes candidate instructions, human experts review/rewrite to align with the per-task taxonomy, 50 cases per sub-task. Evaluation runs through gpt-4.1-2025-04-14 with the four IF/NC/VQ/RA metrics aggregated as a geometric mean.

Results

Dataset quality vs prior open editing datasets (1k-sample GPT-4o protocol, three-run average): UnicEdit-10M VIEScore 8.45 / 8.20 / 8.08 (SC/PQ/Overall), Aesthetics 8.00 / 7.76 (source/target) — best PQ, best overall, best aesthetics; GPT-Image-Edit-1.5M leads on SC (8.68 vs 8.45) due to its own recaptioning pipeline, but its face-consistency score is 0.30 vs UnicEdit-10M’s 0.89 [Table 2, §6.2].
Qwen-Verify vs Qwen2.5-VL-72B: Normal 6.32 vs 5.25, No-Edit 9.80 vs 9.60, Hallucination 6.22 vs 6.12 — and the SFT-only Qwen2.5-VL-7B+SFT scores 5.62 / 9.40 / 5.47, so D-DPO contributes the larger lift on the Normal and Hallucination tracks [Table 5].
UnicBench-EN overall: GPT-Image-1 8.35, Seedream 4.0 8.04, Nano Banana 7.88, SeedEdit 3.0 7.87 lead closed-source; Qwen-Image-Edit 7.73 leads open-source; BAGEL 6.98 and Step1X-Edit-v1.1 6.92 follow. Instruct-Pix2Pix 2.92 and MagicBrush 2.34 anchor the bottom [Table 4].
UnicBench-CN (subset of bilingual models): Qwen-Image-Edit 7.78, BAGEL 7.11, Step1X-Edit-v1.1 7.06, Nano Banana 8.04, Seedream 4.0 8.05, GPT-Image-1 8.45 — broadly tracks EN ranking with a small CN-side advantage for closed-source models [Table 4].
Reasoning Accuracy (RA) drops sharply across the board: GPT-Image-1 8.34 (only model above 8), Seedream 4.0 7.60, Nano Banana 6.87, Qwen-Image-Edit 6.45, BAGEL 5.26, FLUX.1-Kontext 5.50, NextStep-1 6.29. Most open models lose 1.5–2.0 points on RA vs their IF score — the headline diagnosis [Table 4, §6.4].
Direct VIEScore comparison: UnicBench’s NC metric correctly penalizes unintended person-removal and accidental text modifications that VIEScore’s SC continues to score highly, supporting the claim that VIEScore conflates instruction-following with non-edit consistency [§6.4, Appendix D].

Why it’s interesting

Direct overlap with two ongoing wiki threads. First, this is the largest filed instance of the “verification-gate is the load-bearing piece” thesis from Synthetic Training Data — Qwen-Verify is trained specifically for the dual-task of failure-filtration + instruction-recaptioning, with a novel D-DPO objective that conditions on a learned visual-differential embedding rather than treating the source/edited pair as static context. The 26% filtration rate + the 0.89-vs-0.30 face-consistency delta over GPT-Image-Edit-1.5M are the cleanest filed evidence that the gate’s quality (not just its presence) determines downstream training utility. Second, this lands directly on the VLM-as-Evaluator page: UnicBench’s four-metric decomposition (IF / NC / VQ / RA) is the most explicit answer yet to the VIEScore non-edit-consistency blind spot the page already tracks, and the RA metric — a VLM checking whether the semantic/causal outcome of a knowledge-driven instruction was realized — is a sibling to the dual-reference (text + GT image) judge protocol in UniREditBench: A Unified Reasoning-based Image Editing Benchmark. The two papers cover the same problem space (reasoning-enriched image editing benchmarks) with complementary takes: UniREditBench supplies a GT image reference and a smaller 2,700-sample human-curated test set with game-world scenarios, while UnicBench has no GT-image reference but produces a finer-grained 4-metric breakdown and ships with the matching 10M-scale training corpus. Together they bound the design space — the obvious next experiment is fine-tuning a Bagel-class open editor on UnicEdit-10M and evaluating on both UnicBench and UniREditBench to isolate scale (10M vs 100K) from reasoning-curriculum richness (text recaption vs CoT trace + GT image). Tangentially relevant to FireRed-Image-Edit: A General-Purpose Image Editing Model (FireRed’s REDEdit-Bench is another sibling editing benchmark on the wiki).