EdiVal-Agent: An Object-Centric Framework for Automated, Scalable, Fine-Grained Evaluation of Multi-Turn Editing

TL;DR

EdiVal-Agent is an object-centric, agentic evaluation framework for instruction-based image editing — and crucially, the first filed benchmark that scores multi-turn editing trajectories rather than single-shot edits. It decomposes each input image into semantically meaningful objects, synthesizes diverse context-aware editing instructions while dynamically updating object pools across turns, then computes three metrics: EdiVal-IF (instruction following, Grounding-DINO symbolic checks + Qwen2.5-VL semantic verification), EdiVal-CC (content consistency, DINOv3 + pixel-L1 over Grounding-DINO regions for unchanged objects/background), and EdiVal-VQ (visual quality, HPSv3 human-preference model). EdiVal-Bench covers nine instruction types and thirteen state-of-the-art editors spanning in-context autoregressive, flow-matching, and diffusion paradigms; Seedream 4.0 leads overall (EdiVal-O 83.81 / 69.95 / 59.76 across three turns), Nano Banana is the fastest high-performer (9.7 s/img), and Qwen-Image-Edit shows the sharpest turn-over-turn drop — diagnosed as exposure bias from training only on single-edit pairs.

Key claims

Multi-turn instruction-based editing has been empirically unmeasurable: prior protocols depend on paired reference images (limited coverage, biases from prior generative models) or rely on zero-shot VLMs whose prompt-based assessments are imprecise [§1, project page].
An object-centric agent stack — decompose into objects, synthesize edit instructions, dynamically update object pools across turns — lets the same evaluator handle both single-turn and multi-turn editing while keeping all metrics interpretable [§2, Framework figure].
EdiVal-IF combines open-vocabulary object detectors (Grounding-DINO) for symbolic existence/attribute checks with VLMs (Qwen2.5-VL) for detector-guided semantic verification — a hybrid that’s distinct from pure VLM-as-judge or pure object-detector approaches [§2, Framework figure].
EdiVal-CC evaluates content consistency by computing semantic similarity (DINOv3 features) and pixel-L1 distance over the evolving object pools, with separate object-consistency and background-consistency tracks; the background mask is derived by excluding all detected object regions [§Content Consistency].
EdiVal-VQ uses HPSv3 as the aesthetic scorer, with an absolute change metric Δ = |HPSturn_i − HPSbase|; the authors note HPSv3 generalized reliably to their generated images where alternatives like RAHF underperformed in this setting [§Visual Quality].
HPSv3 is not folded into the aggregate EdiVal-O score because preferences differ on whether an edited image should strictly preserve the input style or pursue beautification — the team reports Δ separately to disentangle style-fidelity from beautification [§Visual Quality].
Seedream 4.0 is the overall multi-turn leader: EdiVal-O 83.81 / 69.95 / 59.76 across turns 1/2/3, with strong instruction following maintained across turns; latency 15.78 s/img [Results table].
Nano Banana is second overall (EdiVal-O 81.48 / 67.70 / 56.24) but the fastest high-performing model at 9.7 s/img — and shows no single dominant weakness across the IF/CC/VQ axes [Results table, Overview].
GPT-Image-1 has the highest autoregressive EdiVal-IF (T1 73.12, T3 38.35) but the slowest latency (71.30 s/img) and weaker content consistency — favors eye-catching outputs over repeatable edits [Results table, §Multi-turn vs Complex].
Qwen-Image-Edit is the best open-source model on turn-1 EdiVal-IF (72.90, close to closed-source leaders) but collapses by turn 3 (22.55) — the steepest open-source drop, attributed to exposure bias from training on perfect single-edit pairs without exposure to the model’s own multi-turn outputs [§Marginal IF, Results table].
In-context (autoregressive) editors benefit from step-by-step multi-turn execution and outperform complex single-shot prompts; flow-matching models (Qwen-Image-Edit, FLUX.1-Kontext-dev) instead prefer complex single-shot prompts because they avoid repeated self-exposure [§Multi-turn vs Complex, comparison table].
Aside from Qwen-Image-Edit, EdiVal-Bench’s rankings match a major community human-vote leaderboard as of September 12 2025 — supplying external validation for the automated stack [§Results].
HPSv3 is sensitive to over-exposure artifacts and luminance shifts in a way that frozen VLMs are not: Qwen-Image-Edit’s qualitative T2/T3 outputs exhibit elevated luminance and high-frequency bright artifacts, HPS drops from 6.19 → 4.19 → 3.34 turn-over-turn, but VLM-returned visual-quality scores remain consistently above 50 on a [0,100] scale and unchanged across turns [§Visual Quality, qualitative examples].

Method

The framework is a three-stage agentic pipeline. (1) Object decomposition: each input image is decomposed by Grounding-DINO and a VLM (Qwen2.5-VL) into semantically meaningful objects with contextual relationships — e.g. “metal yellow sign” and “metal brown pole” both in the “foreground”. This object pool is the unit of accounting across the whole evaluation trajectory. (2) Editing-scenario generation: given the object pool, the agent synthesizes diverse, grounded editing instructions across nine instruction types (e.g. “Change the color of metal brown pole to gray”, “Remove brick beige house”). For multi-turn evaluation, the object pool is dynamically updated after each edit so that turn-N instructions reference the post-turn-(N−1) state. (3) Three-metric scoring: EdiVal-IF combines Grounding-DINO open-vocabulary detection (does the requested object/attribute exist?) with Qwen2.5-VL semantic verification (does it match the instruction?); EdiVal-CC uses Grounding-DINO to localize unchanged objects across the trajectory, computes DINOv3 feature similarity + pixel-L1 over those regions, and computes background consistency over the per-image background mask (whole image minus all detected object regions); EdiVal-VQ is HPSv3 plus a base-relative Δ to disentangle style fidelity from beautification. EdiVal-O is the geometric mean of EdiVal-IF and EdiVal-CC; HPSv3 is reported separately. The agentic pipeline is tool-agnostic — stronger detectors / VLMs / aesthetic scorers can be swapped in as they emerge.

EdiVal-Bench instantiates this pipeline with nine instruction types and thirteen editor models grouped by paradigm: unknown closed-source (Seedream 4.0, Nano Banana, GPT-Image-1, Gemini 2.0 Flash), flow-matching (FLUX.1-Kontext-max, Qwen-Image-Edit, Step1X-Edit, FLUX.1-Kontext-dev, OmniGen), and diffusion (AnyEdit, UltraEdit, MagicBrush, InstructPix2Pix). Closed-source latencies are measured in the provider’s hosted web UI; open-source latencies on a single NVIDIA A100 GPU with default settings.

Results

Three-turn EdiVal-O (geometric mean of IF and CC): Seedream 4.0 83.81 / 69.95 / 59.76; Nano Banana 81.48 / 67.70 / 56.24; GPT-Image-1 76.96 / 65.34 / 53.81; FLUX.1-Kontext-max 80.79 / 65.29 / 53.04; Qwen-Image-Edit 78.36 / 59.56 / 41.93; Step1X-Edit 75.77 / 55.64 / 38.98; FLUX.1-Kontext-dev 75.61 / 54.91 / 38.71; OmniGen 71.34 / 46.52 / 29.91; InstructPix2Pix at the bottom 53.62 / 26.99 / 12.99 [Results table].
EdiVal-IF turn-1 vs turn-3 drop (multi-turn IF degradation): Seedream 4.0 75.93 → 41.59; Nano Banana 70.70 → 35.35; GPT-Image-1 73.12 → 38.35; Qwen-Image-Edit 72.90 → 22.55 (steepest open-source drop, ≈50.4 absolute); InstructPix2Pix 37.41 → 2.80 [Results table].
EdiVal-CC turn-3 (content consistency after three edits): FLUX.1-Kontext-dev best at 90.22, followed by FLUX.1-Kontext-max 88.40, Nano Banana 89.48, Step1X-Edit 85.21, Seedream 4.0 85.86, Qwen-Image-Edit 77.98, GPT-Image-1 75.50 — the FLUX-Kontext family preserves content best but at the cost of weaker instruction-following [Results table].
Multi-turn (T3) vs single-shot complex (C3) IF: in-context autoregressive editors prefer multi-turn — Nano Banana 35.35 (multi) vs 28.14 (complex); GPT-Image-1 38.35 vs 28.78; Gemini 2.0 Flash 28.42 vs 21.89. Flow-matching editors prefer complex single-shot — Qwen-Image-Edit 22.55 (multi) vs 27.62 (complex); FLUX.1-Kontext-dev 16.61 vs 19.58. Diffusion editors are roughly indifferent or slightly prefer multi-turn [§Multi-turn vs Complex table].
HPSv3 by model (T1 → T3): GPT-Image-1 highest absolute and most stable (6.65 / 6.59 / 6.56), Qwen-Image-Edit second-highest absolute (5.86) but drops to 5.15; Gemini 2.0 Flash least drift from base (Δ 0.05–0.32) so best at preserving input style; Nano Banana climbs across turns (4.94 → 5.26) [§Visual Quality table].
All open-source flow/diffusion editors except Qwen-Image-Edit show large turn-3 HPS drops (Step1X-Edit 4.06 → 2.76, MagicBrush 3.85 → 2.36, AnyEdit 3.66 → 1.95, InstructPix2Pix 3.20 → 1.44) — visible aesthetic degradation that the VLM judge does not catch [§Visual Quality table, qualitative examples].

Why it’s interesting

This is the third filed editing-benchmark paper in roughly a month, alongside UniREditBench: A Unified Reasoning-based Image Editing Benchmark and PICABench: How Far Are We from Physically Realistic Image Editing?, and the only one of the three to score multi-turn editing. The other two test single-shot reasoning or physics-aware editing under a one-step instruction; EdiVal-Bench is the first to surface the exposure-bias collapse in Qwen-Image-Edit (turn-1 SOTA, turn-3 collapse) — a finding that’s invisible to single-shot benchmarks and has direct implications for any team training an open-source editor that competes with closed-source in-context editors.

The agentic scoring stack also extends VLM-as-Evaluator in a way the existing concept page does not cover: EdiVal-IF is the first filed instance of a hybrid detector + VLM judge rather than a pure VLM-as-judge or pure object-detector pipeline. PICAEval (PICABench: How Far Are We from Physically Realistic Image Editing?) constrains the VLM to a human-annotated ROI; UniREditBench (UniREditBench: A Unified Reasoning-based Image Editing Benchmark) gives the VLM a GT reference image; EdiVal-Agent instead uses Grounding-DINO as a symbolic gate before the VLM ever sees the image, which means object-existence checks are deterministic and only the harder semantic checks go to the VLM. The qualitative finding that VLMs miss the Qwen-Image-Edit luminance/over-exposure failure mode (HPS catches it, VLM doesn’t) is the strongest filed datapoint that frozen VLM-as-judges have a systematic blind spot for low-level perceptual quality — which is consistent with ERNIE-Image-Aes: Robust Image Aesthetics Scoring with Balanced Category Generalization‘s broader argument that the previous-generation aesthetic scorers (LAION-Aesthetic, ArtiMuse, UniPercept) have category-specific biases that need calibration.

For the Luma editing/multimodal team, the actionable finding is the autoregressive-vs-flow-matching split on multi-turn-vs-complex prompting: in-context editors gain from being broken into multi-turn chains, but flow-matching editors lose because they don’t safely consume their own previous outputs. Multi-turn training (or self-distillation on the editor’s own multi-turn outputs) is the obvious path to closing the Qwen-Image-Edit exposure-bias gap.