Learning an Image Editing Model without Image Editing Pairs

TL;DR

NP-Edit trains a 4-step image editing diffusion model with zero paired edit data by combining differentiable gradient feedback from a frozen VLM (LLaVA-OneVision-7B) with a distribution matching loss (DMD) to the pretrained text-to-image teacher. The VLM scores each generated edit on two binary questions — edit-success and identity-preservation — producing direct gradients into the generator, while DMD keeps outputs on the real-image manifold. At 4 NFE on GEdit-Benchmark, the 2B student matches or beats much larger few-step baselines (FLUX.1-Kontext 12B, Step1X-Edit 12B, Qwen-Image-Edit 20B) on overall VIEScore and outperforms SFT+Flow-GRPO using the same VLM as reward.

Key claims

Training an image editing model from scratch with only a VLM editing loss diverges to unrealistic outputs; only-DMD trains to a non-editing identity ( $SC=4.93$ ); both losses together are needed ( $SC=6.16$ , $PQ=7.69$ , Overall $6.10$ ) [Table 3].
A two-step unrolled sampling trajectory ( $\sigma_2 < 0.5\sigma_1$ ) is what makes VLM feedback usable — single-step noise-to-edit is too low fidelity, and VLMs give unreliable judgments on noisy intermediate samples, so few-step generators are a prerequisite, not an optimization [§4.2].
Binary cross-entropy over yes/no token logits (restricting normalization to those two tokens) outperforms standard cross-entropy over the full vocabulary as the VLM loss, and a paired identity-preservation question is essential for preventing over-editing [§4.2, Table 3].
NP-Edit at 4 NFE (2B params) beats Turbo-Edit (1B, 4 NFE, also zero-shot) on overall VIEScore $6.10$ vs $3.84$ and matches or exceeds 4-NFE versions of FLUX.1-Kontext-12B ( $5.04$ ), Step1X-Edit-12B-v1.1 ( $6.01$ ), and Qwen-Image-Edit-20B ( $6.06$ ) on GEdit-Benchmark English; it has the highest PQ score ( $7.69$ ) of any model in the table including multi-step baselines [Table 1].
Differentiable VLM gradient feedback beats Flow-GRPO (RL with the same Llava-OneVision reward) — SFT+RL Overall $4.19$ vs SFT+NP-Edit Overall $6.06$ — and obviates the SFT warm-start that RL methods need [Table 5].
Performance scales monotonically with VLM size (LLaVA-0.5B → 7B: Overall $4.59 → 6.10$ ) and dataset size (1× → 100× unique reference images: Overall $4.66 → 6.10$ ), under the same training recipe [Table 4].
On the DreamBooth customization benchmark NP-Edit at 4-8 NFE (Overall $7.10$ – $7.33$ ) beats 8-NFE OminiControl ( $6.22$ ), DSD ( $6.29$ ), and is competitive with 8-NFE FLUX.1-Kontext-12B ( $7.39$ ) and Qwen-Image-Edit-20B ( $7.62$ ) — without using any synthetic paired customization data [Table 2].

Method

NP-Edit adapts a 2B internal DiT-based latent T2I model to image editing by concatenating the VAE encoding of a reference image to the noisy target along the token sequence dimension. The training set is 1M (local edits) + 100K (free-form / customization) reference images with edit instructions and target captions generated by Qwen2.5-32B — no paired before/after images.

The training objective combines two losses. VLM editing loss: the generator unrolls a two-step sampling trajectory starting from noise, producing a provisional clean image $\hat{x}_0$ , then re-noises with $\sigma_2$ and runs one more denoiser step to produce the final $\hat{x}_0'$ . LLaVA-OneVision-7B is fed (reference, $\hat{x}_0'$ , instruction) and asked two binary yes/no questions: an edit-verification question and an identity-preservation question. The loss is binary cross-entropy over the yes vs no token logits only (full-vocabulary normalization is worse), so a single VLM forward call yields the gradient — no autoregressive sampling, no reward-model surrogate. Distribution matching loss: DMD between the student and the frozen T2I teacher, with an auxiliary network $\mu_\phi$ that tracks the student’s current distribution and is updated 5× per generator step. A short initial pretraining phase has the generator simply reconstruct the concatenated reference, which stabilizes joint text+image conditioning before the main losses turn on.

Results

On GEdit-Benchmark English (VIEScore from GPT-4o), at 4 NFE the 2B NP-Edit reaches SC $6.16$ / PQ $7.69$ / Overall $6.10$ , beating all 4-NFE baselines and matching or exceeding several multi-step ones including BAGEL-7B at 50 steps (Overall $6.14$ ) and FLUX.1-Kontext-12B at 28 steps (Overall $5.65$ ); Qwen-Image-Edit-20B and Step1X-Edit-12B at multi-step settings remain ahead (Overall $7.36$ and $6.79$ ) [Table 1]. On DreamBooth customization, NP-Edit at 4 NFE Overall $7.10$ vs SynCD-12B at 30 steps Overall $7.54$ [Table 2]. Against Flow-GRPO with the same VLM reward, SFT alone scores Overall $3.64$ , SFT+RL $4.19$ , SFT+NP-Edit $6.06$ — and unlike RL, NP-Edit needs no SFT warm-start ( $6.10$ from scratch) [Table 5]. Scaling ablations show monotonic gains from larger VLM backbones (LLaVA-0.5B → 7B: $4.59 → 6.10$ ; InternVL-2B → 14B: $5.45 → 5.89$ ) and from 100× the dataset ( $4.66 → 6.10$ ) [Table 4].

Why it’s interesting

NP-Edit is the clearest filed instance of the “VLM as differentiable loss” position on the VLM-as-Evaluator axis — and it directly contradicts the “RL is the right wrapper” assumption underlying Unified Personalized Reward Model for Vision Generation (UnifiedReward-Flex) and TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward (TDM-R1). UnifiedReward-Flex trains an MLLM judge then samples from it via GRPO; TDM-R1 builds an unbiased per-step reward estimator on top of deterministic ODE trajectories. NP-Edit’s claim is structurally different: a frozen VLM’s yes/no-token logits are already differentiable end-to-end through the unrolled few-step sampling chain, so RL’s policy-gradient detour is unnecessary and empirically worse at matched reward model.

It also sharpens Diffusion Distillation by making the few-step student a prerequisite (rather than a downstream compression) for non-paired supervision: the VLM can only judge sharp enough intermediates, which is exactly what 4-step DMD students produce. This is the inverse of TDM-R1’s pipeline — TDM-R1 distills first then RLs; NP-Edit jointly distills via DMD and supervises via VLM gradients. The technique is also a natural pairing with EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing (EditReward as a human-aligned reward model for editing) — NP-Edit shows the simpler move of using a frozen general-purpose VLM works at competitive quality, which raises the question of whether a specialized EditReward backbone would lift NP-Edit further.