RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details

TL;DR

RefineAnything formalizes region-specific image refinement as a distinct task from instruction editing: given an input image and a scribble/box, restore fine-grained local details (text, logos, thin structures) while keeping every non-edited pixel strictly unchanged. The method is a LoRA fine-tune of Qwen-Image-Edit (2509) augmented with two ideas — Focus-and-Refine, a crop-resize-refine-paste-back pipeline motivated by the counter-intuitive observation that simply cropping a small target region under a fixed VAE input resolution sharply improves VAE reconstruction within that region; and a Boundary Consistency Loss that upweights training supervision in a band along the edit boundary to suppress paste-back seams. Trained on a 30K synthetic dataset built by Gemini3 + SAM3 + scribble-inpainting degradation, and evaluated on a new 402-input RefineEval benchmark, it cuts edited-region MSE in half versus FLUX Kontext (0.020 vs 0.040) while driving background MSE and LPIPS to ~0 and SSIM to 0.9997.

Key claims

Region-specific refinement is a new problem setting distinct from general instruction editing, defined by three simultaneous requirements: region-accurate, detail-effective, background-preserving — none of which are reliably met by Gemini 2.5/3, GPT-4o, OmniGen2, BAGEL, FLUX Kontext, or Qwen-Image-Edit on the authors’ RefineEval [§1, Fig. 1, Tab. 1].
Counter-intuitive observation: under a fixed VAE input resolution, cropping a small target region and resizing it back to the full resolution substantially improves VAE reconstruction within the region, even though no new effective pixel information is added [§3.2, Fig. 3].
Focus-and-Refine reallocates the fixed-resolution budget to the target by (i) computing a tight bounding box around the user mask, (ii) expanding by margin τ=0.1, (iii) cropping and resizing the focus view, (iv) running RefineAnything on the focus, (v) softening the cropped mask via morphological dilation (kernel 16) + Gaussian blur (kernel 21) for a blended paste-back [§3.2, Eqs. 3–9].
The Boundary Consistency Loss defines a thin boundary band around the user mask via dilation/erosion (kernels 11), then multiplies the per-location flow-matching loss in the band by λ_b = 3 to upweight seam supervision during LoRA fine-tuning [§3.3, Eqs. 10–11].
Fine-tuning is LoRA-only on attention projections (to_q, to_k, to_v, to_out.0) of Qwen-Image-Edit 2509: rank 64, lora_alpha 64, AdamW lr 1e-4, batch 16, 50K steps, BF16 [§3.4].
Refine-30K = 20K reference-based + 10K reference-free samples; the reference-based pipeline cross-grounds a subject between two images with Gemini3, segments the subject in the target with SAM3, samples random scribbles inside the dilated mask, and inpaints to synthesize a degraded input — the original target serves as ground truth [§4.1, Eq. 12].
Reference-free data uses single images: Gemini3 picks a salient object, SAM3 segments it, scribble-inpainting degrades it, then Gemini3 filters out samples where the degradation is not visually noticeable or is semantically implausible — keeping only well-defined refinement tasks [§4.2].
RefineEval is 402 inputs from 67 manually annotated cases (31 reference-based + 36 reference-free) × 3 inpainting backbones (Flux-fill, SDXL, Qwen-Edit) × 2 selected scribble seeds per cell [§5.1].
On reference-based RefineEval, the method posts edited-region MSE 0.020 / LPIPS 0.155 vs. best baseline Kontext’s 0.040 / 0.264, with DINO/CLIP/SSIM 0.793 / 0.885 / 0.591 vs Kontext’s 0.685 / 0.785 / 0.538 [Tab. 1].
Background preservation reaches L1/L2 = 0.000 / 0.000 and SSIM 0.9997, against Kontext’s 0.011 / 0.019 / 0.9660 — eliminating the background drift seen in every other baseline [Tab. 1].
On reference-free RefineEval, scored 1–5 by Gemini2.5-Pro across VQ / Naturalness / Aesthetics / Detail / Faithfulness, the method beats the strongest open baseline Qwen-Edit by +0.725 / +0.758 / +0.771 / +0.745 / +0.430 [Tab. 2].
Ablations: removing Focus-and-Refine raises edited-region MSE 0.020 → 0.021 and LPIPS 0.155 → 0.177 (with introduced artifacts visible qualitatively); removing the Boundary Consistency Loss raises LPIPS to 0.191 and DINO drops 0.793 → 0.736, producing visible seams [Tab. 3, Fig. 9].

Method

RefineAnything is a fine-tuned Qwen-Image-Edit (2509) MMDiT denoiser whose conditioning is three branches: (i) Qwen2.5-VL frozen encoder ingests the input image (plus optional reference), the scribble mask, and a refinement instruction, emitting multimodal tokens that joint-attend with the latent stream; (ii) the Qwen-Image VAE encodes the input (and optional reference) into latents that are packed with the noisy target latent into the patch sequence as low-level visual conditioning; (iii) the noisy target latent itself. Only LoRA parameters on attention projections are trained — the VAE and Qwen2.5-VL stay frozen.

Inference runs in three steps. (1) Region localization: compute the tight bounding box of the user scribble (or use a user-provided box), expand by margin τ=0.1 of image size, and clip to bounds; crop and resize the input image and mask into the focus view. (2) Focused generation: feed the cropped image + mask + (optional reference) + instruction into the RefineAnything denoiser, sampling a refined crop at full input resolution. (3) Seamless paste-back: dilate the cropped mask (kernel 16), Gaussian-blur (kernel 21), composite refined crop with original crop using the blended mask, resize back to the original bounding-box location, and paste onto the unchanged background. The blended mask enforces strict background preservation by construction; the Boundary Consistency Loss (applied during training only) ensures the refined content blends naturally across the band where the soft mask transitions from 1 to 0.

Refine-30K is constructed via a Gemini3 + SAM3 + scribble-inpainting pipeline. For reference-based samples, Gemini3 grounds a single salient subject across an (input, target) pair, SAM3 segments it in the target, random scribbles sampled inside the dilated mask drive an inpainting model to synthesize a degraded version of the target — this degraded image is the model’s input, the original target is the ground truth. For reference-free samples, the same pipeline runs on single images, with an extra Gemini3 filter to discard samples where the degradation isn’t visually noticeable or where the implied refinement is semantically implausible.

Results

Reference-based RefineEval (Tab. 1) edited-region: MSE 0.020 / LPIPS 0.155 / VGG 0.401 / DINO 0.793 / CLIP 0.885 / SSIM 0.591 vs the strongest open baseline Kontext (0.040 / 0.264 / 0.540 / 0.685 / 0.785 / 0.538) — roughly a 50% reduction in MSE and 41% in LPIPS [Tab. 1].
Reference-based RefineEval background: L1 / L2 / SSIM = 0.000 / 0.000 / 0.9997 vs Kontext (0.011 / 0.019 / 0.9660); Qwen-Edit and OmniGen2 are an order of magnitude worse on background drift [Tab. 1].
Reference-free RefineEval (Tab. 2, Gemini2.5-Pro 1–5 scoring): VQ 3.806, Naturalness 3.868, Aesthetics 3.876, Detail 3.720, Faithfulness 3.644 — all dimensions first place, ahead of Qwen-Edit (3.081 / 3.110 / 3.105 / 2.975 / 3.214) [Tab. 2].
Ablation (Tab. 3): w/o Focus-and-Refine raises edited-region LPIPS to 0.177, drops DINO 0.793 → 0.779; w/o Boundary Consistency Loss raises LPIPS to 0.191 and drops DINO to 0.736 while keeping background L1/L2/SSIM perfect (the blended-mask paste-back guarantees background preservation independent of the loss) [Tab. 3, §5.6].
Qualitative (Figs. 6–9): the strongest baselines (Gemini 3, BAGEL, Kontext) either fail to recover sub-pixel text strokes or drift the background; ablating Focus-and-Refine reproduces the baselines’ detail collapse; ablating Boundary Consistency Loss produces visible seams at the crop boundary even with the same denoiser [§5.5, §5.6].

Why it’s interesting

Two pieces are independently useful and one is genuinely surprising. The surprising piece is the crop-and-resize beats whole-image at fixed VAE resolution observation in §3.2: it isolates the VAE’s input resolution — not the diffusion model’s capacity — as the binding constraint on local detail fidelity, and gives a near-free workaround (resize the region into the VAE budget instead of asking the VAE to encode the whole image at that budget). This is mechanistically adjacent to TC-AE: Unlocking Token Capacity for Deep Compression Autoencoders‘s observation that deep-compression autoencoders’ token capacity is the limiting factor for what fine detail can be reconstructed, and to Making Reconstruction FID Predictive of Diffusion Generation FID‘s argument that reconstruction FID predicts generation FID — RefineAnything’s fix is a runtime-only workaround for the same underlying VAE bottleneck rather than a new autoencoder.

The benchmark contribution sits cleanly alongside the three other 2025–2026 editing benchmarks the wiki tracks. UniREditBench: A Unified Reasoning-based Image Editing Benchmark contributes the GT-image second reference, PICABench: How Far Are We from Physically Realistic Image Editing? contributes a region-grounded VLM-as-judge constrained to human-annotated ROIs, UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits contributes a 4-metric rubric with Qwen-Verify; RefineAnything’s RefineEval contributes a pixel-precise dual-region metric (edited-region MSE/LPIPS/SSIM against GT and background L1/L2/SSIM against input) that directly captures background-drift in numbers, a failure mode the rubric-based judges can miss. Together these four benchmarks bound the practical design space for VLM-as-Evaluator applied to image editing — RefineAnything is uniquely the one where the task (pixel-strict background preservation) admits direct pixel metrics rather than requiring a judge.

The data-construction recipe is a textbook Synthetic Training Data instance: Gemini3-grounded subjects + SAM3-segmented masks + scribble-inpainting degradation + VLM-based plausibility filter. The synthesis works because the inverse of the noise process is the task: you generate plausibly-degraded versions of clean images and train a model to invert. Compared to PICABench: How Far Are We from Physically Realistic Image Editing?‘s I2V-as-physics-simulator pipeline and UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits‘s general-purpose corpus, Refine-30K is the smallest (30K) but most narrowly targeted — and lands meaningful gains via a Parameter-Efficient Finetuning LoRA on a frozen 2509 Qwen-Image-Edit backbone.