PICABench: How Far Are We from Physically Realistic Image Editing?

TL;DR

PICABench is a 900-sample diagnostic benchmark for physical realism in instruction-based image editing, structured along three dimensions — Optics (light propagation, reflection, refraction, light-source effects), Mechanics (deformation, causality), and State Transition (global/local) — with eight sub-dimensions and three instruction-specificity levels (superficial / intermediate / explicit). It introduces PICAEval, a region-grounded VLM-as-judge protocol that decomposes each test case into 4–5 binary QA pairs anchored to human-annotated ROIs (reflection surfaces, contact interfaces, etc.) to suppress hallucination. The companion PICA-100K is a 105k synthetic instruction-edit dataset built by chaining FLUX.1-Krea-dev (T2I scene rendering) → Wan2.2-14B-I2V (state-transition simulation) → first/last-frame extraction → GPT-5 instruction recalibration; fine-tuning FLUX.1-Kontext-dev on it with LoRA-256 gains +1.71% on the benchmark and improves non-edit PSNR from 24.57 to 25.23 dB. Across 11 open- and closed-source editors, all open-source models score below 60 and only GPT-Image-1 and Seedream 4.0 narrowly exceed it — physical realism is the residual gap that semantic-fidelity benchmarks miss.

Key claims

Existing editing benchmarks emphasize semantic and perceptual fidelity but systematically overlook physical effects (shadows that should move with their object, reflections, refractions, deformation, causality, state transitions), so the question of how far current models sit from physically realistic image editing has been empirically unanswerable until now [§1].
The benchmark decomposes physical realism into three dimensions × eight sub-dimensions with concrete checkable criteria — Optics (light propagation, reflection, refraction, light-source effects), Mechanics (deformation, causality), State Transition (global, local) — covering common editing operations (add, remove, attribute change) rather than scientific-domain edge cases [§3.1].
Each of the 900 benchmark images is paired with three nested instruction levels — superficial (plain command), intermediate (with brief physical rationale), explicit (with explicit description of expected result) — to test whether models can internalize physics or only follow surface commands [§3.2, Tab. 2].
PICAEval is a region-grounded QA-based metric: per benchmark entry, GPT-5 generates 4–5 binary yes/no questions tied to human-annotated ROIs (reflective surfaces, deformation zones, cast shadows), then a VLM (GPT-5 or Qwen2.5-VL-72B) answers each question constrained to the visible content within the ROI; the score is the exact-match proportion [§3.3].
PICAEval beats unstructured VLM-as-judge prompting on agreement with human raters in an Elo-style human study, attributable to spatial grounding reducing hallucination, decomposition increasing interpretability, and the QA format mirroring human plausibility checks [§3.3, §4.3, Fig. 6].
PICA-100K is built by a fully synthetic two-stage pipeline: (1) Subject × Scene prompt dictionaries → GPT-5 template expansion → FLUX.1-Krea-dev T2I scene rendering; (2) motion-instruction templates (“tilt the vase until it tips over”, “swing the lantern gently in the wind”) → GPT-5 expansion → Wan2.2-14B-I2V video synthesis; first and last frames of each video form the (source, edited) pair with GPT-5 auto-annotating the preferred output [§3.4].
The final dataset has 105,085 synthetic instruction-edit samples across eight physics categories — no human labeling required at any stage [§3.4].
Fine-tuning FLUX.1-Kontext-dev (12B flow-based DiT) with LoRA rank 256, batch 64, 10K steps on 16 A100s yields +1.71% overall accuracy on PICABench and improves non-edit PSNR from 24.57 dB to 25.23 dB over the base model [§3.4, Tab. 3].
A real-video baseline (MIRA400K, 400K edited images via the UniReal pipeline over Miradata) trained under matched settings performs worse on overall PICABench accuracy than the 105K synthetic PICA-100K — direct evidence that the synthetic T2I → I2V pipeline beats real-video pseudo-labeling on this physics-aware target [§4.2, Tab. 3].
Among 11 evaluated models, all open-source editors (FLUX.1 Kontext, Step1X-Edit, Bagel, Bagel-Think, HiDream-E1.1, UniWorld-V1, OmniGen2, Qwen-Image-Edit, DiMOO) score below 60; only closed-source GPT-Image-1 and Seedream 4.0 slightly exceed 60 — physical realism is a persistent open gap [§4.2, Tab. 1].
More detailed prompts help, but unevenly: gains from intermediate prompts are smaller than gains from explicit prompts, attributed to models lacking internalized physics principles that would let them leverage the intermediate rationale. Bagel’s long-text comprehension lets it overtake FLUX.1-Kontext only under explicit prompts [§4.2, Tab. 2].
Unified MLLMs (presumed advantage: stronger world understanding) consistently underperform dedicated edit models on PICABench among open-source baselines — understanding alone is insufficient; coupling understanding with generation remains an open challenge [§4.2].
A local state-transition slight drop is observed after fine-tuning, attributed to using only first and last frames to represent state changes (intermediate frames discarded) — a known limitation flagged for future work [§4.2, §5].

Method

PICABench operationalizes “physical realism in image editing” along the Optics / Mechanics / State Transition taxonomy. Data curation is two-stage: a GPT-5-expanded keyword vocabulary mapped to the 8 sub-dimensions drives image retrieval from licensed/public sources, prioritizing visually diverse scenes with salient physical cues (directional lighting, transparent/reflective media, deformable objects, phase-changeable substances); human annotators filter duplicates/artifacts and tag sub-dimensions. Each retained image is paired with a human-written instruction designed to induce a physics-relevant edit, then GPT-5 expands the instruction into the superficial / intermediate / explicit triple, followed by manual review.

PICAEval at evaluation time: (1) annotators mark physics-critical regions in the input; (2) GPT-5 generates 4–5 binary QA pairs per entry from the edit instruction + region, manually reviewed for clarity and coverage; (3) at test time, a VLM is prompted with the edited image + instruction + region + question and emits a constrained yes/no answer over visible content within the region. The aggregate score is the proportion of questions whose VLM answer exactly matches the reference label.

PICA-100K training pipeline: a Subject Dictionary and Scene Dictionary are combined via handcrafted T2I templates, refined by GPT-5, and rendered by FLUX.1-Krea-dev into static source images. Separately, motion-oriented I2V instruction templates (rotations, movements, tilts) are GPT-5-expanded and applied to the source images via Wan2.2-14B-I2V to produce short clips. First and last frames are extracted as (source, edited) pairs; GPT-5 auto-labels the final frame as the preferred output. Training is LoRA-256 on FLUX.1-Kontext-dev, batch 64, AdamW, 10K steps on 16 A100s.

Results

11-model evaluation under superficial prompts: every open-source model scores below 60 overall; only GPT-Image-1 and Seedream 4.0 (closed) slightly exceed 60 [Tab. 1].
Prompt-specificity ablation: model performance improves monotonically from superficial → intermediate → explicit, with the explicit lift dominating the intermediate lift; Bagel overtakes FLUX.1-Kontext under explicit prompts thanks to long-text comprehension [Tab. 2].
PICA-100K fine-tuning of FLUX.1-Kontext-dev: +1.71% overall PICABench accuracy, non-edit PSNR 24.57 → 25.23 dB, with consistent gains across most sub-dimensions; slight drop on global state-transition accuracy and causality consistency (attributed to using only the first/last frame as supervision) [Tab. 3, Appendix A.4].
Direct head-to-head against a real-video editing dataset: training on MIRA400K (400K samples derived from Miradata using the UniReal pipeline) yields worse overall accuracy than PICA-100K (105K samples) at matched settings — the synthetic T2I+I2V pipeline beats real-video pseudo-labeling on the physics-aware target [Tab. 3].
Human-validation Elo study: PICAEval scores correlate more strongly with human pairwise rankings than baseline unstructured VLM-as-judge prompting [Fig. 6].

Why it’s interesting

PICA-100K is structurally identical to a growing family of physics/edit datasets the wiki already tracks — and it’s the cleanest filed example so far of a physics-targeted synthetic editing corpus. Three concrete connections:

It sits exactly on the Synthetic Training Data thesis that a verification gate + generator stack can substitute for real labels, but with a physics-aware twist: the I2V model (Wan2.2-14B) is itself the physics simulator — the world-model assumption is that the I2V model’s learned dynamics are physically plausible enough that first/last-frame pairs constitute valid (source, edited) supervision. Contrast with Action100M: A Large-scale Video Action Dataset (real videos + automated labeling via VLM stack) and UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (10M general edit corpus, recaptioning gate via Qwen-Verify) — PICA-100K is smaller (105K) but uniquely uses the I2V model as the data engine. The MIRA400K-vs-PICA-100K head-to-head is the first filed datapoint that more real-video-derived data can underperform less synthetic-video-derived data when the target axis is physics — relevant to the team’s interest in Ray-curated extensions.
It joins UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits and UniREditBench: A Unified Reasoning-based Image Editing Benchmark as the third filed reasoning/physics-aware editing benchmark in a month, all using VLM-as-judge with region- or rubric-grounding to suppress hallucination (VLM-as-Evaluator). PICAEval’s specific contribution to that lineage is the human-annotated ROI as a spatial constraint on the VLM’s answer space — distinct from UnicBench’s 4-metric decomposition (IF/NC/VQ/RA) and UniREditBench’s GT-image second reference. The three benchmarks bound the design space for VLM-judged editing evaluation: rubric decomposition, GT-image reference, and ROI constraint.
The “fine-tune FLUX.1-Kontext on a targeted synthetic dataset for a specific gap” recipe directly parallels what FireRed-Image-Edit: A General-Purpose Image Editing Model does for general editing and what UniREdit-Bagel does for reasoning-driven editing — the open editing stack is now consistently competitive with closed frontier models on the specific axis the targeted corpus covers, while general-purpose closed editors retain their lead overall.

Wendy’s note flags this for Ray-model curation: the obvious next experiment is to substitute Wan2.2-14B-I2V with Luma’s video model in the PICA-100K pipeline and see whether higher-quality I2V dynamics (especially for weather and state-transition categories that dominate the public dataset preview) yield more usable (source, edited) pairs.