Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos

TL;DR

Magic Fixup is a 2024 Adobe/UMD paper (ECCV camera-ready style) that turns a coarsely user-edited photo — produced by cutting out segments and applying simple 2D transforms (translate / scale / rotate / mirror / duplicate / delete), what the authors call the Collage Transform — into a photorealistic image. The model is a Stable Diffusion v1.4 latent diffusion fine-tune with a dual-network architecture (a synthesizer + a frozen-init detail extractor that injects features by cross-attention at every diffusion step), trained on 5M paired video frames where the source frame is warped toward the target via either optical flow (RAFT-Large + softmax-splatting) or a per-segment piecewise-affine motion model. Inference takes <5 s and a 27-user 2AFC study prefers it over SDEdit on 89% of edits.

Key claims

Coarse 2D segment-level edits (“Collage Transform” — cut, transform, duplicate, delete with disocclusion-aware hole tracking) plus an automatic harmonization model is a more controllable image-editing UI than text-prompt re-synthesis or drag-handle dragging [§3.1, §1, Fig. 1].
A paired image-editing supervision dataset can be manufactured from raw video by extracting frames Δt apart and warping the source frame toward the target with two complementary motion models (dense flow warping, per-segment affine fit to flow), avoiding the need for human-labeled before/after pairs [§3.5, Fig. 5].
Training on a mix of flow-warped and piecewise-affine-warped pairs is necessary: flow-only training collapses the model to a refiner that barely deviates from the input, piecewise-affine-only training fails to preserve identity, and the mix lands in the sweet spot [§4.2, Fig. 9, Table].
A two-diffusion-model architecture (synthesizer conditioned on coarse edit + mask + cross-attention to a detail extractor that processes the noisy reference) preserves fine object identity that single-model + CLIP-image-embedding conditioning loses; ablating cross-reference-attention in favor of CLIP causes “a severe drop in performance” [§3.2-3.4, §4.2, Fig. 10].
Initializing inference from a noised version of the coarse edit (SDEdit-style, x_T = √α_T x_edit + √(1-α_T) ε) rather than pure Gaussian noise is required to match arbitrary user-input color distributions and recover deep dynamic range [§3.4, Eq. 4, Fig. 4].
In a 30-edit / 27-rater 2AFC user study, Magic Fixup is preferred over SDEdit by ≥75% of users on 80% of edits, with the remaining edits also preferring Magic Fixup except one out-of-domain artistic-painting case where preference splits 52/48 for SDEdit [§4.1].
Filtering training pairs to bounded motion magnitude (reject pairs where ≥10% of pixels have flow magnitude ≥350 px) is needed to keep the warp-based supervision well-posed; the final dataset is 2.5M pairs per motion model, 5M total [§3.5].
Training takes 48 h on 8× A100 jointly fine-tuning both networks (batch 32, Adam, lr 5e-5, 120k steps, DDIM-50 inference) — the paper argues stronger conditioning lets the model converge faster than AnyDoor’s 7-day / 64× V100 run [§3.6].

Method

The pipeline (Fig. 3) is two Stable Diffusion v1.4 latent diffusion models trained jointly. The detail extractor F takes the clean reference image, a binary mask of regions to inpaint, and a noisy reference at the current timestep; its CLIP cross-attention is stripped, and the first conv is zero-padded to accept the extra mask channel. F runs once per denoising step and exposes the feature tensors immediately before each of its self-attention blocks. The synthesizer G is conditioned on the coarsely edited image (concatenated with the mask) and queries F’s features by cross-attention layers inserted after each of its own self-attention blocks — at each layer ℓ and timestep t, q_t,ℓ from G, k_t,ℓ and v_t,ℓ from F. G is initialized from noise that is the coarsely edited latent diffused all the way to t=T (Eq. 4), not pure Gaussian, which the paper attributes to closing a train/test domain gap in noisy color statistics [§3.4, Fig. 4].

The training data construction is the paper’s other half. A 12M-clip stock-video corpus is keyword-filtered down to photorealistic, dynamic-but-not-extreme content (2.5M clips after filtering). For each clip a pair of frames (I, I*) is sampled with a Δt drawn uniformly from [0.5 s, 2 s]; the pair is rejected if optical flow between them is too extreme. Two warps are computed: (1) flow-based — RAFT-Large flow composed across intermediate frames, then softmax-splatting forward-warp I → Î, with disocclusion holes recorded in mask M; (2) piecewise-affine — MiDaS depth + Segment Anything segments, each segment given the affine transform that best matches its local flow, composited back-to-front by mean segment depth, with unsegmented regions falling back to flow-warping. Both warps produce a (coarse edit, mask, reference, target) tuple. The full 5M-pair dataset trains both motion models simultaneously.

Results

Headline user-study win: 89% preference over SDEdit aggregated across the 30-edit / 27-rater study; ≥75% on 80% of edits individually [§1, §4.1].
Compute economy: 48 h on 8× A100, vs the cited compositing baseline (AnyDoor [12]) at 7 days × 64× V100 [§3.6].
Inference latency: <5 s per edit using DDIM-50, vs MotionGuidance [20] at ~70 min/sample with flow-based diffusion guidance [§2].
Qualitative wins documented in Figs. 6-8: re-focuses moved objects when they cross focal-plane boundaries; synthesizes shadows and reflections after object moves; deletes a dog (and its shadow) and re-inserts it elsewhere coherently where AnyDoor leaves a visible inpainting halo and changes the dog’s identity; re-poses a lion’s head without DragDiffusion’s full-body drift.
Ablation table (held-out validation, LPIPS): training on flow-only data wins on flow test, training on affine-only data wins on affine test, training on both with cross-reference attention wins on both — and dropping cross-reference attention in favor of CLIP causes a “severe” LPIPS regression on both test splits [§4.2].
Failure mode acknowledged: on a stylized non-photorealistic painting edit, user preference between Magic Fixup and SDEdit is statistically tied (52/48 SDEdit) — the training corpus is photorealistic stock video, and the model does not generalize to out-of-distribution art styles [§4.1].

Why it’s interesting

This is the original (and now ~2 years old) paper for what has become the dominant “Collage Transform → harmonize” pipeline now showing up productized — Adobe Photoshop beta’s Rotate Object + Harmonize buttons (Photoshop beta: Rotate Object + Harmonize for 2D images (Adobe product announcement)) implement essentially this two-stage UX (coarse 2D segment manipulation, then a learned harmonization pass), and the paper’s first author is at Adobe. The video-as-paired-supervision recipe — sample two frames Δt apart, warp the source toward the target, train the model to invert the warp — has since been picked up at scale by AnyDoor and the recent layered-decomposition line (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) uses a similar simulated-triplet pipeline, VOID: Video Object and Interaction Deletion generalizes it to physics-aware counterfactual removal). The dual-network detail-extractor + cross-reference-attention architecture is also recognizably the design that AnimateAnyone and the broader reference-image-conditioning lineage in video-character work (InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing, Wan-S2V: Audio-Driven Cinematic Video Generation) inherits. Worth filing as the historical anchor for the segment-edit-then-harmonize axis of Layered Image/Video Decomposition and as an early instance of the “video frames as paired editing data” pattern in Synthetic Training Data.