Generative Image Layer Decomposition with Visual Effects (LayerDecomp)

TL;DR

LayerDecomp is an Adobe Research + UC Santa Cruz framework that decomposes a single input image into a photorealistic clean background and a transparent RGBA foreground layer that retains natural visual effects (shadows, reflections) associated with the foreground object. It is fine-tuned from a 5B-parameter pre-trained text-to-image DiT (text encoder dropped) with an RGBA-VAE adapted from the base VAE following LayerDiffusion. The trick that gets it to work on real images without ground-truth triplets is a consistency loss that enforces alpha_blend(predicted_BG, predicted_FG) ≈ input in pixel space — supplying supervision for the transparent foreground when no annotated triplet exists. Training uses a hybrid of large-scale simulated triplets (random BG + RGBA foreground assets with synthesized shadows) plus a small corpus of camera-captured pairs (with/without the object). Outperforms ControlNet-Inpainting, SD-XL-Inpainting, PowerPaint, ObjectDrop, Emu-Edit, MGIE, and OmniGen on object-removal and spatial-editing benchmarks (RORD, MULAN, DESOBAv2) plus user studies.

Key claims

Existing layer methods (LayerDiffusion, MULAN) cannot decompose an input image into RGBA layers that retain object-coupled visual effects like shadows and reflections in the foreground layer [§1, §2.2].
A two-layer DiT decomposer can be supervised on real photographs without ground-truth foreground triplets by training with a consistency loss: re-composite the predicted BG and predicted RGBA FG via alpha blending and L1/L2 match against the original composite [§3.2, Eq. 3].
Visual effects can be packed into the foreground’s alpha channel by re-purposing the RGBA-VAE from LayerDiffusion — minimal disturbance to the base latent space, so the pre-trained DiT prior transfers [§3.1].
A scalable simulated-triplet pipeline (entity-segmented object assets + depth-based occlusion filtering + shadow synthesis on a white background, blended onto random backgrounds) supplies the large-scale supervised half of training [§3.3, Simulated Data].
A small set of camera-captured before/after pairs (counterfactual photography à la ObjectDrop) generalizes the model to natural shadows and reflections that the simulator cannot match [§3.3, Camera-Captured Data].
On RORD, MULAN, and DESOBAv2 the model is robust to mask tightness — performance does not collapse on tight masks the way prior mask-based inpainters do — and beats ControlNet-Inpainting, SD-XL-Inpainting, PowerPaint, and ObjectDrop on PSNR/LPIPS/FID/CLIP-FID for object removal [Tab. 2, §4.2].
On instruction-driven removal (Emu-Edit Remove Set, user study), LayerDecomp is preferred over Emu-Edit, MGIE, and OmniGen in ≥50% of cases on a tri-axis evaluation (removal effectiveness, plausibility, background integrity) [Tab. 3, §4.2].
On spatial editing (move/resize), shadows and reflections move with the foreground because they live in the alpha channel — preferred over DiffEditor, DragAnything, DesignEdit, and DiffusionHandles by ≥15-percentage-point margins on plausibility in a user study [Tab. 4, §4.3].
The pipeline can be applied sequentially with different instance masks to peel a multi-object scene into multiple BG/FG layers, enabling per-layer spatial edits, recoloring, and filtering followed by re-composition [§4.4, Fig. 7].

Method

The decomposer is a DiT denoiser whose token sequence concatenates four streams: the noisy target latents, the original composite image latent (RGB), and a binary decomposition mask, each tagged with a learnable image-type embedding so the model knows which stream is being conditioned on and which is being denoised. Output is two streams: clean RGB background and transparent RGBA foreground. The foreground latent passes through a RGBA-VAE — the original DiT VAE fine-tuned with L1 + GAN + perceptual losses to encode an extra alpha channel into the same latent shape — chosen to minimize disturbance to the pre-trained DiT prior.

The crucial idea is the consistency loss. For real-world images, ground-truth foreground triplets do not exist, so the standard denoising loss can only supervise the background. To supervise the foreground, the paper reparameterizes the model’s prediction back to an estimate of the clean latent $\hat{z}_0$ , decodes both layers, and forces their alpha-blended re-composite to match the original input image in pixel space. This gives the foreground branch a usable gradient even with no FG annotation, and biases the model toward putting shadows/reflections in the FG alpha channel (because they appear in the input and would otherwise have to be invented in the BG).

Training combines two data sources. Simulated triplets: collect a corpus of RGBA object assets by entity-segmenting natural photos, filter with depth-based occlusion checks, synthesize per-object shadow intensity maps with an off-the-shelf shadow synthesizer, write the shadow into the alpha channel; at training time, paste each asset onto a random background to form the triplet. Camera-captured pairs: a small set of before/after photos (à la ObjectDrop) supply real shadows and reflections that the simulator can’t easily produce. Trained with Adam at lr 1e-4, batch size 64, on 32 A100s for 200K iterations; sampling uses DDIM with 50 steps.

Results

Background reconstruction (Tab. 1 ablation): Adding an RGBA-FG output head to a naive RGB-only DiT improves background PSNR (28.21 → higher; exact numbers redacted in fetch), indicating that having to also predict the foreground regularizes BG inference. Adding visual effects to the FG and adding the consistency loss each give further lifts on PSNR / LPIPS / FID / CLIP-FID for both BG and composite [§4.1, Tab. 1].
Object removal benchmarks [Tab. 2, §4.2]: beats ControlNet-Inpainting, SD-XL-Inpainting, PowerPaint, and ObjectDrop on RORD, MULAN, and DESOBAv2 by large margins. Robust to mask tightness — competitors’ FID collapses on tight masks because they leave shadow residue; LayerDecomp does not because it doesn’t need shadows in the mask.
Shadow removal on DESOBAv2: outperforms competitors even without using the dataset’s shadow-region annotation in the input mask [§4.2].
Instruction-driven removal user study [Tab. 3]: ≥50% preference over Emu-Edit, MGIE, and OmniGen across removal effectiveness, plausibility, background integrity, and overall quality, on a 300-image Emu-Edit-Remove Set subset.
Spatial editing user study [Tab. 4]: ≥15-percentage-point preference over DiffEditor, DragAnything, DesignEdit, and DiffusionHandles on result plausibility for object move/resize; shadows and reflections move naturally because they ride in the FG alpha channel.
Multi-layer extension [§4.4]: sequential application yields per-instance BG/FG layers for multi-object scenes; demonstrated for spatial edit + recolor + filter + recomposition.

Why it’s interesting

This is the decomposition pipeline that Controllable Layered Image Generation for Real-World Editing (LASAGNA) (LASAGNA) builds its training data from — LASAGNA-48K is real photos decomposed by LayerDecomp and then VLM-curated. Filing LayerDecomp closes the loop: the wiki now holds both the decomposer and the generator trained on its outputs, by the same lead author (Jinrui Yang). The consistency-loss recipe — re-composite predicted layers in pixel space and force-match the input — is a reusable supervision trick for any inverse-decomposition problem without ground-truth (worth considering for video matting / relighting setups). It also pairs cleanly with OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models on the video side: OmnimatteZero does the same BG/FG/effects decomposition for video, but training-free via latent arithmetic on a pre-trained DiT, where LayerDecomp pays training cost up-front to get a model that handles shadows and reflections specifically. Yunong’s Slack note flags the right caveat — the camera-captured supplement is small, so generalization to in-the-wild photographs with complex lighting (the failure mode also flagged by Qwen-Image-Layered-Control: text-guided layer extraction) is the open question for any layered-editing stack that wants to use this on natural imagery.