Skip to content
Bud's Wiki

Layered Image/Video Decomposition

TL;DR

Section titled “TL;DR”

Methods that decompose a natural image or video into a clean background plus one (or more) transparent RGBA foreground layers that carry the object’s visual effects — shadows, reflections, smoke — and re-compose via alpha blending. The shared problem is that most prior decomposers ignore visual effects (LayerDiffusion, MULAN), so editing the layers produces obvious compositing artifacts. Current approaches split along three axes: (a) trained DiT decomposer with a consistency loss (Generative Image Layer Decomposition with Visual Effects (LayerDecomp)), (b) training-free decomposition via attention-readout + latent arithmetic on a frozen video DiT (OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models), and (c) text-conditioned layer extractors retrofitted onto base image-editing models (Qwen-Image-Layered-Control: text-guided layer extraction). Joint generation of layered output (rather than decomposition of a given image) is the natural follow-up, now covered by two distinct backbones: LASAGNA’s single-pass joint generator (Controllable Layered Image Generation for Real-World Editing (LASAGNA)) and ART’s region-crop MMDiT that scales to ~50 layers (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation).

Key claims

Section titled “Key claims”
  • Putting visual effects (shadows, reflections) into the foreground layer’s alpha channel lets shadows/reflections move automatically with the foreground during spatial edits — a property that pixel-space alpha-compositing methods cannot get for free (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) §4.3, Tab. 4).
  • A pixel-space consistency loss (alpha-blend predicted BG and predicted RGBA-FG, match against input) supplies usable gradient for the transparent foreground branch even when ground-truth foreground triplets are unavailable for real photographs (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) §3.2, Eq. 3).
  • Pre-trained video diffusion models internalize object-effect coupling: self-attention maps from a single forward pass at flow-matching t≈0.5 already identify which background pixels co-vary with a foreground object, removing the need for a separately trained effects detector (OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models §3.2, Fig. 1).
  • Foreground layers can be transferred between scenes by simple latent arithmetic on a shared VAE latent space: z_fg = z_orig − z_bg, then compose into a new scene via z_newbg + z_fg followed by a short low-strength refinement pass — no per-scene optimization (OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models §3.3-3.4).
  • An RGBA-VAE adapted from a pre-trained RGB VAE (LayerDiffusion recipe) can encode an extra alpha channel into the same latent shape with minimal disturbance to the base DiT prior, so existing pre-trained DiT backbones transfer to the layered task (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) §3.1). Refinement: ART instead bakes α into RGB as ÎRGB = (0.5·α + 0.5)·IRGB (gray-background encoding), reuses a frozen FLUX VAE encoder for all layers, and trains only a separate small ViT-Base decoder (86M params) to recover RGBA — sidestepping the channel-dimension change in the VAE altogether (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation §3.1).
  • Text-instruction-following layer extraction can be retrofitted onto a base layered-image model that was never trained for instructions, by swapping the multi-image output head for a single-image one and fine-tuning on a layered-graphics dataset (Qwen-Image-Layered-Control: text-guided layer extraction Usage Tips).
  • Decomposition outputs can seed layered generation training data: LASAGNA-48K is built by running LayerDecomp over COCO-style imagery and filtering with VLMs, then used to train a joint BG+FG generator (Controllable Layered Image Generation for Real-World Editing (LASAGNA) §Data; Generative Image Layer Decomposition with Visual Effects (LayerDecomp)). Refinement: template-native datasets are an alternative substrate — LICA ships 1.55M multi-layer compositions sourced from design templates (typed text/image/vector/group hierarchy, ~30× larger than LASAGNA-48K), sidestepping the decomposition step entirely (LICA: A Dataset of 1.5M Layered Graphic Design Compositions). Further refinement: for anime/illustration domains, See-through harvests ground truth from commercial Live2D model files (per-part textures, drawing order, hidden geometry pre-painted by the artist) — neither counterfactual photography nor template scraping, but artist-supplied multi-layer decompositions (See-through: Single-image Layer Decomposition for Anime Characters §Abstract).
  • Region-wise prompts are not required for layered generation: given a global prompt and bounding boxes alone, a fine-tuned MMDiT autonomously binds visual tokens in each anonymous region to the right entities via cross-attention; adding per-region captions can hurt inter-layer coherence (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation §1, §4.2, Fig. 2). This contradicts the conventional GLIGEN / InstanceDiffusion / MS-Diffusion design that injects region-wise text into the layout-conditioned diffusion process.
  • Region-crop attention (each foreground token attends only within its bounding box plus the global reference+background tokens) is what lets joint generation scale to 50+ layers — >12× faster than full attention on long sequences, with fewer layer conflicts (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation §1, §3.2).
  • Layout-conditional 3D RoPE that splits channels into (x, y, layer) thirds gives every foreground token an explicit (region position, layer index) identity inside one concatenated sequence — letting one MMDiT pass denoise reference + background + all foreground layers simultaneously (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation §3.2, Eq. 6).
  • The decompose-edit-recompose UX is being productized: Adobe Photoshop beta now ships a “Rotate Object” feature that rotates 2D image content as if it were 3D, paired with a “Harmonize” pass that re-renders shadows and lighting to blend the edit into the scene (Photoshop beta: Rotate Object + Harmonize for 2D images (Adobe product announcement)). No method details disclosed, but the two-step UI (rotate, then harmonize) mirrors the research-side split between object manipulation and effects re-rendering.
  • Animated layered design is emerging as a data modality: LICA includes 27,261 animated compositions with per-component keyframes and motion parameters (LICA: A Dataset of 1.5M Layered Graphic Design Compositions §A New Modality) — no prior layered-image dataset on the wiki ships this, and it is the substrate a video-domain analogue of ART would need.
  • Coupling a Body Part Consistency Module with pseudo-depth is sufficient to stratify fine interleaving structures (e.g. hair strands) into separate layers from a single image — important because most prior decomposers either treat layer count as fixed (LayerDecomp’s BG+FG split) or rely on bounding-box layouts (ART’s region-crop) rather than depth ordering (See-through: Single-image Layer Decomposition for Anime Characters §Abstract). This adds a third stratification primitive (pseudo-depth) to the existing two (alpha-channel-as-effects, region-crop layout).
  • Object removal can target physical dynamics, not just appearance. VOID extends the removal target from “object + its visual effects (shadows, reflections, wake)” to “object + its downstream physical consequences (falling held items, halted domino chains, uninterrupted spinning tops)” — supervised by paired counterfactual Kubric / HUMOTO simulations where the physics engine rolls forward differently with and without the target object (VOID: Video Object and Interaction Deletion §1, §3, Fig. 1). This adds a third removal-target axis: (1) appearance attribution (OmnimatteZero, EffectErase), (2) layout/depth stratification (See-through, ART), (3) counterfactual physics (VOID).

Open question — generalization to natural photographs: LayerDecomp supplements simulated triplets with a “small set” of camera-captured before/after pairs (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) §3.3); Qwen-Image-Layered-Control explicitly fails on “photographic images with complex lighting/shadows” while working on poster graphics (Qwen-Image-Layered-Control: text-guided layer extraction Usage Tips). ART claims strong photo-bench wins over LayerDiffuse on photorealistic prompts (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation Fig. 3) — suggesting the joint-generation route trained on a large MLTD dataset may dodge the photo-failure mode that retrofitted extractors hit. See-through (See-through: Single-image Layer Decomposition for Anime Characters) sidesteps the issue entirely by restricting to the anime/illustration domain where ground truth is artist-supplied — leaving the natural-photo question still open. It is not settled whether more diverse counterfactual photography, better simulator priors, or training-free attention readout (OmnimatteZero’s route) is the right answer for in-the-wild photos. The Slack discussion that prompted this concept page (Yunong, 2026-05-23) flagged exactly this — extending LayerDecomp to natural images, shadows and reflections “can indeed be a problem.”

Recent contributions

Section titled “Recent contributions”
  • [2026-07-08] Seedream 5.0 Pro — Multimodal Image Generation Model (ByteDance Seed): Seedream 5.0 Pro product page advertises “layer separation” as an explicit editing capability of a frontier closed image model — a small product-side signal that layered/editable output is moving from academic decomposition methods into commercial image-generation feature sets. Mechanism (RGBA head vs prompt-level) not disclosed.
  • [2026-06-05] Ideogram 4.0 Technical Details: Open model at the forefront of design: Ideogram 4.0 today ships layered output via a separate Background Remover post-processing step, but the press release announces a follow-up 4.0 release that returns alpha channels and editable text layers directly from inference — the model’s output as a stack of components rather than a flat frame. Roadmap datapoint rather than shipped result, but a strong native-layered-decoder claim from a production T2I lab.
  • [2026-06-04] The Next Frontier of Visual AI Is Code: a16z position essay frames layer-style decomposition (DOM nodes, Lottie layers, SVG paths, scene-graph parts) as the symbolic source-of-truth that makes any visual artifact iteratively editable — a generalized version of the layered-decomposition thesis extended to vector/code representations beyond raster RGBA layers
  • [2026-06-02] Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning: Stable-Layers (Stability AI) — first filed instance of RL post-training applied to the decomposition side: Flow-GRPO + LoRA on top of Qwen-Image-Layered, supervised entirely by a VLM judge with no paired ground-truth layered images. Reward design: per-sample CoT scoring on five edit-centric criteria, plus a grid-based side-by-side re-scoring pass to recover within-group variance that pointwise scoring collapses. Reports stronger layer separation, fewer blank/artifact layers, and lower per-layer reconstruction error vs the Qwen-Image-Layered base on Crello.
  • [2026-05-29] MRT: Masked Region Transformer for Layered Image Generation and Editing at Scale: MRT (20B masked region transformer for layered image generation, editing, and decomposition) unifies T2L/I2L/L2L via selective token masking on Qwen-Image, introduces an overflow-aware canvas layer for content extending beyond canvas boundaries, and claims user-study wins over the concurrent Qwen-Image-Layered (Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition) with 10-100× faster I2L inference and 50-90% lower activation memory; 8-step distillation for real-time. Weights not yet released.
  • [2026-05-23] VOID: Video Object and Interaction Deletion: VOID (Netflix / INSAIT) — physics-aware video object removal built on CogVideoX-Fun-V1.5-5B-InP. Two-pass fine-tune with optical-flow-warped latent init for temporal consistency; conditioned on a 4-value quadmask (background / primary / overlap / affected). Trained on paired counterfactuals from Kubric (rigid-body sim) and HUMOTO (re-rendered in Blender), where removing the object causes the physics engine to roll forward to a different outcome (objects fall, dominos halt, spinning tops keep spinning). At inference a VLM populates the “affected regions” channel of the quadmask. First wiki entry that targets dynamics counterfactuals rather than appearance attribution — directly extends the EffectErase/OmnimatteZero line into physics.
  • [2026-05-23] See-through: Single-image Layer Decomposition for Anime Characters: See-through (Saint Francis U / Spellbrush / Penn, SIGGRAPH 2026 conditionally accepted) — single anime illustration → up to 23 fully-inpainted semantic layers (PSD) with inferred drawing order. Couples a diffusion-based Body Part Consistency Module (LayerDiff 3D, SDXL-based) with an anime-fine-tuned Marigold pseudo-depth model. Trains on ground-truth harvested from commercial Live2D model files (per-part textures, drawing order, hidden geometry). Open-sourced code + 3 model checkpoints + HF Space demo; NF4-quantized inference at 1280 in ~8 GB VRAM. First illustration-domain entry in this cluster; sidesteps the photo-supervision problem by restricting to a domain with artist-supplied ground truth.
  • [2026-05-23] LICA: A Dataset of 1.5M Layered Graphic Design Compositions: Lica World (Anything.design) announces LICA — 1.55M layered graphic-design compositions with typed component hierarchy, per-element typography/geometry/opacity metadata, and a 27,261-composition animated subset. A public template-sourced alternative to ART’s private 1M MLTD dataset, and ~30× larger than the decomposer-curated LASAGNA-48K.
  • [2026-05-23] Photoshop beta: Rotate Object + Harmonize for 2D images (Adobe product announcement): Adobe Photoshop beta ships Rotate Object (3D-rotation of 2D image content) + Harmonize (re-light and re-shadow to blend). A product-side data point that the two-stage decompose/edit/re-composite-effects pipeline studied in this concept is now mainstream UX. No method details disclosed.
  • [2026-05-23] ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation: ART (MSRA / Tsinghua, CVPR 2025; rumored Canva production arch) — text-to-layers joint generation on FLUX MMDiT with anonymous (caption-less) region layout, layout-conditional 3D RoPE, region-crop attention, and a ViT alpha decoder. Scales to 50+ layers; beats LayerDiffuse on photorealistic and COLE on graphic-design user studies.
  • [2026-05-23] Generative Image Layer Decomposition with Visual Effects (LayerDecomp): introduces the BG + transparent RGBA-FG-with-visual-effects formulation; consistency loss as supervision substitute for missing real-image triplets; simulated-triplet pipeline + counterfactual photography hybrid training data. Pipeline whose outputs seed LASAGNA-48K.
  • [2026-05-23] Controllable Layered Image Generation for Real-World Editing (LASAGNA): LASAGNA — flips the problem from decomposition to joint generation, using LayerDecomp outputs (filtered with VLMs) as training data. Single attention backbone handles Text2All / FG_Gen / BG_Gen via learnable role embeddings.
  • [2026-05-23] Qwen-Image-Layered-Control: text-guided layer extraction: Qwen-Image-Layered-Control — retrofits text-conditioning onto a base layered-image model by output-head swap + fine-tune on PrismLayersPro. Demonstrates the failure mode (photographic complex lighting) that motivates LayerDecomp-style training data.
  • [2026-05-23] OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models: video-domain training-free decomposition via self-attention readout + latent arithmetic. Same BG/FG/effects target as LayerDecomp but for video, and without the training cost.

Open questions

Section titled “Open questions”
  • What is the right scaling axis for the camera-captured supplement to LayerDecomp’s simulated triplets? Order-of-magnitude more photos? Or a different acquisition recipe (multi-light captures, scene-graph control)?
  • For in-the-wild natural photographs with hair, fur, foliage, and complex multi-bounce lighting, does the trained-decomposer route (LayerDecomp) or the training-free attention-readout route (OmnimatteZero, applied per-frame to image diffusion) generalize better? ART’s photo-bench wins (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation Fig. 3) suggest a third option — train a joint generator on enough multi-layer data and skip decomposition entirely.
  • Can the consistency loss be extended to multi-layer decomposition (per-instance FG layers) in one forward pass, instead of LayerDecomp’s sequential per-mask application? ART shows this is achievable on the generation side with region-crop attention + 3D RoPE; the open question is whether the same trick ports back to decomposition. See-through (See-through: Single-image Layer Decomposition for Anime Characters) achieves 23-layer decomposition in one pipeline, but in two diffusion stages (LayerDiff 3D + Marigold) coupled by a body-part-tag interface — not a single forward pass.
  • Does the alpha-channel-as-effects-carrier convention compose with downstream relighting / shadow editing models, or does it force baked-in shadows that can’t be re-shaded?
  • How well does the LayerDecomp → LASAGNA-48K data-curation recipe generalize — i.e. can any reasonable decomposer + VLM-curator pair produce training data for a generator that beats specialized layered datasets like MULAN? ART’s private MLTD dataset (1M designs, K=2-50) is the current quality bar; LICA (LICA: A Dataset of 1.5M Layered Graphic Design Compositions) is a candidate public substitute at comparable scale — but it’s sourced from design templates, not decomposed from photos, so it tests a different recipe (template-native vs decomposer-curated). See-through’s Live2D-sourced ground truth (See-through: Single-image Layer Decomposition for Anime Characters) is a third substrate type: artist-supplied per-part textures + drawing order, restricted to one stylistic domain.
  • Does LICA’s typed scene-graph representation (text/image/vector/group with per-element typography) port well to the gray-background-encoded RGBA latent that ART/LayerDecomp converged on? Vector and text components are not trivially RGBA layers and may need a separate tokenization path.
  • ART’s “anonymous regions suffice; per-region prompts can hurt coherence” finding (ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation §4.2) is at odds with the GLIGEN / InstanceDiffusion lineage of layout control. Is this specific to the multi-layer joint-generation regime, or does it generalize to single-image layout control?
  • What backbone powers Adobe’s “Rotate Object” (Photoshop beta: Rotate Object + Harmonize for 2D images (Adobe product announcement))? Single-image NVS via a video diffusion prior, an image-to-3D pipeline, or a dedicated multi-view diffusion model? Productized but undocumented.
  • Can LICA’s animated-layout subset (27,261 compositions with per-component keyframes) serve as the training substrate for a video-domain analogue of ART — joint generation of multi-layer animated designs with K up to ~50?
  • Can the Live2D-supervision recipe (See-through: Single-image Layer Decomposition for Anime Characters) generalize to other domains with artist-supplied multi-layer source files (e.g. Spine / Adobe Animate / After Effects projects, PSD/AI design files at scale)? Each ships layer order + per-part hidden geometry “for free” as training signal — the bottleneck is data acquisition + parsing infrastructure rather than novel ML.
  • Can the VOID counterfactual-physics recipe (VOID: Video Object and Interaction Deletion) scale beyond the rigid-body / human-object regimes covered by Kubric and HUMOTO? Soft-body deformation, fluid splashes, cloth dynamics, and articulated objects (vehicles, mechanisms) are not represented; the bottleneck is differentiable simulator coverage rather than the diffusion model itself.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-06

Section titled “2025-06”

2025-03

Section titled “2025-03”

2024-11

Section titled “2024-11”

2024-10

Section titled “2024-10”

2024-03

Section titled “2024-03”

Unknown date

Section titled “Unknown date”