Qwen-Image-Layered-Control: text-guided layer extraction

TL;DR

A DiffSynth-Studio fine-tune of Qwen’s Qwen-Image-Layered diffusion model that adds text-prompt conditioning to the layer-extraction task. Where the upstream Qwen-Image-Layered emits an unordered set of segmented layers from an input image with no language control, this variant accepts a natural-language description of which layer to extract and returns the single matching RGBA layer. Trained on the artplus/PrismLayersPro dataset with a single-image output head replacing the multi-image one. Native 1024² inference, English/Chinese prompts, negative-prompt support.

Key claims

Architectural change: the model’s output head is switched from multi-image (full layer stack) to single-image (the one layer matching the prompt) [Usage Tips].
Trained on English-only text but inherits Chinese understanding from the base Qwen-Image text encoder [Usage Tips].
Native training resolution is 1024×1024 with inference supported at other resolutions [Usage Tips].
Failure mode called out by the authors: heavily occluded/overlapping entities (e.g. a cartoon skeleton head and its hat) cannot be cleanly separated [Usage Tips].
Performs strongly on poster-like graphics, poorly on photographic images with complex lighting/shadows [Usage Tips].
Supports negative prompts to specify content the user wants excluded from the extracted layer [Usage Tips].

Method

Starts from the open Qwen/Qwen-Image-Layered checkpoint and fine-tunes on artplus/PrismLayersPro, a layered-graphics dataset, until the network learns to gate which output layer fires on the text prompt rather than emitting all layers. The inference pipeline (provided in the model card) plugs three components together: the Qwen-Image-Layered-Control transformer, the standard Qwen-Image text encoder, and the Qwen-Image-Layered VAE for layer-aware decoding; the processor is reused from Qwen-Image-Edit. Call signature is pipe(prompt, layer_input_image=..., layer_num=0, ...) — the user provides the composited input image and the description of the layer they want isolated.

Results

The model card surfaces three qualitative example sets and no quantitative benchmarks. The author’s framing of strengths/weaknesses (works on posters, fails on photographs with complex lighting) is the only systematic evaluation provided. There is no comparison against Qwen-Image-Layered’s stock multi-layer output, nor against other layered-image generators (e.g. LayerDiffuse, MULAN-style RGBA stacks).

Why it’s interesting

Two angles for the team. (1) This is a quick, public demonstration that a text-instruction-following layer extractor can be retrofitted onto a base model that was never trained for instruction-following at all — the architectural change is just swapping the output head, plus a layered dataset. Useful template if anyone here wants language-controlled decompositions out of an otherwise vision-only generator. (2) Sits in the growing pile of Qwen-Image-* derivatives (Qwen-Image, Qwen-Image-Edit, Qwen-Image-Layered, now Qwen-Image-Layered-Control) — the base model is becoming a default open scaffold for image-to-image diffusion tasks, similar to how Qwen-Image-Edit is being repurposed by DiffThinker for reasoning (see DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models).

The layer-decomposition framing also connects to the omnimatte / training-free decomposition strand on the video side (OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models) — but here the decomposer is trained and the input is a single image, not a video.