OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models

TL;DR

OmnimatteZero is a training-free omnimatte pipeline that decomposes a video into background + foreground object layers (with their associated effects — shadows, reflections, smoke) using only off-the-shelf pre-trained video diffusion models (LTX-Video 0.9.1 → upgraded to 0.9.7 in the released code). Two core mechanisms: (1) self-attention-derived effect masks — the diffusion model’s own self-attention reveals which background regions are causally linked to the object, supplying the “object + effects” mask without supervision; (2) latent arithmetic — encoding the original video and the object-removed clean background, taking the latent difference to isolate the foreground layer, then re-adding that layer to a new background’s latents and running a short noising–denoising refinement. SIGGRAPH Asia 2025; runs at 25 FPS on an A100, claimed fastest omnimatte to date. Of direct interest to anyone building layered video editing or compositing on top of pretrained video DiTs.

Key claims

Pretrained video diffusion models associate objects with their effects (shadows, reflections) inside their self-attention maps, so the model already “knows” which background pixels co-vary with the object [§3, Fig. 1; project page §Self-Attention].
Zero-shot image inpainting techniques, applied frame-by-frame, fail at video object removal — they leak content, lose temporal consistency, and don’t remove effects [§3 motivation, project page].
A spatio-temporal self-attention map derived from a single forward pass (with controlled noise at flow-matching t=0.5) can be thresholded to expand an object mask into an object-plus-effects mask, eliminating the need for a separately trained effects detector [README §self_attention_map.py, §3.2].
Foreground layers can be isolated by simple latent arithmetic: encode the original video and the inpainted clean background, take the latent difference, and that difference is a transferable foreground layer [§3.3, project page §Foreground extraction].
Compositing onto a new scene works by adding the foreground latents to a new background’s encoded latents and running a short refinement (few-step noising–denoising at low denoise strength), preserving the object plus its shadows/reflections [§3.4, README §foreground_composition.py].
The released code on LTX-Video 0.9.7 achieves “good object removal results without explicit temporal and spatial attention guidance,” indicating that the underlying base model’s quality dominates the need for the paper’s TAP-Net-driven guidance modules [README “Important Note on Attention Guidance”].
Achieves real-time performance — 25 FPS on a single A100, claimed as the fastest omnimatte approach published [project page §Abstract; author bio at levymsn.github.io].

Method

Inputs are an RGB video and an object mask (which the user can produce with SAM2). Stage 0, optional: derive the total mask (object + effects) via self-attention. The video is encoded to latent space with the LTX VAE, controlled noise is injected (flow matching at t=0.5), and a single forward pass through the transformer collects self-attention maps from all 48 layers. For each frame, the script computes how strongly each spatial position attends to the object regions across all frames; values are upsampled, adaptively thresholded (default mean + 0.5·std), and morphologically dilated, then unioned with the input object mask to yield the total mask. Empirically, mask expansion ratios are 1.5–2× of the object area. The paper also describes Temporal and Spatial Attention Guidance modules driven by TAP-Net point tracking, but the released code (LTX-Video 0.9.7) runs without them and still gets good results.

Stage 1 is object removal as video inpainting. The total mask indicates regions to inpaint; the pre-trained video diffusion model fills them in while maintaining temporal consistency — i.e. the inpainting is a normal sampling pass with mask-conditioned conditioning, not a fine-tuned removal model. Output: a clean background video.

Stage 2 is foreground extraction by latent arithmetic. Encode the original video → z_orig; encode the clean background → z_bg. The latent foreground is z_fg = z_orig − z_bg. Pixel-space injection in the object region (overwriting the difference with the original pixels there) preserves fine details that the latent encoder smooths over.

Stage 3 is layer composition. Encode the new target background → z_newbg. Form z_composed = z_newbg + z_fg. Then run a short refinement pass: re-noise to a low timestep and denoise for a handful of steps (denoise_strength ≈ 0.3) to clean up boundary artifacts and harmonize colors / lighting. Default sampling uses 30 inference steps and guidance scale 3.0.

Results

Speed: 25 FPS on a single NVIDIA A100, described by the authors as a “new record for the fastest Omnimatte approach” and as real-time [project page §Abstract; author bio]. Memory: paper recommends 32GB+ VRAM for inference.
Quality: “Superior performance in terms of background reconstruction” relative to prior omnimatte methods, evaluated on standard omnimatte benchmarks [project page §Abstract]. Specific tables/FID/PSNR numbers live in the paper PDF — only abstract-level claims surface on the project page.
Effect coverage: demo gallery includes hair, fur, foliage, shadows, reflections, smoke, rippling water — claimed to be preserved during extraction and removed during inpainting [techxplore press; project page demos].
Ablation on attention guidance: the released LTX-Video-0.9.7 code achieves “good object removal results without the explicit temporal and spatial attention guidance” that the paper proposed for LTX-Video-0.9.1 — i.e. base-model quality erodes the marginal value of the proposed guidance modules [README “Important Note on Attention Guidance”].
Caveat: the project page does not surface quantitative tables; the abstract-level “superior background reconstruction” + “fastest” claims need the PDF to verify against specific baselines (Omnimatte3D, Layered Neural Atlases, OmnimatteRF).

Why it’s interesting

This is the second paper filed in two days that re-uses a pre-trained video diffusion model as a training-free engine for a per-pixel video understanding task — the VideoMaMa: Mask-Guided Video Matting via Generative Prior route is “fine-tune SVD into a one-step mask→matte head, then distill into SAM2-Matte”; OmnimatteZero’s route is “don’t fine-tune at all, just read self-attention maps and do latent arithmetic.” Both lean on the same prior — that a video generation backbone has internalized object–effect coupling and natural motion statistics — but partition the work differently: VideoMaMa pays a fine-tune cost up-front to get a fast, generalizable matter; OmnimatteZero pays nothing up-front but ties itself to the specific quirks (and bugs) of the base diffusion model (cf. the LTX-0.9.1 → 0.9.7 attention-guidance regression noted in the README). For a Luma layered-editing or compositing stack, the appealing primitive here is the latent-difference foreground layer: if z_fg = z_orig − z_bg is a transferable layer that re-composes via simple latent addition + short refinement, that is a much cleaner mental model than alpha-compositing in pixel space — and it composes naturally with any video DiT that shares a VAE latent space.

The bigger pattern: across Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention (TempCache + AnnCA + AnnSA — training-free attention surgery for AR video diffusion) and OmnimatteZero (training-free attention readout for video decomposition), the same Dvir Samuel / Bar-Ilan / NVIDIA group is pushing a consistent thesis: meaningful video tasks can be solved by interpreting and editing the attention maps of a frozen pre-trained video DiT rather than by adding new training. Worth tracking as a research thread.