VideoMaMa: Mask-Guided Video Matting via Generative Prior

TL;DR

VideoMaMa (Video Mask-to-Matte Model) converts coarse segmentation masks into pixel-accurate temporally-consistent alpha mattes by fine-tuning Stable Video Diffusion as a mask-to-matte conditional generator. Trained purely on synthetic data, it generalizes zero-shot to real footage; the authors then use it as a pseudo-labeler to build MA-V, a 50K+ real-video matting dataset, and fine-tune SAM2 on it to obtain SAM2-Matte — a feed-forward matter that beats SAM2 trained on existing matting data on in-the-wild video. CVPR 2026. Of direct interest to anyone using alpha mattes as conditioning or compositing signals in video generation pipelines.

Key claims

Pretrained video diffusion models carry a strong enough prior over natural-video appearance and temporal dynamics that they can be fine-tuned into a video matting head, without needing matting-scale labeled video [Abstract].
Trained only on synthetic data, the SVD-based matter generalizes zero-shot to real-world videos across diverse scenes and motions [Abstract, project page].
A scalable pseudo-labeling loop — synthetic-trained VideoMaMa → applied to 50K+ in-the-wild videos → MA-V dataset — produces matting annotations that improve a feed-forward student (SAM2) more than existing matting datasets do [Abstract].
The student model SAM2-Matte, obtained by fine-tuning SAM2 on MA-V, outperforms SAM2 fine-tuned on existing matting datasets on robustness to in-the-wild videos [Abstract].
A coarse segmentation mask is a sufficient conditioning signal for matting once a video generative prior is in place — no trimap required [Abstract, title].

Method

The pipeline has two stages. Stage 1 is the diffusion-prior matter: take Stable Video Diffusion (img2vid-xt), keep its temporal U-Net architecture, and re-purpose it as a conditional one-step generator that maps (RGB frames, coarse mask) → alpha matte sequence. Training is on synthetic composites (clean foregrounds composited onto background videos so ground-truth alpha is known by construction), with a DINO-feature projection MLP (dino_projection_mlp.pth in the repo) suggesting a perceptual / feature-space loss in addition to the diffusion objective. Inference is inference_onestep_folder.py — i.e. the model is run as a single denoising step, not the full SVD multi-step trajectory.

Stage 2 is distillation into a feed-forward matter via pseudo-labeling. VideoMaMa is applied to ~50K real in-the-wild videos to produce the Matting Anything in Video (MA-V) dataset. SAM2’s mask decoder is then fine-tuned on MA-V (with its segmentation masks as the coarse input cue) to predict alpha directly, yielding SAM2-Matte — a lightweight, real-time-friendly model with no diffusion in the loop.

Results

Zero-shot generalization to in-the-wild videos despite training only on synthetic composites (qualitative video gallery on the project page; quantitative tables in the paper’s results section, not in the abstract).
SAM2-Matte > SAM2-fine-tuned-on-prior-matting-datasets on in-the-wild robustness — the claim that motivates publishing MA-V [Abstract].
The released artifacts already have downstream adoption: ComfyUI-VideoMaMa (Feb 2026), CorridorKey (Mar 2026), Sammie-Roto-2 (Apr 2026) [README “News”]. (Adoption signal, not a benchmark.)

Why it’s interesting

For Luma, the result is two leverage points stacked on top of each other: (1) SVD’s learned prior over natural-video appearance is strong enough to be repurposed for a dense-per-pixel perception task with no real labeled data — a fine-tunable video-generation backbone bought you a matting model for free. (2) That generative-prior model is then distilled into a non-generative student (SAM2-Matte) via pseudo-labels, recovering real-time inference. This is a recipe — diffusion model as data engine for a feed-forward perception head — that generalizes well beyond matting (depth, normals, optical flow, intrinsics, video relighting cues). The mask-conditioned formulation is also directly compatible with how segmentation-driven editing pipelines already work, so alpha mattes from this become drop-in conditioning for compositing or layered video generation.