Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists

TL;DR

Señorita-2M is a ~2M-pair instruction-based video editing dataset (NeurIPS D&B 2025) built by routing source clips through four in-house specialist video editing models — a global stylizer, a local stylizer, an inpainting expert, and a super-resolution expert — plus 14 modified existing experts, then filtering at both visual-content and instruction levels. Editing tasks are taxonomized into 2 broad classes and 18 sub-categories (object addition / removal / swap, global + local stylization, super-resolution, etc.). Downstream end-to-end editors trained on Señorita-2M with Wan2.1 and CogVideoX-5B as bases are reported to outperform inversion-based baselines on visual quality, frame-to-frame consistency, and instruction following. The dataset is open-sourced on Hugging Face under CC-BY-NC-2.0.

Key claims

End-to-end video editors fail not because the architecture is wrong but because high-quality (source, target, instruction) training triplets are missing; Señorita-2M is built to be that missing supervision substrate [Abstract, §1].
Four newly-trained CogVideoX-based specialists do most of the heavy lifting: a global stylizer, a local stylizer (inpainting + ControlNet hybrid for region-specific appearance/texture edits while preserving the background), an inpainting expert, and a super-resolution expert [§3, GitHub README].
Editing tasks are taxonomized into 2 classes × 18 sub-categories, including object addition, object removal, object swap, global stylization, local stylization, color editing, super-resolution, and motion-level edits (“make the cat’s head lower”, “let the swan spread its wings”) [project page, §3].
A cascade of filtering algorithms (visual-quality filters + instruction-quality filters) is applied to remove poorly-edited pairs; an LLM also rewrites object names and source prompts into clean editing instructions [§3, project page].
Models trained on Señorita-2M on top of Wan2.1 and CogVideoX-5B exhibit “superior visual quality, robust frame-to-frame consistency, and strong instruction following” relative to inversion-based baselines, validating the dataset across two distinct base architectures [§4 / project page].
Inversion-based methods (Video-P2P, FLATTEN) are inherently time-consuming at inference (extra inversion stage), struggle with fine-grained edits, and produce artifacts/jitter; the paper positions Señorita-2M as the input that unlocks fast end-to-end editors as an alternative [§1].
The dataset has been adopted and benchmarked by follow-up work — In-Context Learning with Unpaired Clips (2510.14648) cites it as a primary 2M-pair editing baseline, and OpenVE-3M (2512.07826) runs an explicit ablation using Senorita-2M as the comparison training mix, reporting quality issues at the lower end of the distribution [follow-up citations, not the paper’s own claim].

Method

The data pipeline has three stages. Stage 1 — specialist training: four high-quality editing experts are trained on CogVideoX-5B with task-specific recipes. The local stylizer combines an inpainting head with a ControlNet branch (same architecture as the global stylizer; the differentiating factor is the local mask + control conditioning), letting it modify a region’s appearance / texture / style while leaving the background untouched. Stage 2 — pair generation: the four in-house specialists plus 14 modified existing task experts are used to generate edited target videos from a curated source-video pool, producing ~2M (source, target, instruction) triplets across the 18 sub-tasks. Stage 3 — filtering: a cascade of automated filters operates at both the visual-content level (frame consistency, artifact detection) and the instruction level (LLM-based prompt rewriting from object names / source prompts into clean imperative edit instructions). The resulting dataset has diverse resolutions and frame counts.

To validate the dataset, four downstream editing models are trained — covering four common video-editing architectures — on top of Wan2.1 and CogVideoX-5B; the paper reports quantitative metrics on local-stylization, object-swap, and removal benchmarks (Table 6 in the paper covers local stylization with Ewarp, CLIPScore, Temp-Cons, PSNR, SSIM, LPIPS as the axes).

Results

Headline numbers were not retrievable in detail from the fetched abstract (paper PDF cited via arxiv 2502.06734). The paper reports SOTA local-stylization on standard metrics (Ewarp, CLIPScore, Temp-Cons, PSNR, SSIM, LPIPS) when training on Señorita-2M, and qualitative wins across the project page’s 18-task gallery covering object addition, removal, swap, color edits, global and local stylization, and structural / motion edits.

The dataset itself is the headline result: ~1.95M rows in the HF release, multi-resolution, multi-frame-count, CC-BY-NC-2.0 licensed.

Why it’s interesting

Señorita-2M is the data-side counterpart to the architectural unification trend in video editing — papers like UNIC: Unified In-Context Video Editing (UNIC, Kling) and OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer (OmniTransfer, also Wan-lineage) attack the problem from the model side (“one DiT, many tasks via in-context conditioning”), while Señorita-2M attacks it from the data side (“18 sub-tasks, one dataset, generated by 18 specialists”). The two directions are complementary: a model like UNIC needs (source, target, instruction) triplets across many tasks to fit the in-context paradigm, and Señorita-2M is exactly that supply. It also slots cleanly into the Synthetic Training Data cluster — like Action100M: A Large-scale Video Action Dataset (V-JEPA + VLM + LLM auto-labeling of HowTo100M), it uses a heterogeneous stack of specialist generators plus a verification gate (here, content + instruction filters) to produce supervision at industrial scale without human labelers. The structural analogue on the image side is the layered-image-decomposition lineage (Generative Image Layer Decomposition with Visual Effects (LayerDecomp) etc.) where simulator-generated triplets supervise the model that does the decomposition — Señorita-2M is the video-editing equivalent at 2M scale.