Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto)

TL;DR

Ditto is a data-synthesis pipeline that produces Ditto-1M, a 1M-triplet (source-video, instruction, edited-video) dataset at 1280×720, 101 frames, 20 FPS for instruction-based video editing — and Editto is the SOTA editor trained on it. The pipeline’s key move is to use an instruction-image-editor (Qwen-Image-Edit) to make a high-quality edited keyframe, then push that keyframe + a CoTracker3-derived depth video through an in-context video generator (VACE on Wan2.1) so the keyframe edit propagates coherently across the clip. To make 12,000 GPU-days of this affordable, the in-context generator is a distilled few-step variant with a temporal enhancer (Wan2.2 MoE fine denoiser, 4-step low-noise reverse process). A Qwen2.5-VL agent generates the instructions and filters the outputs. Editto then bridges the visual→textual conditioning gap with modality curriculum learning — start with the edited keyframe + instruction, anneal the keyframe to zero, end up as a purely instruction-driven editor.

Key claims

The pipeline produces 1M (source, instruction, edited) video triplets at 1280×720 / 101 frames / 20 FPS — strictly larger and higher-quality than prior datasets: InsViE (1M, 1024×576, 25 frames, 7 FPS), InsV2V (404k, 256×256, 16 frames), EffiVED (155k, 512×512, 8 frames) [Table 1].
Roughly 700k of the 1M triplets are global edits (style, environment, season, lighting) and ~300k are local edits (object swap / add / removal); source videos are filtered Pexels footage rather than uncurated web scrapes [§3.6, §3.1].
“Lift-and-propagate” prior pipelines (VEGGIE, InsViE) edit a keyframe with an image editor then use an image-to-video model to propagate; their temporal coherence and visual quality are capped by the I2V propagator. Ditto replaces the I2V step with an in-context video generator (VACE) that conditions jointly on the edited keyframe (appearance), a depth video (structure/motion), and the text instruction (semantics) [§2.2, §3.4].
Computational cost is reduced to ~20% of the unoptimized baseline (≈50 GPU-min/sample on a single GPU) via post-training quantization of the VACE generator + a distilled few-step student (Yin et al. 2025) [§3.4].
After in-context generation, every triplet passes a Wan2.2-MoE fine-denoiser refinement: small Gaussian noise + 4-step low-noise reverse process targets perceptual artifact removal without semantic drift, leveraging Wan2.2’s high-noise-vs-low-noise MoE split [§3.5].
A Qwen2.5-VL agent does both ends of the loop: it generates dense source-video captions, conditions on the caption to write the editing instruction, then later judges every (source, instruction, edited) triplet on (a) instruction fidelity, (b) source-content preservation, (c) visual quality, and (d) safety [§3.2, §3.5].
Editto is fine-tuned on Ditto-1M from the VACE+Wan2.1 in-context generator: only the linear projection layers of the context blocks are unfrozen; the rest stays frozen. 16,000 steps × AdamW × LR 1e-4 × 64 H-series GPUs. First 5,000 steps are the curriculum warm-up [§5.1].
Modality curriculum learning (MCL): at the start of training every example is conditioned on text instruction + edited reference keyframe; the probability of providing the keyframe is annealed to zero over the curriculum, so by the end the model edits from text alone. Ablation shows that without MCL the model “often struggles to interpret the instruction’s full semantic intent” [§4, §5.3, Fig. 8].
Editto wins on every reported metric vs TokenFlow / InsV2V / InsViE: CLIP-T 25.54 vs 23.63 / 22.49 / 23.56; CLIP-F 99.03 vs 98.43 / 97.99 / 98.78; VLM score 8.10 vs 7.10 / 6.55 / 7.35. 1,000-rating human user study (Edit-Accuracy / Temp-Consistency / Overall, 1–5) puts Editto at 3.85 / 3.76 / 3.86 vs the next-best 2.28 / 2.30 / 2.36 (InsViE) — a ≈+1.5 absolute gap on every axis [Table 2].
Sim2real bonus: by training Editto to map stylized Ditto outputs back to the real source videos, the same model gains a synthetic-to-real capability — i.e. the Ditto-1M data carries enough photorealistic structure to reverse the edit direction [Fig. 6].
Editto outperforms the raw data generator (the in-context VACE+Wan2.1 pipeline) on newly-encountered content, which the paper attributes to training-data filtering plus the curriculum strategy [Fig. 7].

Method

The Ditto pipeline runs four stages per source video. (1) Filtering: Pexels-licensed footage is deduplicated via DINOv2 features + similarity threshold, then scored for motion via CoTracker3 trajectory displacement; static videos are dropped. Survivors are resized and resampled to 20 FPS. (2) Instruction generation: a Qwen2.5-VL agent writes a dense caption of the source video and then, conditioned on the caption, drafts an editing instruction (global style/environment or local object operation). (3) Visual-context preparation: an image editor (Qwen-Image-Edit class) applies the instruction to a chosen keyframe → edited reference frame; a video depth predictor produces a per-frame depth video for the entire clip. (4) In-context generation: VACE (built on Wan2.1) takes the text instruction + edited keyframe (as appearance condition) + depth video (as spatiotemporal scaffold) and produces the edited video via its full-attention in-context conditioning mechanism. VACE is quantized post-training and replaced with a distilled few-step student (Yin et al. 2025) so each sample costs ~20% of the unoptimized run. (5) Curation/enhancement: the Qwen2.5-VL agent rejection-samples triplets on instruction fidelity / source preservation / visual quality / safety; survivors are pushed through Wan2.2’s fine-MoE-denoiser for a 4-step refinement targeting only low-noise textural artifacts. Total compute: ~12,000 GPU-days for 1M triplets.

Editto itself reuses the VACE+Wan2.1 weights as init. The Context Branch ingests the source video and (during MCL warm-up) the edited reference frame; the DiT-based Main Branch denoises the edited target conditioned on the joint visual context + text-instruction embeddings. Training uses standard flow-matching loss on the edited target latent. The curriculum is the load-bearing post-training trick: it lets a model that was pretrained as a reference-image-conditioned generator learn the harder text-only task without destabilizing.

Results

Automatic metrics on the editing benchmark: CLIP-T 25.54 (text-video alignment), CLIP-F 99.03 (inter-frame consistency), VLM holistic score 8.10 / 10 — all SOTA vs the three baselines [Table 2].
1,000-rating user study, 1–5 scale: Edit-Acc 3.85, Temp-Con 3.76, Overall 3.86 — comfortably above InsViE (next best) at 2.28 / 2.30 / 2.36 [Table 2].
Scaling ablation: editing fidelity to source content + style-edit quality both improve monotonically as the Ditto-1M training subset grows; data scale is empirically still on the upward slope [Fig. 8].
MCL ablation: removing the curriculum (training directly from text-only conditioning) yields visibly worse instruction-following — confirms the visual-scaffold-then-anneal recipe is doing real work and isn’t replaceable by longer training [Fig. 8].
Qualitative wins on stylization (temporally coherent vs blurry / inconsistent baselines) and local edits like “black suit” (preserves identity and background where Gen4-Aleph slightly shifts identity and earlier baselines fail) [§5.2, Fig. 5].
The trained Editto generalizes beyond the raw VACE+Wan2.1 generator on out-of-distribution content [Fig. 7].

Why it’s interesting

This is the cleanest current example of the “data side beats architecture side” thesis for instruction-based video editing — close cousin to Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (Señorita-2M, 2M pairs via 4 specialist editors + 14 modified experts on CogVideoX-5B) but with a fundamentally different generation primitive: rather than per-task specialists, Ditto uses one in-context video generator (VACE) conditioned on a strong keyframe prior from an image editor. The architectural counterpart that consumes this kind of data is UNIC: Unified In-Context Video Editing — UNIC’s “one DiT, many tasks via in-context tokens” recipe needs exactly the (source, target, instruction) triplets Ditto-1M and Señorita-2M supply. The modality curriculum learning trick — start with a visual scaffold, anneal to text — is a transferable idea: it converts an in-context video generator that the community already has good base weights for into a text-only editor without the usual fine-tuning instability, and the same trick should apply anywhere a strong reference-conditioned generator wants to become a purely text-conditioned one. Independent of the data, the pipeline is also a non-trivial datapoint for Diffusion Distillation: it shows that a distilled few-step VACE student + a 4-step Wan2.2-fine-MoE refinement can replace a 50-GPU-min/sample teacher at the data-generation layer, not just at inference, with quality good enough to train a SOTA editor on. Hosted on fal as “Editto” per the Slack pointer.