VideoCoF: Unified Video Editing with Temporal Reasoner

TL;DR

VideoCoF takes the “thinking with modalities” idea and applies it to instruction-based video editing: instead of concatenating clean source frames with noisy target frames and hoping self-attention figures out where the instruction applies, the model is forced to emit reasoning frames — a short clip of progressively gray-highlighted edit regions — before generating the edited target. Built on WAN-14B with a 50k-triplet curated dataset (vs ICVE’s 1M+150k), it reports +15.14% Instruction Following and +18.6% Success Ratio over ICVE on the new VideoCoF-Bench. A second contribution is a temporal-RoPE alignment trick that resets source and target indices to share the range [1, F] while pinning the reasoning clip at index 0, which both prevents index-collision artifacts in the first generated frame and enables 4× length extrapolation (33-frame training → 141-frame inference).

Key claims

Naive temporal in-context concatenation (UNIC / ICVE / EditVerse-style) lacks an explicit mapping between the editing instruction and the spatial region to edit, producing weak accuracy on multi-instance and spatial-reasoning cases [§1, Fig. 2].
Inserting explicit reasoning frames between source and target in the temporal-concat sequence enforces a “see → reason → edit” procedure and lifts Instruct Follow by +10.65% and Success Ratio by +5.46% on VideoCoF-Bench at fixed RoPE design [§4.4 ablation, Table 2 col 1 vs col 3].
The optimal reasoning-frame format is a progressively transparent gray highlight (0 → 75%) of the edit region; pure black masks fail on deletion tasks and red-tinted highlights mis-localize, while gray with progressive transparency wins on all GPT-4o metrics [§4.4 ablation, Table 3, Fig. 9].
The naive RoPE reset (source 0..F-1, reasoning 0, target 0..F-1) introduces index collisions at temporal position 0 shared by source, reasoning, and target frames, which leaks reasoning-mask artifacts into the first target frame [§3.3, Fig. 4].
Their fix — set source and target to 1..F, reasoning fixed at 0 — isolates the reasoning token, prevents artifact leakage, and lets a 33-frame-trained model run at 81 and 141 frames without blurriness or motion misalignment [§3.3, §4.4 RoPE ablation, Fig. 7].
VideoCoF on 50k curated video pairs beats ICVE pre-trained on 1M + fine-tuned on 150k samples by +15.14% Instruct Follow (8.97 vs 7.79), +0.14 Preservation (8.20 vs 8.06), and +18.60% Success Ratio (76.36% vs 57.76%) on VideoCoF-Bench [Table 1].
The data pipeline uses Qwen-VL 72B for multi-instance identification, Grounding-SAM2 for instance segmentation, Minimaxremover for object-erase pairs (with the addition direction built by reversing removal), and VACE-14B inpainting + GPT-4o prompts for swap and local style transfer; final filtering uses Dover Score + VIE Score [§3.5].
VideoCoF-Bench: 200 videos from Pexels across four tasks (Object Removal / Addition / Swap / Local Style Transfer) with half of each task being instance-level cases requiring left/right or multi-instance disambiguation; evaluation uses GPT-4o-as-judge over Instruct Follow, Preservation, Quality, Success Ratio [§4.2].

Method

The architecture is a WAN-14B VideoDiT operating on a temporally-concatenated three-segment latent: clean source frames at one end, noised reasoning frames in the middle, and noised target frames at the other end. The Video VAE encodes each segment separately to preserve intra-causal relations and inter-video independence, then they are stacked along the temporal axis. Self-attention does the in-context mixing across all three segments; cross-attention handles the language instruction. Only the reasoning and target latents are noised during training — the source stays clean — and the loss is a velocity-prediction MSE supervised only on reasoning + target frames.

Two non-obvious design choices: (1) the reasoning ground truth is a progressively transparent (0% → 25% → 50% → 75%) gray highlight over the edit region, derived from Grounding-SAM2 masks during data curation, chosen over black or red because video diffusion models are insensitive to extremal pixel values; (2) the 3D factorized RoPE is configured so that source frames carry temporal indices 1..F, the reasoning clip carries 0, and the target frames re-use 1..F. This index re-use is what enables length extrapolation: the model never sees a temporal index larger than the training frame count, so the learned position-to-content mapping is invariant to absolute frame index. The reasoning clip’s solo position at 0 keeps it isolated from the target frames it conditions.

The training data is curated through an instance-level pipeline: Qwen-VL 72B filters Pexels videos for multi-instance scenes; Grounding-SAM2 produces per-instance masks; Minimaxremover removes a specific instance to produce object-removal pairs; the reverse direction gives object-addition pairs; VACE-14B inpainting with GPT-4o prompts gives swap and local-style-transfer pairs. The output is 50k filtered triplets at multiple aspect ratios (336×592 through 400×944 and verticals). Training runs for ~8k iterations at global batch size 16 with AdamW.

Results

VideoCoF-Bench (averaged over Removal / Addition / Swap / Local Style Transfer, GPT-4o judge): Instruct Follow 8.97 (ICVE 7.79; Lucy Edit 5.24; VACE 7.47; Señorita 3.26; InsV2V 3.41); Preservation 8.20 (ICVE 8.06); Quality 7.77 (ICVE 8.14 wins this column); Success Ratio 76.36% (ICVE 57.76%; next-best Lucy Edit 29.64%) [Table 1].
Perceptual quality: CLIP-T 28.00 (ICVE 27.49); CLIP-F 0.992 (close to top); DINO 0.991 (top) [Table 1].
Naive in-context vs CoF ablation: at fixed RoPE design, removing the reasoning frames drops Instruct Follow from 8.97 to 8.11 and Success Ratio from 76.36% to 72.41% [Table 2 col 1 vs col 3].
RoPE ablation: the source-target-overlap design (0..F-1 for both) drops Preservation by 0.41 points and DINO by 0.009 vs the disjoint-with-reasoning-at-0 design — and visually causes blurriness and motion misalignment when extrapolating from 33 to 81 frames [Table 2 col 2 vs col 3, Fig. 7].
Length extrapolation: the model is trained on 33-frame video pairs (33 source + 4 reasoning + 33 target) and reported to run at 141 frames (4× training length) at inference without quality degradation [Fig. 1, §4.4].
Reasoning-format ablation: progressive gray (0–75%) beats static 50% gray, 50% red, and full black mask on every GPT-4o metric; static gray-50% itself already beats red and black [Table 3, Fig. 9].
Qualitative failure modes of baselines on instance-level cases: ICVE confuses left/right instances on multi-instance removal; VACE alters non-target people on face/clothing swap; Lucy Edit changes clothing instead of face on the same prompt; VideoCoF correctly localizes the edit in all four [§4.3, Fig. 6].

Why it’s interesting

Three things to flag for the team:

This is a clean instance of the Thinking with Modalities substrate (b) “diffusion-native CoT” applied to editing. DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models used image diffusion to reason about answers (mazes, Sudoku, TSP) — VideoCoF uses video diffusion to reason about where to act, which is structurally similar (model emits a visual intermediate before the final visual answer) but is targeted at controllability rather than problem-solving. The Chain-of-Frames name and the “see → reason → edit” framing also directly echo Demystifying Video Reasoning‘s Chain-of-Steps finding — though VideoCoF reasons along the temporal axis (extra frames) while CoS reasons along the denoising axis (extra steps). Both substrates are open; whether they compose is untested.
This is the third major architectural family for unified video editing now filed, and it’s worth contrasting head-on with EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning and UNIC: Unified In-Context Video Editing. EditVerse argues the in-context recipe just needs more data + a 4D RoPE + <sov>/<eov> brackets and no spatial cues are necessary. UNIC adds task-specific RoPE rearrangement and a condition-bias task indicator. VideoCoF goes further and says: no — the in-context recipe is missing an explicit spatial cue, and the cleanest way to add one is to make the model emit it as reasoning tokens. The 50k-vs-1M data-efficiency claim against ICVE is the strongest argument for the page-level “explicit spatial reasoning is doing real work” thesis; the headline numbers all come from a self-chosen GPT-4o judge on a self-built benchmark, so the comparison needs replication.
The RoPE-index-collision finding is more broadly useful than the editing application. Any in-context-concatenation scheme that re-uses positional indices across source and target segments (EditVerse, UNIC, OmniWeaving) is at risk of the same first-frame-artifact failure if a third segment shares index 0. VideoCoF’s solution — disjoint reasoning at 0, repeated 1..F for source and target — is a small, reusable trick for anyone training a temporal-concat in-context video model.