Region-Constraint In-Context Generation for Instructional Video Editing

TL;DR

ReCo (HiDream-ai, ICML 2026) is yet another instruction-based video editor in the in-context-concatenation family, but with a specific architectural bet: instead of temporal concatenation (UNIC, EditVerse, VideoCoF) it concatenates source and target width-wise (left/right panel), then adds two region-constraint regularizers on top of the standard flow-matching loss. A latent-space term, computed on one-step-backward denoised latents, pushes editing-region latents apart between source and target while pulling non-editing latents together; an attention-space term suppresses attention from target-edit-region tokens to the corresponding source region (interference) while strengthening attention to the target’s own background (harmonization). The companion ReCo-Data dataset — 500K instruction-video pairs across add/remove/replace/style, synthesized via VACE + Gemini-2.5-Flash-Thinking captioning + depth/first-frame conditioning at ~76,800 RTX-4090-GPU-hours + $13.6K of Gemini calls — is publicly released, as are training code and weights under Apache-2.0.

Key claims

Standard in-context video editing (concatenate source + noisy target, denoise jointly) suffers two failure modes the paper isolates: (a) the model cannot reliably localize the editing region from instruction alone, and (b) tokens in the target editing region attend strongly to the same region in the source, causing the new content to inherit the old object’s appearance instead of replacing it [§1, §3].
Width-wise (left/right) concatenation of source and target along the panel axis is the structural choice; the source enters via an auxiliary video condition branch rather than being noised alongside the target [§3, Fig. 2].
Latent-space regularization: a one-step backward diffusion pass produces estimated source and target latents; a pair-wise constraint increases the L2 latent discrepancy inside the editing-region mask and decreases it outside, which steers the joint denoiser to localize modifications [§3.2].
Attention-space regularization: on DiT block attention maps, suppress target-edit-region → source-edit-region attention (the interference path) and strengthen target-edit-region → target-background attention (the harmonization path) [§3.3, Fig. 2].
The two regularizers are added to a vanilla flow-matching loss; no auxiliary task heads, no MLLM bridge, and no DDIM inversion are needed [§3.4].
ReCo-Data: 500K+ instruction-video pairs across four task types (add, remove, replace, style), built by a six-stage pipeline: (1) raw video filtering, (2) Grounding-SAM-style object segmentation, (3) instruction generation via Gemini-2.5-Flash-Thinking, (4) condition pair construction (first-frame editing + depth map for VACE), (5) VACE video synthesis, (6) VLLM filtering and re-captioning (Gemini-2.5-Flash-Thinking again) [project page §ReCo-Data].
Total ReCo-Data synthesis cost: ~76,800 NVIDIA RTX 4090 GPU-hours plus ~$13,600 in Gemini-2.5-Flash-Thinking API calls [project page §Collection Pipeline].
ReCo is built on a Wan-family base (the repo lists WAN, ObjectClear, VACE, and Flux-Kontext-dev as direct inspirations) [GitHub README].
Reports state-of-the-art across four instruction-based video editing tasks vs prior instructional and captioning-based pipelines, per the paper’s own ReCo-Bench harness (Gemini-2.5-Flash-Thinking as VLM judge) [§4, §abstract].
A follow-up (Kiwi-Edit, NUS, 2026-03-05) curates HQ-ReCo from ReCo-Data and adds reference image pairs, indicating the dataset is being adopted as a substrate by other groups [GitHub README].

Method

ReCo is built on a pre-trained text-to-video DiT (Wan lineage). At training time the source video latent and the noised target latent are concatenated along the width axis (left/right panel) rather than along the temporal axis as in UNIC, EditVerse, and VideoCoF. The source is fed through an auxiliary video condition branch — i.e. it is not itself denoised; only the target half of the panel carries flow-matching noise. The DiT’s native self-attention then mixes source and target tokens within the panel-concatenated sequence; the text instruction enters via cross-attention.

The two regularizers are the load-bearing additions. For the latent term, the model runs a single backward diffusion step from the current noise level on both halves to produce estimated source and target latents, then computes per-token differences. Inside a binary editing-region mask (derived from the target video at data-curation time via Grounding-SAM), the loss maximizes the source/target latent distance; outside the mask, it minimizes it. The effect is to enforce “change here, preserve there” at the latent level without an explicit segmentation head at inference. For the attention term, the same mask is projected onto the DiT block attention maps; the loss suppresses attention weights from target-edit-region queries to source-edit-region keys, and strengthens attention from target-edit-region queries to target-background keys. The combined objective is L_flow + λ_lat · L_lat + λ_attn · L_attn.

ReCo-Data, the companion 500K-pair dataset, is built end-to-end with VACE-14B as the in-context video generator: a Gemini-2.5-Flash-Thinking agent generates per-video editing instructions; an off-the-shelf first-frame editor (Flux-Kontext-dev family) produces the edited keyframe; a video depth predictor produces the per-clip depth video; VACE generates the edited video from those conditions; Gemini-2.5-Flash-Thinking then filters and re-captions the resulting triplets. The pipeline is structurally identical to Ditto-1M’s recipe in Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) — both use VACE as the in-context generator and a Gemini-class agent at both ends of the loop — but at half the scale (500K vs 1M) and four tasks vs Ditto’s global+local split.

Results

The paper claims superiority across four instruction-based video editing tasks (object addition, object removal, object replacement, video stylization) on the introduced ReCo-Bench, using Gemini-2.5-Flash-Thinking as a VLM-as-judge over multiple evaluation dimensions [§4, §abstract; ReCo-Bench README]. The full quantitative tables were not retrievable from the project page (which carries only qualitative example videos) or from arxiv at filing time; see arxiv:2512.17650 §4 for the per-task numbers. Headline qualitative claims from the project page: replacement examples preserve background composition under significant identity changes (human → cartoon penguin, robot vehicle → Jeep SUV); addition examples insert new objects (deer crossing road, mushroom next to woman) without disturbing the surrounding scene; removal examples cleanly erase objects with effects (helicopter, bird on trunk); stylization covers flat vector, watercolor, 3D Chibi, LEGO, comic, Japanese anime, Paper Cutting, claymation, etc. — all reported as temporally coherent vs baselines.

The architectural ablations the paper claims (latent regularizer on/off, attention regularizer on/off, joint vs source-only conditioning) are not retrievable at filing but are the necessary load-bearing evidence — the design’s whole pitch is that the two regularizers individually contribute, separately from the width-concat backbone.

Why it’s interesting

ReCo is the fourth major architectural family for unified instruction-based video editing now on the wiki, and the contrast with its three siblings is sharp:

Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) (Ditto / Editto) — temporal-concat, no regularizers; bets on data scale (1M Ditto-1M + modality-curriculum learning) and uses VACE only at the data-generation layer.
UNIC: Unified In-Context Video Editing (UNIC, Kling) — temporal-concat with task-specific RoPE rearrangement and a task-indicator bias on condition tokens; bets the DiT’s native attention is sufficient if you disambiguate position.
VideoCoF: Unified Video Editing with Temporal Reasoner (VideoCoF) — temporal-concat with an explicit reasoning-frames segment between source and target; bets the in-context recipe needs a visual chain-of-thought spatial cue.
ReCo (this paper) — width-concat with two attention/latent regularizers; bets the in-context recipe needs implicit region cues injected through the loss rather than through tokens or RoPE.

The width-concat choice is a non-obvious design alternative to the temporal concatenation everyone else converged on: it means source and target tokens see each other through spatial self-attention rather than across the temporal axis. Whether that matters for editing quality vs the regularizers is the paper’s open empirical question — the design isn’t ablated against a temporal-concat baseline with the same regularizers.

The dataset story is the second contribution worth flagging. ReCo-Data is structurally very close to Ditto-1M (Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto)): both use VACE + a Gemini-class agent + first-frame conditioning + depth video. The fact that two groups independently converged on this recipe within a few months, and that a third group (Kiwi-Edit) is already curating HQ-ReCo from it, suggests the “VACE-as-data-engine for instruction video editing” recipe is now standard. The differences are in scale (500K vs 1M), task coverage (4 explicit tasks vs Ditto’s global/local split), and cost transparency (ReCo publishes the 76,800 GPU-hour / $13.6K Gemini-API total).

Independent of the data, the attention-regularization-as-region-cue trick should generalize beyond editing. Any in-context generation task where the model has tokens for both the conditioning content and the target content — image editing, video editing, in-context video generation — has the same “target tokens attend to source tokens of the same region and inherit appearance” failure mode that ReCo’s attention term explicitly suppresses. The recipe ports trivially.