UNIC: Unified In-Context Video Editing

TL;DR

UNIC (UNified In-Context Video Editing) is a Kling/Kuaishou framework that handles diverse video editing tasks — style transfer, video propagation, object insertion/removal/swap, re-camera control — inside a single DiT with no task-specific adapters or DDIM inversion. It packs three token types (source video, noisy target latent, task-varying multimodal conditioning) into one sequence and lets the DiT’s native full-attention do the cross-task mixing. Three small additions make this work in practice: task-specific RoPE rearrangement to disambiguate position, an additive “task-indicator” condition-bias on tokens, and an in-context attention mask that prevents condition tokens from interfering with each other. The companion paper FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers addresses the resulting quadratic cost.

Key claims

A single sequence of [source-video tokens · noisy-target latent · task-specific multimodal condition tokens] processed by the DiT’s native full-attention unifies global editing (style transfer, propagation), local editing (insertion, removal, swap), and re-rendering (re-camera control) inside one model, with no per-task adapter modules and no DDIM-inversion stage [§1, Abstract].
Previous methods cluster into (a) DDIM-inversion-based (Video-P2P, FLATTEN) — sub-optimal performance and ~2× inference cost from the inversion stage — and (b) adapter-based — extra parameter redundancy and task-specific modules per condition signal [project page, Motivation].
Three architectural primitives are introduced on top of the bare “concatenate-and-attend” sequence: task-specific RoPE rearrangement to give each task a distinct positional scheme over the concatenated sequence; conditioning token bias as a task-indicator additive bias that lets the DiT discriminate which task is active without changing tokens; and in-context masking strategy to control attention between condition groups and the noisy target [§3, project page].
Concatenating context conditions into a single token sequence and jointly modeling them with the DiT’s full attention “eliminates the need for task-specific architectural designs” — the central design bet behind in-context video editing [Abstract].
Reported across six diverse video editing and generation tasks the model handles in one weight set [§5 / project page].

Method

UNIC is built on a pretrained text-to-video DiT (lineage of Wan / Kling-class backbones). The architectural primitive is the unified token sequence: source-video latent tokens come first, then the noisy target latent the model is denoising, then a variable-length block of multimodal conditioning tokens whose shape depends on the task (text caption for style transfer; reference image + mask for object swap; camera pose embeddings for re-camera; etc.). The DiT’s existing self-attention layers process this concatenated sequence end-to-end. Three additions sit on top:

Task-specific RoPE rearrangement — each task uses a different positional encoding layout over the concatenated sequence so that the model can distinguish “this region of the sequence is the reference video” vs “this region is the noisy target” without learned segment tokens.
Condition-bias as task indicator — a learned additive bias on the condition tokens encodes which task is active; this is cheaper than a task-embedding lookup and integrates with the existing token stream.
In-context masking — an attention mask gates which condition tokens can attend to which others, preventing e.g. style-transfer text tokens from interfering with object-insertion mask tokens when the model is run on a different task.

The denoising loss is the standard flow-matching / diffusion objective on the noisy target latent; nothing about the underlying DiT training recipe changes.

Results

The paper claims unified handling of style transfer, video propagation, object insertion, object removal, object swap, and re-camera control inside one set of weights, with the model presented as competitive with or superior to task-specific baselines on each. The full quantitative tables were not retrievable at filing (arxiv HTML 429); see project page. Independent later work treats UNIC as a state-of-the-art task-specific expert baseline alongside AnyV2V and VideoPainter [UniVideo, 2510.08377]. The companion paper FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers reports 2–3× per-diffusion-step speedup over this in-context paradigm with minimal quality loss.

Why it’s interesting

Same architectural bet as OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer (also Wan-lineage, also “one model, many reference-conditioned tasks via the DiT’s native attention”) but on the editing side rather than transfer side — UNIC owns global edit / local edit / re-rendering, OmniTransfer owns ID/style/motion/camera/effect transfer of an entire reference video. Both abandon the explicit-control-signal parameterization in favor of in-context conditioning. The shared author roster (Quande Liu, Pengfei Wan, Di Zhang, Kun Gai) suggests this is the unified Kling editing/transfer stack being published in two complementary directions. Connects to Unified Multimodal Models as another Decoupled architecture (the LLM-or-encoder choice is decoupled from the generator), and to the broader in-context-conditioning paradigm that started with FullDiT and now branches into FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers for efficiency.