OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer

TL;DR

OmniTransfer (ByteDance Intelligent Creation Lab) is a single Wan2.1-I2V-14B-based framework that unifies five reference-video customization tasks — ID, style, motion, camera-movement, and effect transfer — by treating the reference video as a parallel branch that interacts with the target only through one-directional cross-attention. Three contributions make the unification work: Task-aware Positional Bias (a spatial-width RoPE offset for temporal tasks, a temporal-frame RoPE offset for appearance tasks), Reference-decoupled Causal Learning (kills bidirectional copy-pasting and shortens inference 20%), and Task-adaptive Multimodal Alignment (Qwen-2.5-VL with task-specific MetaQueries replaces T5). It matches pose-guided motion-transfer SOTA without using any pose prior, and beats the camera/effect baselines decisively on user studies.

Key claims

A single model handles ID, style, motion, camera, and effect transfer, plus unseen task combinations, by switching only a per-task mask-latent flag and a per-task RoPE offset [§4, Table 1, Fig. 1].
Video diffusion models inherently maintain temporal consistency through spatial context — side-by-side T2V generations stay motion-consistent across columns while two consecutive shots drift; this motivates using a spatial-width RoPE offset to inherit temporal alignment from spatial in-context learning [§4.2, Fig. 3].
Task-aware Positional Bias adds offset (0, W_target, 0) to the reference’s 3D RoPE for temporal tasks and (F_target, 0, 0) for appearance tasks, with no other architectural change [§4.2, Eq. 2].
Reference-decoupled Causal Learning replaces joint self-attention with one-way attention: reference does intra-branch self-attention; target concatenates reference K/V into its self-attention — eliminates the “copy-paste” artifact joint attention causes [§4.3, Eqs. 3-4].
Fixing the reference branch’s diffusion-time embedding to t=0 makes it time-invariant, so it runs once per generation; this drops total inference from 180s → 145s, a ~20% wall-clock reduction vs. the full-attention baseline [§4.3, Table 7].
Replacing Wan’s T5 text encoder with Qwen-2.5-VL (LoRA-tuned) plus per-task learnable MetaQuery tokens injected only into the target branch lifts both appearance and temporal user-study scores by ~0.2 points [§4.4, Table 7 +TMA row].
Reference latent is kept noise-free (no timestep noise added) to maximally preserve information, in contrast to standard noisy-conditioning setups [§4.1].
On ID transfer, OmniTransfer beats ConsisID, Phantom, and Stand-in on all three video-similarity face-recognition metrics — VSim-Arc 0.48 vs 0.45 (Phantom), VSim-Glint 0.51 vs 0.47 [Table 2].
On style transfer beats StyleCrafter and StyleMaster on VCSD (0.51 vs 0.29), CLIP-T (27.16 vs 26.82), and aesthetics (0.61 vs 0.59) [Table 3].
On effect transfer (user study, 5-point scale) scores 3.45 effect-fidelity vs 1.95 (Seedance I2V) and 1.81 (Wan 2.1 I2V) — text-only conditioning is insufficient for unseen VFX [Table 4].
On camera-movement transfer beats MotionClone and CamCloneMaster decisively in a user study: 4.19 vs 1.79 camera-fidelity and 3.89 vs 1.45 image-consistency [Table 5].
On motion transfer matches WanAnimate (which uses a larger 28B Wan2.2 and explicit pose input) without any pose guidance — 3.62 vs 3.71 motion-fidelity, 3.88 vs 3.53 image-consistency [Table 6].
Three-stage training: (i) train DiT blocks via in-context learning, (ii) freeze DiT and train the MLLM connector, (iii) joint fine-tune — 10k / 2k / 5k steps respectively, lr 5e-5, batch 16 [§5.1].
Ablation order Baseline → +TPB → +RCL → +TMA monotonically improves both appearance and temporal user-study consistency/quality, and only the +RCL step yields wall-clock improvement (180→142s) [Table 7].

Method

OmniTransfer is built on Wan2.1-I2V-14B. The target latent is constructed as in the base model (noised VAE latent + condition image latent + binary mask). A separate reference latent is built from the reference video’s clean VAE features, with a task-specific flag value baked into the mask channel (0.5 temporal, 0.7 ID, 0.9 style) — this is the only task-conditioning signal in the latent. The reference is fed through the same DiT blocks but with three modifications: (1) Task-aware Positional Bias adds a spatial-width or temporal-frame offset to the reference’s 3D-RoPE depending on whether the task is appearance- or temporal-focused, so the reference video is positioned either “to the right of” the target (spatial in-context for motion / camera / effect) or “in earlier frames than” the target (temporal in-context for ID / style). (2) Reference-decoupled Causal Learning replaces the obvious joint-self-attention design — which would let the reference attend to the target and would also double sequence length, quadrupling FLOPs — with one-way attention: reference Q attends only to reference K/V, while target Q attends to concat([target K/V, reference K/V]). Additionally the reference branch uses a fixed t=0 diffusion-time embedding, making it time-invariant and effectively a one-shot encoder over the denoising schedule. (3) Task-adaptive Multimodal Alignment strips Wan’s T5 text encoder and substitutes Qwen-2.5-VL (LoRA-fine-tuned) consuming the first-frame target tokens + reference-video tokens + a template + a per-task set of learnable MetaQuery tokens; its outputs go through a 3-layer MLP and are injected as cross-attention K/V into the target branch only, so the reference branch stays a pure visual conduit.

The model is trained in three sequential stages on internally collected reference-pair data (no public dataset exists): (1) train DiT only via in-context learning, 10k steps; (2) freeze DiT, train the MLLM connector, 2k steps; (3) unfreeze everything for joint fine-tuning, 5k steps. Inference adds task selection at the latent-mask flag and the per-task MetaQuery slot — no extra fine-tuning per task.

Results

ID transfer (Table 2): VSim-Arc 0.48 (vs Phantom 0.45, ConsisID 0.34, Stand-in 0.30); VSim-Glint 0.51 (vs 0.47); CLIP-T comparable at 20.35.
Style transfer (Table 3): VCSD 0.51 vs 0.29 (StyleMaster) — a 1.76× style-consistency gain; CLIP-T 27.16 vs 26.82; Aesthetics 0.61 vs 0.59.
Effect transfer (Table 4, 5-point user study): 3.45 / 3.49 / 3.27 (fidelity / image-consistency / quality) vs Seedance 1.95 / 3.20 / 2.42 and Wan2.1-I2V 1.81 / 2.89 / 2.03. Note Seedance is a far larger commercial model; OmniTransfer wins purely from the reference channel.
Camera transfer (Table 5): 4.19 / 3.89 / 3.85 vs CamCloneMaster 1.79 / 1.45 / 1.29 and MotionClone 1.75 / 1.23 / 1.29. ~2.5× on camera-fidelity user score.
Motion transfer (Table 6): 3.62 / 3.88 / 3.45 vs WanAnimate 3.71 / 3.53 / 3.48 — beats on image-consistency, ties on motion fidelity and quality, while WanAnimate runs on the larger Wan2.2-28B with explicit pose conditioning. MimicMotion baseline scores 2.67 / 1.84 / 2.02.
Inference time (Table 7): 180s baseline → 142s with RCL → 145s full model. The RCL one-shot reference encoding is the source of all the wall-clock savings; TMA is essentially free.
Ablation monotonicity (Table 7): every component additive on both appearance and temporal user scores. Largest jump on consistency from +TPB (2.36→2.82 appearance, 2.69→2.95 temporal); largest jump on quality from +TMA (3.16→3.56 appearance).

Why it’s interesting

OmniTransfer is the cleanest published instantiation of the recipe the team has been seeing repeatedly in the camera-controlled-video-diffusion concept: take a frozen large Wan2.1 backbone and bolt on a small, geometric / reference adapter that handles all conditioning. The novelty here is that the adapter unifies five heterogeneous reference modes (ID, style, motion, camera, effect) through a single switch — a per-task mask-latent flag plus a per-task RoPE offset — and demonstrates generalization to unseen task combinations. That last fact is the strongest evidence so far that task-specific reference-conditioning models (ReCamMaster, Phantom, StyleMaster, etc.) are over-specialized and can be subsumed by a single backbone with a small task-aware bias. The space/time-disentanglement framing — spatial-width offset for temporal tasks, temporal-frame offset for appearance tasks — is also a striking dual to SpaceTimePilot’s space/time-disentangled animation+camera control, suggesting “in-context bias on RoPE” is becoming the universal mechanism for injecting reference-video signal into Wan-class DiTs. Worth tracking whether the Reference-decoupled Causal Learning trick (fixed t=0 reference branch → time-invariant → free 20% inference) generalizes beyond reference-video conditioning to other static-conditioning settings.