Video-As-Prompt: Unified Semantic Control for Video Generation

TL;DR

Video-As-Prompt (VAP, ByteDance Intelligent Creation + CUHK) treats a reference video as a task-agnostic prompt to drive semantic-controlled video generation across concept, style, motion, and camera — categories that lack pixel alignment between reference and target and so break structure-controlled methods like VACE. VAP augments a frozen Video DiT (CogVideoX-I2V-5B or Wan2.1-I2V-14B) with a parallel Mixture-of-Transformers (MoT) expert, copy-initialized from the backbone; the two branches keep their own QKV/FFN/norm paths and exchange information only via layer-wise full attention. A temporally biased RoPE shifts the reference’s temporal indices before the target’s while keeping spatial indices shared, eliminating the false pixel-mapping prior. On a 38.7% user-preference rate the unified model rivals closed-source per-condition commercial systems (Kling, Vidu) and surpasses all open-source baselines.

Key claims

A single VAP model handles concept, style, motion, and camera transfer through one reference-video channel — no per-condition fine-tuning and no task-specific modules — and shows zero-shot transfer to unseen semantic references not in VAP-Data [§1, §3.2, Fig. 7].
Mixture-of-Transformers beats single-branch fine-tuning, LoRA fine-tuning, unidirectional cross-attention, and unidirectional residual addition on all five reported metrics; LoRA alone is too low-capacity for in-context semantic control [§4.5, Table 2].
Sharing RoPE between reference and target imposes a non-existent pixel-mapping prior and degrades all metrics; shifting only the temporal axis (placing the reference before the target in time) recovers the best score, while adding a width offset on top hurts spatial referencing [§3.4, Table 2 “Identical PE” and “Neg. shift in W” rows].
VAP’s MoT expert keeps the frozen backbone’s generation ability intact, avoiding the catastrophic forgetting that occurs when the backbone is fine-tuned on concatenated reference+target sequences with limited data [§3.3, Fig. 5(b)].
Structure-controlled VACE applied to semantic conditions produces copy-and-paste artifacts because it assumes a pixel-wise mapping; quality climbs monotonically as control moves from raw video (preference 0.6%) to depth (0.7%) to optical flow (1.8%), confirming the pixel-wise prior is the failure mode [Table 1, §4.4].
Condition-specific LoRAs on CogVideoX achieve strong semantic alignment (68.6 alignment score) but degrade base quality (aesthetic 54.23 vs 56.75 backbone) and require one model per condition [Table 1].
VAP scales monotonically on every metric as the paired-sample count grows from 25k to 100k to the full ~250k VAP-Data set [§4.5 scalability rows, Table 2].
Transfer to Wan2.1-I2V-14B (with expert spanning a quarter of layers to match parameter budget) raises dynamic degree to 79.17 and aesthetic to 58.09 — better than the CogVideoX version on those axes, but the more limited in-context interaction surface slightly lowers semantic alignment (70.23 vs 70.44) [§4.5, Table 2 Wan row].
Reference and target captions are both provided to the model to help locate the shared semantic; the conditional distribution is $p(V_t | I_t, V_r, C_t, C_r)$ rather than just $p(V_t | V_r)$ [§3.2].
VAP-Data contains over 100k paired reference–target videos across 100+ semantic conditions (concept / style / motion / camera), built by applying I2V visual-effects templates from Kling, Vidu, and community LoRAs to manually curated reference images [§4.3].

Method

VAP starts from a frozen pretrained Video Diffusion Transformer (CogVideoX-I2V-5B or Wan2.1-I2V-14B). It encodes the reference video $V_r$ and the noisy target $V_t$ into VAE latents and concatenates the first frame of $V_r$ as an image-conditioning latent to inherit the backbone’s I2V abilities. The architecture is a Mixture-of-Transformers: a trainable expert (full-copy of the backbone for CogVideoX; distributed across ¼ of layers for Wan2.1, kept at ~5B parameters) consumes the reference token stream, while the frozen backbone processes the target token stream. Each transformer has its own Q/K/V projections, feed-forward layers, and layer norms; at every block the two streams concatenate Q/K/V and run full attention, fusing information bidirectionally and routing reference guidance into the frozen DiT while preventing catastrophic forgetting.

The RoPE design is the second key piece. Sharing RoPE between reference and target tokens (the in-context image-generation default) would tell the model a target pixel at position $(t, h, w)$ corresponds to the reference pixel at the same coordinates — a prior that is true for depth/pose conditions but false for concept/style/motion/camera, and produces artifacts. VAP shifts the reference’s temporal indices by a fixed offset so the reference appears before the target in time, while leaving spatial indices identical. This matches the temporal in-context order, preserves spatial-semantic referenceability, and ablates strictly better than both shared-RoPE and additional-spatial-offset variants.

Captions for both reference and target are encoded by the standard text encoder to help the model find the transferable semantic (“cover liquid metal”, “Ghibli style”). Training uses flow-matching velocity prediction on AdamW (lr 1e-5, 20k steps, A100s). Inference: 50 denoising steps, CFG 6.

VAP-Data is the largest semantic-controlled paired-video dataset to date: high-quality reference images from the Internet (men, women, children, animals, objects, landscapes, multi-subject) are paired with I2V visual-effects templates from Kling, Vidu, and community LoRAs to generate same-semantic, different-content paired videos at scale (~100k pairs across 100+ conditions).

Results

User-study preference rate: VAP 38.7% (best overall), tied with closed-source commercial Kling/Vidu (38.2%); CogVideoX+LoRA per-condition 13.1%; CogVideoX-I2V (text only) 6.9%; VACE variants 0.6–1.8% [Table 1].
CLIP-Score: VAP 24.13 — highest of all methods, edging Kling/Vidu at 24.05 [Table 1].
Motion smoothness: VAP 98.59 (best); dynamic degree: 77.08 (within 2 points of best, beating CogVideoX-I2V backbone at 72.92); aesthetic quality: 57.71 (Kling/Vidu 59.16) [Table 1].
Semantic alignment score (Gemini-2.5-pro auto-evaluator): VAP 70.44 vs CogVideoX-LoRA 68.60, Kling/Vidu 74.02; VAP is the only unified model in the top tier [Table 1].
Ablation: removing MoT and fine-tuning a single branch drops alignment to 68.74 (cross-attn) or 67.16 (the unidir-cross-attn variant); residual-addition injection collapses to 55.99 [Table 2].
Wall-clock isn’t reported as a headline metric, but the Wan2.1-I2V-14B configuration matches CogVideoX-5B parameter budget by spanning ~¼ of layers, suggesting the MoT expert layer count is a tunable inference cost knob.

Why it’s interesting

VAP is the direct intellectual predecessor of OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer — both come out of ByteDance Intelligent Creation, both unify multiple reference-conditioned video tasks on a single backbone via in-context attention plus RoPE bias, and both argue against per-condition fine-tuning. The interesting contrast is in which design knob carries the in-context signal: VAP uses a parallel MoT expert with bidirectional layer-wise attention and a temporal-only RoPE offset; OmniTransfer keeps a single shared trunk, switches to one-way reference→target causal attention to halve effective sequence length, and uses task-dependent RoPE offsets (spatial-width for temporal tasks like motion/camera, temporal-frame for appearance tasks like ID/style). OmniTransfer’s task-dependent RoPE switch can be read as a generalization of VAP’s single temporal offset — VAP picks the offset that works for the four conditions it tested (all of which sit closer to “temporal” semantic-share than to “appearance” semantic-share), while OmniTransfer recognizes that ID/style transfer needs the opposite axis. The MoT-vs-shared-trunk choice is a cleaner-but-more-expensive design that doubles the parameter count (the expert is a full backbone copy on CogVideoX); OmniTransfer’s reference-decoupled causal learning is a direct alternative that preserves the single backbone and recovers 20% inference speed. This also relates to Camera-Controlled Video Diffusion more broadly: VAP demonstrates that the “frozen Wan/CogVideoX + small adapter” recipe extends to semantic (non-pixel-aligned) reference conditioning, opening the door for the unified reference-conditioning paradigm that OmniTransfer and Kling-MotionControl later refine.