CamCloneMaster: Enabling Reference-based Camera Control for Video Generation

TL;DR

CamCloneMaster lets a user specify camera motion by showing the model a reference video instead of feeding camera parameters. The architecture is deliberately minimal — condition tokens (a camera-motion reference, optionally also a content reference) are concatenated with noisy video tokens in a single sequence, and only the 3D spatial-temporal attention layers of a DiT backbone are trainable. The same model unifies camera-controlled I2V and V2V re-shooting; the latter functions as a post-capture cinematographic editing tool. Supervision comes from a synthetic Camera Clone Dataset rendered in Unreal Engine 5 with paired camera trajectories over identical scene+animation states. SIGGRAPH Asia 2025.

Key claims

Camera motion can be cloned from a reference video without any explicit camera-parameter input and without test-time fine-tuning, sidestepping the user-side cost of constructing trajectories [§1, §3].
A single unified model handles both camera-controlled I2V (camera reference + first frame) and V2V re-shooting (camera reference + content reference); the design difference between the two modes is which conditioning slots are populated [§3, Fig. 1].
Condition tokens are injected by direct sequence concatenation with the noise latent along the frame dimension, with no separate cross-attention module or ControlNet-style branch — only the 3D spatial-temporal attention layers in the DiT blocks are trainable [§3, project-page method description].
A user study comparing baselines fed ground-truth vs MegaSaM-estimated camera parameters shows large preference for ground-truth, framing the reference-video formulation as a remedy for the brittleness of pose estimation in the parameter-based pipeline [§4 user study, Themoonlight review].
On RealEstate10K and a curated set of classic movie clips with complex camera trajectories, CamCloneMaster reports lower RotErr / TransErr / CamMC and better visual+dynamic quality than CameraCtrl, CamI2V, and MotionClone-style baselines [Table 1, Fig. 4].
The Camera Clone Dataset is generated by placing animated foreground characters into 3D background scenes in UE5 and rendering each scene+animation under multiple paired camera trajectories — triples of (camera reference V_cam, content reference V_cont, target V) that real captures cannot reproduce [§4].
The internal backbone is a Diffusion Transformer trained with the Rectified Flow objective; the released code/weights are not the paper model (which is a proprietary internal T2V), with reference Wan2.1-based reimplementation noted in the GitHub README [GitHub release notes].

Method

A 3D VAE encoder converts the camera-motion reference video and (optionally) the content reference video into conditional latents z_cam and z_cont. These are concatenated with the noisy video latent along the frame dimension to form a single input sequence consumed by a DiT backbone trained with Rectified Flow. Only 3D spatial-temporal attention layers are trainable; the rest of the backbone is frozen. At inference, populating only z_cam and the first-frame latent gives I2V camera control; populating both z_cam and z_cont gives V2V re-shooting (the content reference is re-rendered under the reference camera motion).

Results

Quantitative camera-accuracy lead over CameraCtrl / CamI2V on RotErr, TransErr, and CamMC on RealEstate10K test set and movie-clip test set; concurrent gains on visual and dynamic quality metrics [Table 1].
Qualitative cases: leftward trucking + rotation in a sailboat scene where CameraCtrl/CamI2V fail to follow the trajectory; complex monkey motion preserved without visible structure decay [Fig. 4].
User study: comparing baselines under ground-truth vs MegaSaM-estimated camera parameters, baseline preference collapses when forced to use estimated parameters — motivating the reference-based formulation as a more deployable interface [§4 user study].
Head-to-head with reference-conditioning competitors: OmniTransfer (OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer Table 5) reports beating CamCloneMaster ~2.5× on a user-study camera-fidelity score (4.19 vs 1.79).

Why it’s interesting

The wiki’s Camera-Controlled Video Diffusion cluster has tracked a steady shift from explicit-trajectory adapters (ReCamMaster, VerseCrafter, SpaceTimePilot) toward reference-video conditioning (OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer OmniTransfer, this paper). CamCloneMaster is the cleanest minimal expression of that idea: no separate control module, no Plücker / epipolar embedding, just sequence concatenation with the only trainable parameters being the existing attention layers. It also stakes a position the wiki hasn’t catalogued yet — a single unified I2V + V2V model — which generalizes the “re-shoot a captured clip from a different angle” workflow into the same DiT call as “animate a still image along a borrowed camera path.” OmniTransfer’s later result that it beats CamCloneMaster by ~2.5× on user-study camera fidelity is a useful calibration: the concat-and-attend recipe works, but a one-way (causal) reference→target attention plus an axis-disentangling RoPE offset is materially better than symmetric joint attention on the same task.