Skip to content
Bud's Wiki

Camera-Controlled Video Diffusion

TL;DR

Section titled “TL;DR”

Video diffusion models conditioned on explicit camera trajectories (and increasingly on object trajectories too) to re-render or extend a scene under a user-specified viewpoint path. The dominant recent recipe in this cluster is a frozen Wan2.1 latent video diffusion backbone with a lightweight geometric adapter injecting camera (and optionally object) conditioning — ReCamMaster-style camera-embedder lineage. The three filed papers triangulate the design space along a control axis: pure camera (ReCamMaster, ViewCrafter, FlashWorld, Voyager families), space+time (SpaceTimePilot), and space+multi-object (VerseCrafter); NeoVerse threads camera control through a feed-forward 4DGS reconstructor.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-09-01] Atlas: A World Model for Spatial Intelligence: Atlas takes precise 6-DoF camera geometry as a native input modality rather than as text descriptions of cinematic moves; third-party human raters judged it to follow intended camera paths better than recent video models with the gap widening as trajectories become more complex, and it composes camera control with a spatial context of up to 100+ reference images to generate 1-minute 1440p videos along hand-designed paths.
  • [2026-08-27] Gemini Omni 1.1 Flash lets you build with more control: Ships the operating point CineTechBench argued for: first-frame + last-frame keyframe conditioning as the interface for controlled transitions, camera orbits, zooms, and looping clips on a closed frontier video model; also adds up to 3× 3s video-reference conditioning (motion/character transplant) as a sibling to OmniTransfer’s reference-video branch, though no method detail is disclosed.
  • [2026-08-21] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video: 4DAnyone diagnoses camera-controlled DiT scaling to tens of target views (as required for 4DGS reconstruction) as a bounded-attention-context problem: single-DiT passes can’t fit them, and naive grouping causes both O(N) reference-context growth and cross-group structural drift. Fix is dual: Reference Context Packing (fixed-length mixed-resolution reference context, O(1) cost) + Target Context Routing (rotate target-view groupings across denoising steps, mixing globally at high noise and stabilizing locally at low noise). First filed paper in the cluster to explicitly address the multi-forward-pass regime.
  • [2026-08-14] SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers: SCoPE (Tencent ARC + HKU) adds a second positional coordinate — the token’s 6D Plücker ray — additively into pretrained queries and keys of Wan2.2 self-attention, in a flipped Q/K basis where the geometry×geometry cross-term is exactly the Plücker reciprocal product. A Normalize-Gate-Inject (direction/magnitude decoupling + per-token log-magnitude gate + RMSNorm) stabilizes injection across SfM/SLAM/metric pose sources; RoPE is preserved bit-exactly, added parameters are <0.1%. Beats every multiplicative FreqSplit-RoPE variant (projective / camera-UV / Plücker-RoPE / 4-DoF Plücker) on Wan2.2-5B under matched budget, and beats CameraCtrl / ReCamMaster / UCPE / ReRoPE on both 5B and 14B (Wan2.2-14B: RotErr 0.058 vs 0.082 UCPE, FVD 280 vs 493 ReRoPE). Claims “revisit consistency by construction” — a scene looped back returns to the same rooms because outbound/return views share ray coordinates — without any memory module. First filed additive-Plücker-into-attention operating point in the cluster, directly opposing PRoPE-family multiplicative RoPE replacement.
  • [2026-08-14] DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation: DreamX-Phi ports the PRoPE relative-SE(3)-into-attention primitive from cameras (Cameras as Relative Positional Encoding) to per-arm SE(3) transformations in a bimanual robotic-manipulation video WM — a concrete transfer of the relative-geometric-attention recipe from observer-side to actor-side.
  • [2026-08-07] Wan-Animate-2: End-to-End Character Animation with Text-Driven Viewpoint Control and Real-Time Streaming: Wan-Animate-2 (HumanAIGC / Tongyi Lab) introduces text-driven viewpoint control that decouples the output camera perspective from the driving video’s camera — a new axis in the cluster where camera control is composed with character animation on a single DiT, but driven by text rather than an explicit trajectory adapter (VerseCrafter/SpaceTimePilot) or reference-video conditioning (OmniTransfer). Direct successor to Wan-Animate v1 and closest to Kling-MotionControl’s text-driven cinematic camera composition on character motion, but achieved without exposing separate motion channels — the driving video is consumed end-to-end.
  • [2026-08-06] CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation: CineTechBench provides a first-frame / first-last-frame conditioned camera-movement reconstruction task on 120+ Movieclips segments, scored via MonST3R-estimated rotation/translation error against the reference film clip. Key result: I2V models trail badly on high-angular-velocity rotation (fight-scene shots) and can generate rotation with the correct magnitude but reversed direction (Kling v1.6 example); first+last-frame conditioning helps translation error substantially over first-frame-only.
  • [2026-07-29] FilmBench: A Film-Grade Benchmark for Cinematic Video Generation: FilmBench provides a film-craft-graded evaluation harness that plausibly probes camera-motion adherence as one of its sub-metrics — prompts follow real shot lists from award-winning films, and the taxonomy is co-designed with directors, so cinematography-craft components should surface camera-control failures more sharply than web-style benchmarks.
  • [2026-07-21] AlayaWorld: Interactive Long-Horizon World Modeling: AlayaWorld injects camera control via AdaLN modulation of the timestep embedding from Fourier-embedded per-frame relative-pose increments concatenated over 6 pose components (per-axis scales calibrated from real data motion statistics), tightly coupled to a GEN3C-style spatial cache that shares one pose source with the AdaLN condition — a design that ties camera control and 3D memory to the same coordinate stream rather than treating them as independent conditions.
  • [2026-07-14] Motion4Motion: Motion Transfer Across Subjects at Inference: Motion4Motion (SIGGRAPH ‘26) is a training-free, skeleton-free motion-transfer recipe on frozen WAN-T2V: extract per-anchor motion flow from a source (Grounded SAM-2 + diffusion-feature matching + point tracking), then inject it into self-attention by caching target K/V and re-embedding them with new RoPE positions from the flow (“TransPE”). Third filed instance of inference-time RoPE manipulation as the control primitive on WAN (after ReTime and OmniTransfer), and the first to cover the cross-topology regime (human → goose → panda → animate table) that skeleton-based methods like Wan-Animate and Kling-MotionControl cannot bridge.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion’s predicted 3D point trajectories are used as the motion-control signal to DaS on top of CogVideoX-5B; the combination is reported to beat the much larger Wan2.2-I2V-A14B on four of five VBench motion metrics — a concrete recipe for trajectory-conditioning I2V via a separately pretrained language-conditioned motion forecaster rather than baking trajectory control into the diffusion model.
  • [2026-06-09] Latent Spatial Memory for Video World Models: Mirage realizes camera control implicitly: novel-view synthesis is direct latent warping of cached tokens into the target view via depth-guided projection, with no explicit camera-trajectory conditioning channel. Reports state-of-the-art WorldScore (70.36 avg) and strong RealEstate10K reconstruction. Sibling recipe to point-cloud-rendering-as-ControlNet (SPMem, NeoVerse), but indexing latents instead of RGB.
  • [2026-06-02] EgoX: Egocentric Video Generation from a Single Exocentric Video: EgoX (KAIST AI / DAVIAN Robotics) pushes the “frozen video DiT + LoRA + decoupled conditioning” recipe to the extreme no-overlap regime — exo-to-ego viewpoint translation. Uses 4DNeX as the frozen backbone, LoRA adaptation, a unified conditioning scheme (width-wise concat of the exocentric source video + channel-wise concat of an estimated egocentric prior video), and geometry-guided self-attention that restricts attention to spatially relevant regions. Trained on EgoExo4D, reports SOTA on exo-to-ego benchmarks and generalizes to in-the-wild clips (Dark Knight). First filed paper in the cluster that explicitly targets the zero-pixel-overlap camera transformation case; the egocentric prior plays the structural role the depth / Gaussian-trajectory channels play in VerseCrafter and SpaceTimePilot.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything (Wan2.2-TI2V-5B base) introduces Subject and Camera Motion Decoupled CFG — subject-pose conditioning to the positive CFG anchor, camera-motion conditioning to the negative anchor — yielding camera-trajectory control at inference time from a model trained only on subject-pose data. First filed instance of using CFG anchor structure (not a trained control channel) as the camera-control primitive. Also ships XPose, a 50K-clip non-human pose-video dataset for universal-skeleton pose-driven generation.
  • [2026-05-24] Video Analysis and Generation via a Semantic Progress Function: Introduces a measure-then-warp paradigm for temporal control on Wan2.2 and LTX-2 — fits a 1D Semantic Progress Function to SigLIP pairwise embedding distances, then inverts it to warp temporal RoPE positions toward uniform semantic velocity. Frequency-aware blending of warping strength across RoPE bands, exponential timestep-decay multiplier, three iterations. Within 1σ of original on VBench Aesthetic / Motion / Temporal Flicker, 88% user-study preference on pacing. For closed-source first-last-frame models, segments the SPF by piecewise-linear least squares and regenerates clips per segment. LTX-2 audio automatically realigns because the model anchors cross-modal attention to linear temporal coordinates. SIGGRAPH ‘26.
  • [2026-05-23] Kling-MotionControl Technical Report: Kuaishou’s Kling-MotionControl — unified DiT for holistic character animation with three-channel divide-and-conquer (body/face/hand) motion + an explicit semantic-motion track, identity-agnostic geometric motion learning, a subject-library multi-reference mechanism, 3D-aware motion representations enabling text-driven cinematic camera control composed with character motion, and a multi-stage CFG-merged distillation stack yielding >10× inference speedup. GSB human evaluation reports preference over Wan-Animate (Overall 4.00), Dreamina (3.44), and Runway Act-Two (16.25) on a 150-case 1080p benchmark across all five granular dimensions.
  • [2026-05-23] Wan-Animate: Unified Character Animation and Replacement with Holistic Replication: Wan-Animate post-trains Wan-I2V (productized as Wan 2.2 Animate) for unified character animation + character replacement. Uses a region-flagged input paradigm that preserves Wan-I2V’s latent-input structure, decouples body-motion (spatially-aligned 2D skeleton added to noise latents) from expression (implicit features), and adds a Relighting LoRA only for replacement mode. Reports SOTA against Animate Anyone / UniAnimate / VACE on SSIM/LPIPS/FVD and user-study preference over DreamActor-M1 and Runway Act-Two; productized May 2026 on fal as “Wan Motion” with built-in motion retargeting and optional Flux-2 Edit identity sharpening.
  • [2026-05-22] OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer: Unifies camera transfer with ID/style/motion/effect transfer in a single Wan2.1-I2V-14B model. Replaces explicit camera-trajectory parameterization with a clean reference-video branch and a per-task RoPE offset (spatial-width for temporal tasks, temporal-frame for appearance tasks). Adds Reference-decoupled Causal Learning (one-way reference→target attention + fixed t=0 on reference branch → 20% faster inference) and a Qwen-2.5-VL-with-MetaQueries connector replacing T5. Beats MotionClone and CamCloneMaster ~2.5× on user-study camera fidelity. Argues this is the first cross-task reference-conditioning recipe that generalizes to unseen task combinations.
  • [2026-05-22] LongVie 2: Multimodal Controllable Ultra-Long Video World Model: Stretches the cluster from clip-scale (5–10s) camera/object control to multi-minute AR generation. Treats control as one of three sequenced training stages (multimodal guidance → degradation-aware training → history-context guidance) rather than a single conditioning channel. Project-page-only; specifics of which control signals (camera trajectories? sparse poses? depth maps?) not yet disclosed.
  • [2026-05-22] VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control: Introduces 3D Gaussian trajectories as the object-motion control space, with a decoupled background + foreground GeoAdapter on frozen Wan2.1-14B; ablates against 3D bounding boxes, 3D point trajectories, depth removal, and merged-channel variants; builds VerseControl4D (35k clips) with fully automatic 4D annotation.
  • [2026-05-22] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: Threads camera control through a feed-forward 4DGS reconstructor (“Gaussianized” VGGT) + frozen Wan-T2V-14B; introduces a quality/compute knob (11→full keyframes) and achieves 6–8× faster inference than TrajectoryCrafter and ReCamMaster at parity or better VBench.
  • [2026-05-22] SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time: Disentangles space (camera) and time (animation) control on a frozen Wan-2.1 T2V-1.3B with an animation-time embedder and source-aware camera conditioning; supplies the synthetic CamTime dataset (180k Blender videos, full camera×time grid).

Open questions

Section titled “Open questions”
  • Does a single frozen Wan + multiple specialized adapters (camera, time, object, character) compose? The filed papers all train one adapter at a time on top of Wan; OmniTransfer composes 5 task switches inside a single adapter but does not stack independent adapters — nobody has demonstrated that VerseCrafter’s GeoAdapter, SpaceTimePilot’s E_ani, and Wan-Animate’s character/skeleton head can be combined. Kling-MotionControl is the first instance of camera + character control inside one model (via 3D-aware motion representations), but it is a from-scratch DiT and does not test composability with the Wan-adapter recipes. ReTime is composable in principle — it warps RoPE at inference time and doesn’t touch weights — but composability with adapter-trained warping has not been tested.
  • Where does 3D Gaussian trajectory representation break? VerseCrafter notes ellipsoid ambiguity for near-isotropic shapes (humans) — under what data distribution does this hurt enough to justify hybrid Gaussian + SMPL-X? Wan-Animate’s skeleton-based body-motion channel and Kling-MotionControl’s three-channel body/face/hand decomposition are two concrete alternatives for the human-character case.
  • Reconstruction-based (NeoVerse) vs control-first (VerseCrafter, SpaceTimePilot, Wan-Animate, Kling-MotionControl) vs reference-video-conditioning (OmniTransfer) vs measure-then-warp (ReTime): now four-plus quite different paradigms. Is there a regime where one dominates? OmniTransfer beats CamCloneMaster on user-study camera fidelity but has not been head-to-head with ReCamMaster / NeoVerse / Kling-MotionControl on VBench-style trajectory metrics.
  • The control-axis explosion (camera, time, multi-object, character, lighting, semantic pacing?) implies a combinatorial trajectory-annotation cost. OmniTransfer sidesteps part of this by using reference videos themselves as the control signal; Kling-MotionControl sidesteps it for character-motion via identity-agnostic geometric motion learning + semantic motion tagging; ReTime sidesteps it entirely by deriving the control signal from the model’s own output via SPF.
  • All five Wan-based papers use Wan2.x as the frozen prior (Wan-Animate uses 2.2; the others 2.1; ReTime uses 2.2). How much of the reported quality comes from Wan itself vs. the geometric-adapter / reference-branch / region-flag design choices? Kling-MotionControl is the contrast point — it does not start from Wan and still reports preference over Wan-Animate.
  • Does camera/object control survive the autoregressive multi-minute regime? LongVie 2 claims controllability at the 5-minute horizon, but no head-to-head with clip-scale ReCamMaster / VerseCrafter / OmniTransfer / Wan-Animate / Kling-MotionControl quality has been reported.
  • Is OmniTransfer’s “fixed t=0 on reference branch → time-invariant encoder → free 20% inference” trick transferable to the explicit-trajectory recipes? They all run the geometric adapter at every diffusion step today. Kling-MotionControl’s CFG-merging + multi-stage distillation gets to >10× speedup via a different route (student NFE reduction); the two acceleration tricks are likely composable. ReTime’s timestep-decay multiplier β(t) is a third complementary acceleration heuristic — concentrates work at early diffusion steps.
  • Wan-Animate gets away with additive injection of a spatially-aligned skeleton map onto noise latents — no adapter on that path. Could the explicit-camera-trajectory recipes (which use spatially-aligned RGB+depth control maps) drop their adapter blocks in favor of an additive injection too?
  • Three-channel (Kling-MotionControl, body/face/hand) vs two-channel (Wan-Animate, body/face) character-motion decomposition: is the third hand channel load-bearing, or is the win attributable to the explicit semantic-motion track, the multi-stage training, or the 3D-aware representation? No ablation table separates these in Kling-MotionControl’s report.
  • ReTime warps temporal RoPE uniformly across the whole clip toward linear semantic velocity. Could a per-region warp (e.g., enforced only on a subject mask) compose with the global trajectory adapters above for region-specific pacing control?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2023-12

Section titled “2023-12”