Skip to content
Bud's Wiki

Camera-Controlled Video Diffusion

TL;DR

Section titled “TL;DR”

Video diffusion models conditioned on explicit camera trajectories (and increasingly on object trajectories too) to re-render or extend a scene under a user-specified viewpoint path. The dominant recent recipe in this cluster is a frozen Wan2.1 latent video diffusion backbone with a lightweight geometric adapter injecting camera (and optionally object) conditioning — ReCamMaster-style camera-embedder lineage. The three filed papers triangulate the design space along a control axis: pure camera (ReCamMaster, ViewCrafter, FlashWorld, Voyager families), space+time (SpaceTimePilot), and space+multi-object (VerseCrafter); NeoVerse threads camera control through a feed-forward 4DGS reconstructor.

Key claims

Section titled “Key claims”

Recent contributions

Section titled “Recent contributions”
  • [2026-10-22] Interactive World Simulation with Cosmos for Physical AI: Cosmos3-Nano-Sim-Transfer supports structured spatial controls (depth, segmentation, edges, blurred videos) as the per-chunk clean control group U_j encoded via the same video tokenizer as the source video, with source tokens entering the K/V cache keeping their conditioning role for future predictions. Spatial controls are matched to target frames and FPS-modulated to the target’s rotary temporal coordinates.
  • [2026-10-08] Meet Odyssey-3: Our Most Powerful Foundation World Model: Odyssey-3’s research-preview surface exposes three camera-navigation primitives (first-person navigation, third-person navigation, and independent camera movement during generation) as first-class interaction modes, with the three-camera multi-sensor driving extension demonstrating that a single Odyssey-3 checkpoint can be adapted after 100 training steps to generate jointly-synchronized sequences for a fixed sensor arrangement — a camera-specification channel distinct from the camera-trajectory-conditioning line.
  • [2026-09-22] WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory: WorldCrafter realises camera control implicitly, like Mirage: the target camera pose queries the memory encoder’s readout module to produce view-specific tokens, with no explicit trajectory-adapter (VerseCrafter/NeoVerse/SpaceTimePilot style) or Plücker-ray injection (SCoPE). Sibling to Latent Spatial Memory for Video World Models but the memory itself is implicit (learned encoder) rather than an explicit 3D-indexed latent cache.
  • [2026-09-11] Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning: Know Your Camera (Jiang et al., ICRA’26 best paper) is the crossover datapoint that shows the same Plücker raymap primitive dominant in this cluster’s frozen-Wan camera-control adapters (VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control, NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time) also works as policy input when concatenated with RGB before the visual encoder — improving viewpoint generalization for ACT, Diffusion Policy, and SmolVLA on paired fixed/randomized-scene manipulation benchmarks. Direct empirical counterpoint to Cameras as Relative Positional Encoding‘s “stop settling for Plücker raymaps” position: KYC argues Plücker raymaps remain the right tool when architectural changes to attention are off-limits and drop-in compatibility with the pretrained visual encoder matters more than the intrinsics-invariance gains PRoPE’s relative-projective encoding provides.
  • [2026-09-08] ID-V2V: Identity-Preserving Video Restylization: ID-V2V extends VACE/Wan2.1 to multi-ControlNet with four control signals (relit-face pixels + face-normals + edited keyframe + depth) for identity-preserving video restylization — keep source performance, swap look via an edited first frame.
  • [2026-09-04] CAT3D: Create Anything in 3D with Multi-View Diffusion Models: Foundational reference on file (NeurIPS 2024 Oral) — a video-diffusion-style architecture where the sequence axis indexes over cameras rather than frames, with camera poses embedded as ray coordinates rather than time embeddings. The reference point that later “frozen video DiT + camera adapter” (VerseCrafter/NeoVerse/SpaceTimePilot/OmniTransfer) and “generate views + reconstruct” (FlashWorld) recipes benchmark against.
  • [2026-09-03] SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models: SolarWM’s unified frame-aligned contract binds metric camera geometry to every clip across 10 heterogeneous data sources, and a shared camera-conditioning interface is applied identically across four different video backbones (Wan2.2, LTX-2.5, MiniMax-H3) — one of the first filed instances of camera-conditioning treated as a backbone-agnostic interface rather than a per-model injection recipe.
  • [2026-09-02] H3-World: Turning Language Understanding into World Control: H3-World adds a natural-language route to camera control: rather than injecting camera geometry through a dedicated adapter, it fine-tunes MiniMax-H3 (0.199% LoRA params) so that short character+camera language instructions align with temporal video-latent windows via attention routing. Counterpoint to the frozen-Wan-plus-camera-adapter recipes — camera control emerges from the pretrained language channel of a large video generator, not from explicit geometry conditioning.
  • [2026-09-01] Atlas: A World Model for Spatial Intelligence: Atlas (World Labs) departs from the frozen-Wan-plus-adapter recipe dominant in the cluster (VerseCrafter / NeoVerse / SpaceTimePilot / OmniTransfer): a from-scratch multimodal AR diffusion transformer where each input image and depth map is bound to an explicit 6-DoF camera pose inside a shared spatial context — camera geometry is a native input type in the AR sequence, not a text approximation or an adapter-injected condition. Reports beating recent video models on third-party human-rater cinematic-camera-motion adherence with the gap widening on more complex trajectories, and generates up to 1-minute 1440p video from hand-designed camera paths over a small number of reference images. First filed instance of “camera pose as a first-class AR sequence element” rather than an adapter or RoPE-offset primitive — a sharp architectural alternative to the entire adapter-based cluster.
  • [2026-08-27] Gemini Omni 1.1 Flash lets you build with more control: Ships the operating point CineTechBench argued for: first-frame + last-frame keyframe conditioning as the interface for controlled transitions, camera orbits, zooms, and looping clips on a closed frontier video model; also adds up to 3× 3s video-reference conditioning (motion/character transplant) as a sibling to OmniTransfer’s reference-video branch, though no method detail is disclosed.
  • [2026-08-21] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video: 4DAnyone diagnoses camera-controlled DiT scaling to tens of target views (as required for 4DGS reconstruction) as a bounded-attention-context problem: single-DiT passes can’t fit them, and naive grouping causes both O(N) reference-context growth and cross-group structural drift. Fix is dual: Reference Context Packing (fixed-length mixed-resolution reference context, O(1) cost) + Target Context Routing (rotate target-view groupings across denoising steps, mixing globally at high noise and stabilizing locally at low noise). First filed paper in the cluster to explicitly address the multi-forward-pass regime.
  • [2026-08-14] SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers: SCoPE (Tencent ARC + HKU) adds a second positional coordinate — the token’s 6D Plücker ray — additively into pretrained queries and keys of Wan2.2 self-attention, in a flipped Q/K basis where the geometry×geometry cross-term is exactly the Plücker reciprocal product. A Normalize-Gate-Inject (direction/magnitude decoupling + per-token log-magnitude gate + RMSNorm) stabilizes injection across SfM/SLAM/metric pose sources; RoPE is preserved bit-exactly, added parameters are <0.1%. Beats every multiplicative FreqSplit-RoPE variant (projective / camera-UV / Plücker-RoPE / 4-DoF Plücker) on Wan2.2-5B under matched budget, and beats CameraCtrl / ReCamMaster / UCPE / ReRoPE on both 5B and 14B (Wan2.2-14B: RotErr 0.058 vs 0.082 UCPE, FVD 280 vs 493 ReRoPE). Claims “revisit consistency by construction” — a scene looped back returns to the same rooms because outbound/return views share ray coordinates — without any memory module. First filed additive-Plücker-into-attention operating point in the cluster, directly opposing PRoPE-family multiplicative RoPE replacement.
  • [2026-08-14] DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation: DreamX-Phi ports the PRoPE relative-SE(3)-into-attention primitive from cameras (Cameras as Relative Positional Encoding) to per-arm SE(3) transformations in a bimanual robotic-manipulation video WM — a concrete transfer of the relative-geometric-attention recipe from observer-side to actor-side.
  • [2026-08-07] Wan-Animate-2: End-to-End Character Animation with Text-Driven Viewpoint Control and Real-Time Streaming: Wan-Animate-2 (HumanAIGC / Tongyi Lab) introduces text-driven viewpoint control that decouples the output camera perspective from the driving video’s camera — a new axis in the cluster where camera control is composed with character animation on a single DiT, but driven by text rather than an explicit trajectory adapter (VerseCrafter/SpaceTimePilot) or reference-video conditioning (OmniTransfer). Direct successor to Wan-Animate v1 and closest to Kling-MotionControl’s text-driven cinematic camera composition on character motion, but achieved without exposing separate motion channels — the driving video is consumed end-to-end.
  • [2026-08-06] CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation: CineTechBench provides a first-frame / first-last-frame conditioned camera-movement reconstruction task on 120+ Movieclips segments, scored via MonST3R-estimated rotation/translation error against the reference film clip. Key result: I2V models trail badly on high-angular-velocity rotation (fight-scene shots) and can generate rotation with the correct magnitude but reversed direction (Kling v1.6 example); first+last-frame conditioning helps translation error substantially over first-frame-only.
  • [2026-07-29] FilmBench: A Film-Grade Benchmark for Cinematic Video Generation: FilmBench provides a film-craft-graded evaluation harness that plausibly probes camera-motion adherence as one of its sub-metrics — prompts follow real shot lists from award-winning films, and the taxonomy is co-designed with directors, so cinematography-craft components should surface camera-control failures more sharply than web-style benchmarks.
  • [2026-07-21] AlayaWorld: Interactive Long-Horizon World Modeling: AlayaWorld injects camera control via AdaLN modulation of the timestep embedding from Fourier-embedded per-frame relative-pose increments concatenated over 6 pose components (per-axis scales calibrated from real data motion statistics), tightly coupled to a GEN3C-style spatial cache that shares one pose source with the AdaLN condition — a design that ties camera control and 3D memory to the same coordinate stream rather than treating them as independent conditions.
  • [2026-07-14] Motion4Motion: Motion Transfer Across Subjects at Inference: Motion4Motion (SIGGRAPH ‘26) is a training-free, skeleton-free motion-transfer recipe on frozen WAN-T2V: extract per-anchor motion flow from a source (Grounded SAM-2 + diffusion-feature matching + point tracking), then inject it into self-attention by caching target K/V and re-embedding them with new RoPE positions from the flow (“TransPE”). Third filed instance of inference-time RoPE manipulation as the control primitive on WAN (after ReTime and OmniTransfer), and the first to cover the cross-topology regime (human → goose → panda → animate table) that skeleton-based methods like Wan-Animate and Kling-MotionControl cannot bridge.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion’s predicted 3D point trajectories are used as the motion-control signal to DaS on top of CogVideoX-5B; the combination is reported to beat the much larger Wan2.2-I2V-A14B on four of five VBench motion metrics — a concrete recipe for trajectory-conditioning I2V via a separately pretrained language-conditioned motion forecaster rather than baking trajectory control into the diffusion model.
  • [2026-06-09] Latent Spatial Memory for Video World Models: Mirage realizes camera control implicitly: novel-view synthesis is direct latent warping of cached tokens into the target view via depth-guided projection, with no explicit camera-trajectory conditioning channel. Reports state-of-the-art WorldScore (70.36 avg) and strong RealEstate10K reconstruction. Sibling recipe to point-cloud-rendering-as-ControlNet (SPMem, NeoVerse), but indexing latents instead of RGB.
  • [2026-06-02] EgoX: Egocentric Video Generation from a Single Exocentric Video: EgoX (KAIST AI / DAVIAN Robotics) pushes the “frozen video DiT + LoRA + decoupled conditioning” recipe to the extreme no-overlap regime — exo-to-ego viewpoint translation. Uses 4DNeX as the frozen backbone, LoRA adaptation, a unified conditioning scheme (width-wise concat of the exocentric source video + channel-wise concat of an estimated egocentric prior video), and geometry-guided self-attention that restricts attention to spatially relevant regions. Trained on EgoExo4D, reports SOTA on exo-to-ego benchmarks and generalizes to in-the-wild clips (Dark Knight). First filed paper in the cluster that explicitly targets the zero-pixel-overlap camera transformation case; the egocentric prior plays the structural role the depth / Gaussian-trajectory channels play in VerseCrafter and SpaceTimePilot.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: PoseAnything (Wan2.2-TI2V-5B base) introduces Subject and Camera Motion Decoupled CFG — subject-pose conditioning to the positive CFG anchor, camera-motion conditioning to the negative anchor — yielding camera-trajectory control at inference time from a model trained only on subject-pose data. First filed instance of using CFG anchor structure (not a trained control channel) as the camera-control primitive. Also ships XPose, a 50K-clip non-human pose-video dataset for universal-skeleton pose-driven generation.
  • [2026-05-24] Video Analysis and Generation via a Semantic Progress Function: Introduces a measure-then-warp paradigm for temporal control on Wan2.2 and LTX-2 — fits a 1D Semantic Progress Function to SigLIP pairwise embedding distances, then inverts it to warp temporal RoPE positions toward uniform semantic velocity. Frequency-aware blending of warping strength across RoPE bands, exponential timestep-decay multiplier, three iterations. Within 1σ of original on VBench Aesthetic / Motion / Temporal Flicker, 88% user-study preference on pacing. For closed-source first-last-frame models, segments the SPF by piecewise-linear least squares and regenerates clips per segment. LTX-2 audio automatically realigns because the model anchors cross-modal attention to linear temporal coordinates. SIGGRAPH ‘26.
  • [2026-05-23] Kling-MotionControl Technical Report: Kuaishou’s Kling-MotionControl — unified DiT for holistic character animation with three-channel divide-and-conquer (body/face/hand) motion + an explicit semantic-motion track, identity-agnostic geometric motion learning, a subject-library multi-reference mechanism, 3D-aware motion representations enabling text-driven cinematic camera control composed with character motion, and a multi-stage CFG-merged distillation stack yielding >10× inference speedup. GSB human evaluation reports preference over Wan-Animate (Overall 4.00), Dreamina (3.44), and Runway Act-Two (16.25) on a 150-case 1080p benchmark across all five granular dimensions.
  • [2026-05-23] Wan-Animate: Unified Character Animation and Replacement with Holistic Replication: Wan-Animate post-trains Wan-I2V (productized as Wan 2.2 Animate) for unified character animation + character replacement. Uses a region-flagged input paradigm that preserves Wan-I2V’s latent-input structure, decouples body-motion (spatially-aligned 2D skeleton added to noise latents) from expression (implicit features), and adds a Relighting LoRA only for replacement mode. Reports SOTA against Animate Anyone / UniAnimate / VACE on SSIM/LPIPS/FVD and user-study preference over DreamActor-M1 and Runway Act-Two; productized May 2026 on fal as “Wan Motion” with built-in motion retargeting and optional Flux-2 Edit identity sharpening.
  • [2026-05-22] OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer: Unifies camera transfer with ID/style/motion/effect transfer in a single Wan2.1-I2V-14B model. Replaces explicit camera-trajectory parameterization with a clean reference-video branch and a per-task RoPE offset (spatial-width for temporal tasks, temporal-frame for appearance tasks). Adds Reference-decoupled Causal Learning (one-way reference→target attention + fixed t=0 on reference branch → 20% faster inference) and a Qwen-2.5-VL-with-MetaQueries connector replacing T5. Beats MotionClone and CamCloneMaster ~2.5× on user-study camera fidelity. Argues this is the first cross-task reference-conditioning recipe that generalizes to unseen task combinations.
  • [2026-05-22] LongVie 2: Multimodal Controllable Ultra-Long Video World Model: Stretches the cluster from clip-scale (5–10s) camera/object control to multi-minute AR generation. Treats control as one of three sequenced training stages (multimodal guidance → degradation-aware training → history-context guidance) rather than a single conditioning channel. Project-page-only; specifics of which control signals (camera trajectories? sparse poses? depth maps?) not yet disclosed.
  • [2026-05-22] VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control: Introduces 3D Gaussian trajectories as the object-motion control space, with a decoupled background + foreground GeoAdapter on frozen Wan2.1-14B; ablates against 3D bounding boxes, 3D point trajectories, depth removal, and merged-channel variants; builds VerseControl4D (35k clips) with fully automatic 4D annotation.
  • [2026-05-22] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: Threads camera control through a feed-forward 4DGS reconstructor (“Gaussianized” VGGT) + frozen Wan-T2V-14B; introduces a quality/compute knob (11→full keyframes) and achieves 6–8× faster inference than TrajectoryCrafter and ReCamMaster at parity or better VBench.
  • [2026-05-22] SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time: Disentangles space (camera) and time (animation) control on a frozen Wan-2.1 T2V-1.3B with an animation-time embedder and source-aware camera conditioning; supplies the synthetic CamTime dataset (180k Blender videos, full camera×time grid).

Open questions

Section titled “Open questions”
  • Does a single frozen Wan + multiple specialized adapters (camera, time, object, character) compose? The filed papers all train one adapter at a time on top of Wan; OmniTransfer composes 5 task switches inside a single adapter but does not stack independent adapters — nobody has demonstrated that VerseCrafter’s GeoAdapter, SpaceTimePilot’s E_ani, and Wan-Animate’s character/skeleton head can be combined. Kling-MotionControl is the first instance of camera + character control inside one model (via 3D-aware motion representations), but it is a from-scratch DiT and does not test composability with the Wan-adapter recipes. ReTime is composable in principle — it warps RoPE at inference time and doesn’t touch weights — but composability with adapter-trained warping has not been tested.
  • Where does 3D Gaussian trajectory representation break? VerseCrafter notes ellipsoid ambiguity for near-isotropic shapes (humans) — under what data distribution does this hurt enough to justify hybrid Gaussian + SMPL-X? Wan-Animate’s skeleton-based body-motion channel and Kling-MotionControl’s three-channel body/face/hand decomposition are two concrete alternatives for the human-character case.
  • Reconstruction-based (NeoVerse) vs control-first (VerseCrafter, SpaceTimePilot, Wan-Animate, Kling-MotionControl) vs reference-video-conditioning (OmniTransfer) vs measure-then-warp (ReTime): now four-plus quite different paradigms. Is there a regime where one dominates? OmniTransfer beats CamCloneMaster on user-study camera fidelity but has not been head-to-head with ReCamMaster / NeoVerse / Kling-MotionControl on VBench-style trajectory metrics.
  • The control-axis explosion (camera, time, multi-object, character, lighting, semantic pacing?) implies a combinatorial trajectory-annotation cost. OmniTransfer sidesteps part of this by using reference videos themselves as the control signal; Kling-MotionControl sidesteps it for character-motion via identity-agnostic geometric motion learning + semantic motion tagging; ReTime sidesteps it entirely by deriving the control signal from the model’s own output via SPF.
  • All five Wan-based papers use Wan2.x as the frozen prior (Wan-Animate uses 2.2; the others 2.1; ReTime uses 2.2). How much of the reported quality comes from Wan itself vs. the geometric-adapter / reference-branch / region-flag design choices? Kling-MotionControl is the contrast point — it does not start from Wan and still reports preference over Wan-Animate.
  • Does camera/object control survive the autoregressive multi-minute regime? LongVie 2 claims controllability at the 5-minute horizon, but no head-to-head with clip-scale ReCamMaster / VerseCrafter / OmniTransfer / Wan-Animate / Kling-MotionControl quality has been reported.
  • Is OmniTransfer’s “fixed t=0 on reference branch → time-invariant encoder → free 20% inference” trick transferable to the explicit-trajectory recipes? They all run the geometric adapter at every diffusion step today. Kling-MotionControl’s CFG-merging + multi-stage distillation gets to >10× speedup via a different route (student NFE reduction); the two acceleration tricks are likely composable. ReTime’s timestep-decay multiplier β(t) is a third complementary acceleration heuristic — concentrates work at early diffusion steps.
  • Wan-Animate gets away with additive injection of a spatially-aligned skeleton map onto noise latents — no adapter on that path. Could the explicit-camera-trajectory recipes (which use spatially-aligned RGB+depth control maps) drop their adapter blocks in favor of an additive injection too?
  • Three-channel (Kling-MotionControl, body/face/hand) vs two-channel (Wan-Animate, body/face) character-motion decomposition: is the third hand channel load-bearing, or is the win attributable to the explicit semantic-motion track, the multi-stage training, or the 3D-aware representation? No ablation table separates these in Kling-MotionControl’s report.
  • ReTime warps temporal RoPE uniformly across the whole clip toward linear semantic velocity. Could a per-region warp (e.g., enforced only on a subject mask) compose with the global trajectory adapters above for region-specific pacing control?

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2023-12

Section titled “2023-12”