SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time

TL;DR

SpaceTimePilot is a video diffusion model that disentangles camera (space) and animation (time) control for monocular-video re-rendering. Given one input video of a dynamic scene, it can independently change the camera trajectory and the temporal playback (reverse, freeze, bullet-time, slow-motion, zigzag) while preserving scene identity. The two contributions that make this work are (1) an animation-time embedding injected via 1D convolution on sinusoidal frame-index embeddings, separate from the camera-conditioning path, and (2) a temporal-warping training scheme plus a new synthetic CamTime dataset (180k videos across 100 scenes, full camera×time grid) that supplies the supervision real multi-view captures cannot. Built on Wan-2.1 T2V-1.3B with ReCamMaster-style camera conditioning extended to source-aware trajectories.

Key claims

A dedicated animation-time signal τ injected as sinusoidal embeddings (compressed from F=81 frames to L=21 latent frames via 1D conv) and added to camera+token features, separate from RoPE frame-index, is needed to disentangle time from camera; RoPE-based temporal control entangles the two [§3.2.1, Eq. 2].
Temporal-warping augmentation (reverse, acceleration, freeze, segmental slow-motion, zigzag) over existing multi-view video datasets is the load-bearing data trick — it manufactures (source, target) pairs that differ in both camera trajectory and temporal dynamics without collecting new footage [§3.2.2, Fig. 3].
The CamTime synthetic dataset — 180k Blender-rendered videos across 100 scenes × 500 animations × 3 camera paths, exhaustively sampling the camera×time grid — supplies dense supervision that no prior real or synthetic dataset provided [§3.2.2, Tab. 1].
A source-aware camera conditioning (jointly encoding source and target trajectories instead of assuming a shared first frame) removes ReCamMaster’s first-frame-copy artifact and is needed for full trajectory control [§3.3, Eq. 3].
On the held-out CamTime test split with fixed camera and varying temporal signal, SpaceTimePilot beats ReCamM+preshuffled and ReCamM+jointdata baselines by large margins: avg PSNR 21.16 vs 15.52 / 17.86, LPIPS 0.1764 vs 0.4529 / 0.3073 [Table 2].
On 1800 real-world generations evaluated by VBench, SpaceTimePilot matches or beats TrajectoryCrafter and ReCamMaster on imaging quality, background consistency, motion, subject consistency, flicker, aesthetic [Table 3].
Source-camera conditioning improves camera-control accuracy over ReCamM and ReCamM+Aug across both relative and absolute trajectory protocols on a 90-video OpenVideoHD evaluation set (1800 generated videos, poses recovered via SpatialTracker-v2) [§4.1.3].
An autoregressive segment-generation strategy enables longer videos (e.g. extended bullet-time rotations from 0→π then π→2π) by conditioning each segment on the previously generated one plus the source [§3.4].

Method

SpaceTimePilot builds on a Wan-2.1 T2V-1.3B latent video diffusion backbone (3D-VAE compressing 81 RGB frames to 21 latent frames; DiT denoiser over multi-modal tokens) and on ReCamMaster’s camera-conditioning recipe. Camera extrinsics are flattened to 12-d vectors per frame, encoded by E_cam, and added to patchified video features before self-attention. The new piece is the animation-time embedder E_ani: it takes a temporal control signal τ ∈ R^F (F=81), computes per-frame sinusoidal embeddings, then applies two 1D conv layers to compress from F to the latent L=21, and adds the result to the same feature stream (Eq. 2). For full trajectory control the source camera trajectory is also injected (Eq. 3), with target and source tokens concatenated along the frame dimension. Only E_cam, E_ani, self-attention, and a small projection layer are trained; the base diffusion model is frozen.

Training uses ReCamMaster + SynCamMaster multi-view datasets with on-the-fly temporal warping (reversal, acceleration, freeze, segmental slow-motion, zigzag) plus the new CamTime synthetic dataset. CamTime is rendered in Blender across 100 scenes × 500 animations × 3 camera paths, where each path covers the full grid of (camera-viewpoint × animation-time); source videos sample the diagonal, target videos sample arbitrary continuous trajectories.

Results

Time-control evaluation on the 50-scene CamTime test split with fixed camera pose: SpaceTimePilot averages PSNR 21.16 / SSIM 0.7674 / LPIPS 0.1764, vs. ReCamM+preshuffled 15.52 / 0.6213 / 0.4529 and ReCamM+jointdata 17.86 / 0.7250 / 0.3073 [Table 2]. The advantage holds uniformly across directional (reverse), speed (slow-motion / zigzag), and bullet-time categories.

VBench visual-quality eval over 1800 generated videos: ImgQ 0.6486, MotionSmoothness 0.9947, SubjectConsistency 0.9325 — competitive with or above TrajectoryCrafter (0.6389 / 0.9888 / 0.9463) and ReCamMaster baselines [Table 3].

Camera-control eval on 90 OpenVideoHD clips × 20 trajectories (10 from same initial pose, 10 from different): source-aware conditioning is the variant that wins on both relative and absolute rotation/translation error metrics; ReCamM+Aug (more augmentation but no source camera) actually regresses vs. plain ReCamM, indicating that augmentation without an explicit reference signal confuses the model [§4.1.3, Table 5].

Qualitative: only SpaceTimePilot correctly executes both camera motion (red boxes) and animation-time state (green boxes) jointly; ReCamMaster handles camera but cannot reverse time, and TrajectoryCrafter’s warp-and-inpaint pipeline misroutes the last-source-frame camera pose into the first generated frame when given a reverse temporal signal [§4.1.4, Fig. 6].

Why it’s interesting

This is the cleanest treatment of space/time disentanglement in a video diffusion model the wiki has seen — most “camera control” work (ReCamMaster, GCD, TrajectoryCrafter) keeps time strictly monotonic and tied to the source, while 4D reconstruction work (CAT4D, Diffusion4D) gets disentanglement but only at sparse discrete viewpoints. The temporal-warping recipe is also notable as a synthetic-supervision trick: it manufactures the conditioning diversity that real multi-view captures structurally cannot provide, which is the same flavor of move seen in pixel-space / agent-trajectory work the wiki already tracks (Synthetic Training Data). The CamTime dataset (full camera×time grid, to be released) could be a useful benchmark independent of the model.