InfCam: Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation

TL;DR

InfCam is a depth-free framework for camera-controlled video generation that encodes 3D camera rotations via infinite-homography warping directly in the 2D latent space, letting the diffusion model focus on learning the residual parallax. Combined with a data augmentation pipeline covering diverse trajectories and focal lengths, the approach sidesteps the dominant failure mode of reprojection-based conditioning — depth-estimation errors poisoning the control signal — and reports stronger camera-pose accuracy and visual fidelity than baselines on both a synthetic AugMCV benchmark and WebVid.

Key claims

Reprojection-based camera conditioning is bottlenecked by depth-estimation error, which propagates into the conditioning signal and produces visible artifacts in generated frames [§Motivation].
The infinite-homography component of reprojection (the rotation-only part, requiring no depth) can be applied directly in the 2D latent space, leaving only the parallax residual for the diffusion model to learn [§Overview].
A trajectory + focal-length data augmentation pipeline is necessary alongside the warping prior to cover the camera-control distribution [§Overview].
Outperforms baselines on AugMCV under both shared-intrinsics and differing-intrinsics evaluation, on every metric of pose accuracy and visual fidelity [§Quantitative Comparison].
Outperforms baselines on WebVid on the same axes, with the gap most pronounced on camera-pose accuracy [§Quantitative Comparison].

Method

InfCam splits the standard reprojection-based camera condition into two parts. The infinite-homography component — the limit of the reprojection map as scene depth goes to infinity — encodes only the camera rotation and is computable in closed form from camera intrinsics and the rotation matrix, without any per-pixel depth. InfCam applies this 2D warp to the conditioning latent so that the diffusion backbone receives a rotation-corrected reference, and is trained to produce the residual parallax (the depth-dependent translation component) on top. This makes the conditioning signal robust to the depth-estimation errors that distort reprojection-based baselines. Training pairs are augmented along the trajectory and focal-length axes to cover the camera-control distribution. Evaluation is on a synthetic AugMCV benchmark (with both matched and mismatched source/target intrinsics) and WebVid.

Results

AugMCV (matched and mismatched intrinsics): InfCam outperforms baselines on every reported metric of camera-pose accuracy and visual fidelity [§Quantitative Comparison].
WebVid: same pattern, with particularly pronounced gains on camera-pose accuracy [§Quantitative Comparison].
Qualitative comparison includes ReCamMaster (with interpolation variant) as one of the baselines [§Qualitative Comparison].

Why it’s interesting

Adds a fourth paradigm to Camera-Controlled Video Diffusion alongside the trajectory-adapter recipes (VerseCrafter, SpaceTimePilot), the reconstruction-based hybrid (NeoVerse), and the reference-video conditioning approach (OmniTransfer): a prior-warp recipe where a closed-form rotation prior is baked into the conditioning latent and the model only learns the residual. This is directly relevant to ReCamMaster’s known first-frame-copy failure that SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time fixed with source-aware encoding — InfCam attacks the same family of failures from a different angle (rotation-only warping in latent space rather than dense reprojection with depth). The depth-free framing also contrasts with VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control‘s finding that depth channels are load-bearing for trajectory accuracy (removing depth raised RotErr 0.89 → 1.18); InfCam suggests at least the rotation component of that signal can be supplied analytically rather than learned from depth maps.