Vista4D: Video Reshooting with 4D Point Clouds

TL;DR

Vista4D is a video-reshooting framework from Eyeline Labs / Netflix that re-renders an input monocular video from a user-specified novel camera trajectory. The core trick is a temporally-persistent 4D point cloud: 4D-reconstruct the source, segment static vs dynamic pixels, persist the static points across all frames, and render that point cloud from the target cameras as conditioning for a finetuned Wan2.1-T2V-14B. To survive real-world reconstruction artifacts at inference, training uses 4D-reconstructed multiview dynamic videos (so the model learns to correct imperfect point cloud geometry instead of doing artifact-free inpainting). The same framework generalizes to 4D scene recomposition (point-cloud editing), dynamic scene expansion (extra scene captures fused into the point cloud), and long-video inference with explicit memory.

Key claims

Static-pixel temporal persistence in the 4D point cloud explicitly preserves seen content and provides camera signal even when the target trajectory shares little per-frame overlap with the source [§3.1].
Training with noisy 4D-reconstructed multiview data — rather than the artifact-free double-reprojected pairs used by TrajectoryCrafter, EX-4D, GEN3C — is what makes the model robust to real-world point cloud artifacts, because it moves the task from inpainting to geometry correction [§3.2, Fig. 3].
Source video and point cloud render are concatenated along the frame dimension with the noisy target latent (“in-context conditioning”) rather than injected via cross-attention as in TrajectoryCrafter; ablated as the most robust to point cloud artifacts [§3.3, Supplementary F].
Camera trajectories are injected as Plücker embeddings via zero-initialized linear projections with an identity-initialized post-self-attention projection, following ReCamMaster [§3.3].
Vista4D outperforms TrajectoryCrafter, EX-4D, GEN3C, ReCamMaster, and CamCloneMaster on camera control accuracy (translation/rotation/intrinsics error) and on per-frame SuperPoint+SuperGlue reprojection error for 3D consistency on a 110-pair DAVIS+Pexels evaluation set [Table 1].
On the iPhone DyCheck multiview benchmark Vista4D leads on PSNR, LPIPS, and optical-flow EPE; trails TrajectoryCrafter on SSIM but the latter’s outputs have visible artifacts SSIM does not catch [Table 2, Fig. 5].
On video fidelity (FID, FVD, VBench aesthetic/imaging/temporal/human-anatomy, CLIP-T) Vista4D beats the explicit-prior baselines; implicit-prior baselines score higher on FID/FVD/consistency only because they fail to follow the target camera, leaving the output close to the source [Table 3].
User study on a random subset of the 110-pair set: users prefer Vista4D over every baseline on content preservation, camera control, and overall fidelity [Table 4].
The 4D point cloud is editable: translate/rotate/scale/remove/duplicate/insert ops on SAM3-segmented point subsets enable 4D scene recomposition, with a worked example of inserting a sunlit rhino into an overcast funeral procession where the model naturally blends dappled lighting [§4.4, Fig. 8].
Dynamic scene expansion fuses casual side-captures of an environment into the same 4D point cloud at reconstruction time, improving environment fidelity without changing the source-video conditioning path [§4.4, Fig. 7].
Long-video inference uses chunk-autoregressive generation with the temporally-persistent point cloud as compressed cross-chunk memory, fitting on a first-frame-conditioned Wan2.1-I2V-14B variant [§4.4, Fig. 9].
Training: finetune Wan2.1-T2V-14B at 672×384 for 30 000 steps then 1280×720 for 3 000 steps, 49-frame clips, global batch size 8, AdamW LR 1e-5; only patchify layers (for source video and point cloud render), self-attention layers, camera encoders, and projectors are trained [§3.4].
Data: synthetic multiview MultiCamVideo (from ReCamMaster) reconstructed with STream3R, plus a 60 K subset of OpenVidHD-0.4M reconstructed with Pi3 for monocular pairs; static-pixel masks via RAM → Llama-3.1-8B-Instruct dynamic-noun filter → Grounded SAM 2 [§3.4].
Vista4D lacks a user-facing knob to interpolate between the explicit point-cloud prior and the implicit (source-video + camera-embedding) prior — flagged as a limitation [§5].

Method

Given a source video V, 4D reconstruction (Pi3 or DA3) produces per-frame depths, extrinsics, and intrinsics; Grounded SAM 2 (with semantic filtering via RAM + Llama-3.1-8B-Instruct) produces a static-pixel mask. Unprojection lifts every pixel to a world-space 3D point and the static mask is reused to make those points visible from every frame, yielding a temporally-persistent 4D point cloud. The point cloud is rendered from the target cameras to produce a control video plus alpha mask. The video diffusion transformer (Wan2.1-T2V-14B) is finetuned to denoise the target video conditioned on three streams concatenated along the frame dimension in latent space: source-video latent, point-cloud-render latent (with alpha mask), and noisy target-video latent. Target cameras are added as Plücker embeddings through zero-initialized linear projections.

The training data trick is the load-bearing piece: instead of double-reprojected monocular pairs (which always view depth maps from their artifact-free frontal direction), training mixes (i) synthetic multiview dynamic scenes from MultiCamVideo with views reconstructed by STream3R — so the point cloud render and target video are spatially mismatched in exactly the way they will be at inference — and (ii) monocular OpenVidHD videos in the double-reprojection mode for diversity. The model thus learns to use the source video to correct geometry the point cloud got wrong, not to inpaint behind a faithful proxy.

Results

110 video-camera pairs from DAVIS (13 videos) + Pexels (38 videos), 2-3 hand-designed target cameras each.
Camera control + 3D consistency [Table 1]: Vista4D outperforms TrajectoryCrafter, EX-4D, GEN3C, ReCamMaster, CamCloneMaster on translation/rotation/intrinsics error and on RE@SuperGlue.
Novel-view video synthesis on iPhone DyCheck [Table 2]: leads on mPSNR, full PSNR, LPIPS, EPE; trails TrajectoryCrafter on SSIM (paper argues SSIM misses the visible artifacts in TC outputs).
Video fidelity [Table 3]: best among explicit-prior methods on FID/FVD/VBench aesthetic+imaging+temporal+human-anatomy; implicit-prior methods score better on FID/FVD/subject-consistency/background-consistency only because they fail to follow the target trajectory and produce near-source outputs.
User study [Table 4]: users prefer Vista4D over every baseline on all three axes (content preservation, camera control, overall fidelity).
Inference variants released: 384p49 (30 000 steps) and 720p49 (3 000 steps, finetuned from the 384p checkpoint). USP multi-GPU inference supported.
WanGP v11.52 has integrated Vista4D for lower-VRAM inference.

Why it’s interesting

Vista4D extends the Camera-Controlled Video Diffusion cluster’s “frozen Wan2.1 + lightweight adapter” pattern, but its design philosophy is closer to NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos (reconstruct-then-clean) than to VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control (control-render-then-generate): both Vista4D and NeoVerse explicitly preserve seen content via a 4D representation and use the video model to correct artifacts. The novel piece is the training-data argument — TrajectoryCrafter, EX-4D, and GEN3C all train on artifact-free pairs, which Vista4D characterizes as silently turning video reshooting into inpainting; training on noisy multiview reconstructions instead forces the model to actually fix geometry, which the authors argue is the real-world distribution. This is also the only filed camera-controlled-video-diffusion paper that explicitly exposes the 4D state for downstream editing (scene recomposition, scene expansion, long-video memory) at inference — a productization story closer to 3D as code‘s 3D-as-code framing than to the other Wan-adapter recipes. The static-pixel temporal persistence trick is also the cleanest filed mechanism for the Context Length / Quality Trade-off in Video Generation regime where target cameras share little per-frame overlap with the source.