Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

TL;DR

Lift4D is a test-time optimization framework that reconstructs a complete, deformable 4D Gaussian Splatting representation of a dynamic object from a single in-the-wild monocular video. It bridges two prior families — direct per-frame 3D regression (data-limited) and video-supervised deformation refinement (no prior beyond initialization) — by (a) adapting a pretrained image-to-3D DiT to emit temporally consistent per-frame Gaussian splats via causal latent propagation and (b) “sculpting” a deformable 3DGS canonical representation against the input video using occlusion-aware optimization driven by a view-conditioned diffusion prior. Targets the regime where large deformations and severe occlusions break feed-forward 4D reconstructors.

Key claims

A pretrained single-view image-to-3D DiT can be turned into a temporally consistent per-frame 3D predictor by causal latent conditioning: each frame’s 3D latent is initialized by mixing fresh noise with the previously denoised latent before decoding to independent Gaussian splat sets [§Method].
The per-frame Gaussian splat sets are consolidated into a single 4D representation of canonical Gaussians animated by two sets of sparse deformation nodes — one fit to per-frame geometry via a reconstruction loss, the second a fine appearance-deformation set optimized jointly with vertex color [§Method].
Appearance is supervised by an occlusion-inpainted novel-view diffusion prior: the 4D scene is rendered from random novel views, noised, and denoised by a video diffusion model conditioned on input frames whose occlusions have been inpainted using the per-frame 3D outputs; this aggregates visible details across frames and hallucinates unobserved regions [§Method].
Lift4D outperforms prior 4D reconstruction baselines on both synthetic and in-the-wild footage, with reported gains on temporally coherent geometry, sharpness, and motion accuracy under heavy occlusion [Results].

Method

Lift4D is structured as two stages glued by an explicit 4D representation. Stage 1 — causal-latent per-frame 3D prediction: a pretrained image-to-3D diffusion transformer is run on each video frame, but the denoising trajectory of frame t is initialized by mixing the denoised latent of frame t-1 with fresh noise. This causal latent propagation regularizes per-frame outputs to a temporally consistent manifold without retraining the DiT on 4D data, sidestepping the 4D-data scarcity bottleneck. Each frame yields an independent set of 3D Gaussian splats. Stage 2 — deformable 4DGS sculpting: the per-frame splat sets are consolidated into one canonical 3DGS plus two sparse deformation-node sets. Geometry deformation nodes are fit to the per-frame predicted splats through a reconstruction loss ℒ_rec. Appearance is refined separately by an ℒ_app term that (i) renders the 4D reconstruction from random novel views, noises it, and (ii) denoises through a view-conditioned video diffusion prior conditioned on per-frame occlusion-inpainted frames where occluded pixels are filled from the per-frame 3D outputs. This is novel-view sample distillation plus a rendering loss on visible pixels, jointly optimizing per-vertex color and the fine appearance deformation nodes.

Results

The project page reports Lift4D outperforms prior 4D reconstruction baselines on both synthetic benchmarks and in-the-wild footage, with particular emphasis on challenging sequences containing severe occlusions and large non-rigid motion. Quantitative tables are in the arxiv preprint (2606.23688); the page itself defers numerical detail to interactive 4D viewers per scene.

Why it’s interesting

Lift4D is the first filed paper in the 4D Scene Generation cluster to combine the reconstruction-then-clean-up pattern (already seen at scene granularity in NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos) with an explicit deformable 3DGS canonical model + diffusion-driven occlusion inpainting at object granularity — a different design point from Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy (canonical mesh + SVD ControlNet, edit-oriented) and from feed-forward 4DGS regressors. It tackles the failure mode none of the existing scene-level methods directly address: heavy occlusion in monocular input, where both feed-forward predictors and video-loss-only refinement degrade. The two-stage decomposition (causal-latent for geometry initialization, novel-view diffusion prior for appearance completion) lines up with the cluster’s broader pattern of using video/3D diffusion models as priors rather than primary generators.