CAT3D: Create Anything in 3D with Multi-View Diffusion Models

TL;DR

CAT3D creates 3D scenes from any number of input images (from one to many) by simulating the real-world capture process with a multi-view latent diffusion model. Given input views and a set of target novel viewpoints, the model generates highly consistent novel views, which are then fed into a robust 3D reconstruction pipeline (Zip-NeRF or 3DGS) to produce a renderable 3D representation. The full process — view generation plus 3D reconstruction — runs in as little as one minute, and CAT3D outperforms prior methods for single-image and few-view 3D scene creation. It was a NeurIPS 2024 Oral and is the reference point that later “generate views + reconstruct” 4D-scene pipelines (WildCAT3D, FlashWorld, Wonderland, and the flashworld cluster) benchmark against.

Method

The core component is a multi-view latent diffusion model conditioned on any number of observed views — input images with corresponding camera poses embedded as ray coordinates — and trained to produce multiple consistent novel images at specified target viewpoints [§3]. The architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings, so the diffusion sequence indexes over cameras rather than frames [§3]. The model is initialized from a text-to-image latent diffusion base and fine-tuned on multi-view data.

Given a small set of inputs and a chosen set of target camera poses, CAT3D denoises all target views jointly so that they are internally consistent, then hands the generated view set to a robust 3D reconstruction pipeline that fits a Zip-NeRF or 3D Gaussian Splatting representation to the union of real inputs plus generated novel views [§3, Fig. method-overview]. For scene-scale generation from few inputs it uses an autoregressive sampling scheme over the target viewpoints so that later views condition on earlier generated ones and geometric alignment is preserved across the full trajectory.

Results

Single-image and few-view scene creation outperforms prior methods; the paper reports both novel-view synthesis quality (in-domain and out-of-domain, including mip-NeRF 360) and 3-view reconstruction on mip-NeRF 360 [§4, Table 3].
Multi-view architecture is favored over PixelNeRF-style single-target conditioning (as used in ReconFusion) when dealing with multiple input views [§4, Table 3].
Total processing time — view generation plus 3D reconstruction — runs in as little as one minute per scene [TL;DR].
Accepted as a NeurIPS 2024 Oral.

Why it’s interesting

CAT3D is the canonical “generate consistent multi-view images with a diffusion model, then hand them to an off-the-shelf 3D reconstructor” recipe that later scene-generation methods explicitly benchmark and iterate on — FlashWorld: High-quality 3D Scene Generation within Seconds reports 10–100× speedups over CAT3D as its headline framing, and Cameras as Relative Positional Encoding scales its relative-pose PRoPE ablation to LVSM/CAT3D-class model sizes to show the technique still helps at that scale. It also anchors one side of the design debate captured in 4D Scene Generation and Camera-Controlled Video Diffusion between the “generate views then reconstruct” family (CAT3D, FlashWorld, NeoVerse’s reconstruction leg) and the “control-first frozen video DiT + adapter” family (VerseCrafter, SpaceTimePilot, OmniTransfer) — filing the reference itself lets those downstream pages cite it as a resolved slug rather than a floating name.