GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

TL;DR

GenRecon casts multi-view 3D scene reconstruction as conditional 3D generation over a set of overlapping spatial chunks that jointly tile the scene, inheriting the fidelity and completeness of Trellis.2’s object-level generative shape prior. A projection-based 3D conditioning pathway lifts posed multi-view image features into a voxel grid anchored to the scene coordinate frame, so pose control and view consistency fall out of the conditioning geometry rather than needing to be learned. At inference all chunks are denoised jointly under a MultiDiffusion-style overlap-averaging scheme, producing an editable PBR mesh of an indoor scene. On ScanNet++ and 3D-FRONT it beats 2DGS, MonoSDF, DA3, FineRecon, and Murre across depth, normal, and 3D metrics, with a claimed ~16% improvement over the strongest baseline.

Key claims

Casting scene reconstruction as joint conditional 3D generation over overlapping fixed-size chunks lets an object-level generative prior (Trellis.2) scale to scene extents while inheriting object-level fidelity and completeness [§3.2].
Lifting per-view DINOv3 features into a shared 3D voxel grid via camera projection and aggregating them via an IBRNet-style permutation-invariant mean + softmax-weighted residual makes the conditioning view-order-invariant and spatially anchored to the scene frame [§3.1, Eq. 2].
Residual, voxel-wise, zero-initialized injection of the 3D condition into every denoiser block preserves the pretrained Trellis.2 prior at training initialization, so LoRA fine-tuning on synthetic scene data does not have to relearn the shape prior [§3.1 “Conditioning injection”].
A single global conditioning grid computed once over the whole scene (then cropped per chunk) plus MultiDiffusion-style overlap averaging of the shared noisy latent enforces cross-chunk consistency throughout the flow-matching trajectory [§3.2 “Global 3D conditioning”, “Joint chunk generation”, Eq. 3].
The 3D conditioning pathway is what enables pose control: a vanilla Trellis.2 (single-image, pose-free) fails on scene chunks; fine-tuned Trellis.2 without the 3D condition produces plausible fragments in the wrong pose; the full method aligns generated geometry with ground-truth pose [§4.2, Table 5].
Chunk generation is pose-correct from a single input image and improves monotonically with more views: on SAGE-10k chunks, Chamfer 0.1345 → 0.0812 → 0.0420 → 0.0291 and F-score@10cm 0.6312 → 0.7608 → 0.9070 → 0.9683 as views go 1 → 2 → 4 → 8 [Table 5].
On out-of-domain real-world ScanNet++ (8 input views, 25 scenes), GenRecon beats 2DGS / MonoSDF / DA3 / FineRecon / Murre on 2D and 3D metrics — Chamfer 0.0688 vs FineRecon’s 0.0819, F-score@10cm 0.7771 vs 0.7694, normal consistency 0.7860 vs 0.7851, and completeness 0.9904 vs MonoSDF’s 0.9716 [Table 1, Table 2].
On synthetic 3D-FRONT the gap is larger: Chamfer 0.0638 vs Murre’s 0.1584 (2.5× lower), F-score@10cm 0.8655 vs 0.6014, normal consistency 0.8199 vs 0.6624, and normal angular error 10.75° vs FineRecon’s 15.41° [Table 3, Table 4].
The 3D generative prior enables plausible PBR material recovery (albedo, metallic, roughness) that supports relighting under novel illumination on real ScanNet++ captures, though absolute material accuracy is bounded by SAGE-10k’s own VLM-predicted PBR labels rather than ground-truth SVBRDF [§4.3].
Training uses parameter-efficient LoRA adapters on the Trellis.2 denoiser with the rest of the base model frozen; the conditioning pathway is trained jointly [§3.1 “Training”].
Failure modes: non-Lambertian surfaces (glass, mirrors) are unreliable due to underrepresentation in SAGE-10k; the chunking is designed for indoor scenes with ≤5 m vertical extent; the strong prior can hallucinate content in weakly-observed regions [§4.4].

Method

Given a sparse set of posed RGB images with known intrinsics/extrinsics of an indoor scene, the pipeline: (1) recovers or ingests camera poses via COLMAP and estimates the scene’s axis-aligned extent from a filtered sparse point cloud; (2) partitions the scene volume into a set of overlapping fixed-size cubic chunks $\{c_i\}$ with world-frame translations $\{t_i\}$ and a prescribed overlap margin; (3) encodes every input image once with a frozen DINOv3, then lifts each 2D feature map into a scene-sized 3D voxel grid by projecting each voxel back into every view (retrieving the corresponding pixel-space feature); (4) aggregates across views into a single global 3D condition grid $C$ via a permutation-invariant IBRNet-style scheme (mean + zero-initialized softmax-weighted residual over cross-view statistics); (5) extracts per-chunk conditions $C_i$ as crops of $C$ ; (6) runs the modified Trellis.2 denoiser jointly over all chunks in a MultiDiffusion-style flow-matching trajectory, maintaining a single global noisy latent and averaging per-chunk predictions in overlap regions at every step; (7) decodes the final global latent with the Trellis.2 decoders into a PBR-textured mesh.

Two Trellis.2 modifications make this work: a voxel-wise, zero-initialized residual injection of $C_i$ into every denoiser block (so the pretrained prior is preserved at $t=0$ ), and a low-rank LoRA adapter on the denoiser weights. Everything else in Trellis.2 stays frozen. Training data is SAGE-10k for the shape and PBR stages, augmented with a subset of 3D-FRONT scenes for the occupancy stage (to add multi-room layouts, windows, and door openings that SAGE-10k lacks).

Results

Evaluation is on 25 unseen scenes from each of 3D-FRONT and ScanNet++, with 8 input views each. Baselines span the dominant reconstruction paradigms: 2D Gaussian Splatting (prior-free per-scene opt), MonoSDF (mono depth prior + per-scene opt), Depth Anything 3 + TSDF fusion (feed-forward monocular depth foundation model), FineRecon (feed-forward 3D refinement of fused monodepth), and Murre (diffusion depth priors + 3D conditioning).

On ScanNet++ (real, entirely out-of-domain since GenRecon was trained on synthetic SAGE-10k / 3D-FRONT): Chamfer 0.0688 m (vs FineRecon 0.0819, Murre 0.0978), F-score@10cm 0.7771 (vs 0.7694), normal consistency 0.7860 (vs 0.7851), completeness 0.9904 (vs MonoSDF 0.9716). GenRecon leads across 2D depth, normal, and perceptual metrics — depth SqRel 0.0532 vs FineRecon 0.0853, depth LPIPS 0.0872 vs MonoSDF 0.1143, normal LPIPS 0.3411 vs MonoSDF 0.4201. [Tables 1, 2]

On 3D-FRONT (synthetic, held-out): Chamfer 0.0638 vs Murre 0.1584 (2.5× lower), F-score@10cm 0.8655 vs 0.6014, normal consistency 0.8199 vs 0.6624, normal angular error 10.75° vs FineRecon 15.41°. [Tables 3, 4]

The view-count ablation on SAGE-10k chunks is monotonic and strong: F-score@10cm 0.6312 at 1 view → 0.9683 at 8 views; Chamfer 0.1345 → 0.0291. Ablating the 3D conditioning pathway (fine-tuning Trellis.2 without it) recovers appearance but loses pose alignment — F-score@10cm drops from 0.6312 to 0.3666, confirming the projection-based conditioning is the load-bearing structure for pose control rather than fine-tuning alone. [Table 5]

Why it’s interesting

GenRecon sits at an unusual intersection of two clusters the wiki has been tracking separately. On one side, it is a very clean instance of the Parameter-Efficient Finetuning recipe that DiffHDR / LumiVid / Fit-VTO / DKT / SPDA-SAM all instantiate — frozen large pretrained generative backbone + tiny LoRA + carefully-shaped structured input conditioning to span a large domain gap — but here the backbone is a 3D generative model (Trellis.2) and the “input transform” is a full 3D projection pipeline rather than a color-space curve. That is the biggest domain gap this recipe has crossed on the wiki so far. On the other side, it is a scene-scale reconstruction system whose closest sibling in scope is SAM 3D: 3Dfy Anything in Images (SAM 3D Objects: single-image object generation via post-training on real-annotated data) — GenRecon is the same “generative prior as reconstruction backbone” bet but at scene scale and via multi-view conditioning rather than a big real-annotation post-training corpus. Compared with WorldGen: From Text to Traversable and Interactive 3D Worlds (Meta’s text-to-traversable-mesh WorldGen), it is the reconstruction counterpart — same mesh output shape and PBR ambition, but conditioned on real captures rather than a text prompt, and inheriting a stronger object-level shape prior via chunk-tiling instead of training a scene-scale generator from scratch. The chunk-tiling + MultiDiffusion-style joint denoising is also structurally similar to InfiniteDiffusion: Bridging Learned Fidelity and Procedural Utility for Open-World Terrain Generation‘s bounded-model-as-infinite-generator trick, just applied on a static bounded volume rather than an unbounded one.