Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation

TL;DR

Voyager turns a HunyuanVideo image-to-video DiT into an explorable 3D-scene generator: from a single image and a user-defined camera trajectory it jointly produces aligned RGB and depth video frames, then unprojects them into a growing point-cloud world cache that conditions subsequent autoregressive clips. The joint RGB+depth output enables direct 3DGS reconstruction without structure-from-motion or multi-view stereo, and the point-cloud cache (with view-angle culling) gives bounded memory cost for arbitrarily long camera paths. Trained on ~100k clips assembled by a data engine that fuses VGGT poses, MoGE depth, and Metric3D scale, the model beats SEVA, ViewCrafter, See3D, and FlexWorld on RealEstate10K image-to-video PSNR/SSIM/LPIPS and tops the WorldScore static benchmark on overall score.

Key claims

Joint RGB+depth generation, by concatenating the depth video to the RGB video along the height axis with a placeholder separator row, lets the DiT’s full attention fuse visual and geometric information at the pixel level [§4.2].
A partial-RGB condition rendered from a warped point cloud is not sufficient for spatial consistency in complex scenes; adding a spatially aligned partial-depth channel as a second geometric condition fixes hallucinated occlusions [§4.1, Fig. 2].
Context-based control blocks (zero-initialized replicas of the first dual-stream and single-stream DiT blocks, injecting early-stage latent features into every later block) are necessary on top of the channel-wise condition concatenation to enforce pixel-level geometric following [§4.2, Eqs. 3–4].
A point-culling rule that adds a new point only if it is invisible from the current view or the angle between its surface normal and the view direction exceeds 90° reduces stored points by ~40% with comparable visual quality to retaining all points [§4.3, Fig. 8].
Smooth video sampling — overlapping segments with shared noise initialization, post-hoc averaging plus a light noise injection followed by a final denoising round — eliminates color discrepancies at clip boundaries during autoregressive extension [§4.3].
On RealEstate10K (150 test clips), Voyager beats SEVA / ViewCrafter / See3D / FlexWorld on image-to-video PSNR (18.751), SSIM (0.715), and LPIPS (0.277) [Table 1].
For scene reconstruction with VGGT post-hoc, Voyager outperforms all four baselines (PSNR 17.742, SSIM 0.712, LPIPS 0.404); initializing the 3DGS from Voyager’s own depth output is even better (PSNR 18.035, LPIPS 0.381), evidence that the generated depth is geometrically usable [Table 2].
On WorldScore static (2,000 examples, 6 baselines), Voyager achieves the highest overall score 77.62, beating WonderWorld (72.69), WonderJourney (63.75), CogVideoX-I2V (62.15), Gen-3, Allegro, and EasyAnimate; particularly strong on subjective quality and on larger camera moves enabled by metric depth [Table 3].
Ablation on WorldScore shows depth fusion alone lifts Camera-Object 48.92 → 65.72 and Style 74.98 → 85.04 over RGB-only conditioning; adding Control blocks gives a further bump to 68.92 / 85.95 [Table 4].
The data engine combines VGGT (pose + rough depth aligned to pose) with MoGE (accurate depth, aligned via least-squares) and Metric3D (metric scale), producing a 100k-clip training set from RealEstate10K, DL3DV, and 10k UE renders without manual 3D annotation [§4.4].

Method

Voyager builds on the HunyuanVideo dual-stream-then-single-stream DiT with a 3D-VAE latent space. Given an input image, the model unprojects it to a point cloud using estimated depth + camera. For each frame of a user-specified camera trajectory, the cloud is rendered to produce a partial RGB image (with blank regions for unseen areas) and an aligned partial depth map plus a rendering mask. The DiT generates the missing regions of both the RGB and depth videos jointly: the two modalities are concatenated along the height axis as a single tall video latent (with a placeholder separator row), the condition latents and masks are concatenated along the channel axis to the noisy latents and passed through the patch embedding, and a pair of zero-initialized Control blocks (replicas of the first dual-stream and single-stream blocks) inject early-stage latents into every later block to amplify the geometric signal.

For long-range exploration the model is run autoregressively. After each clip, the newly generated RGB-D frames are unprojected and merged into a world cache using a point-culling rule (skip new points that are visible from prior viewpoints and whose surface normal aligns within 90° of an existing point’s view direction — ~40% storage reduction). Adjacent clips overlap by half a segment; the overlap is noise-initialized from the prior clip and a final post-denoising pass smooths the boundary.

The data engine is a three-stage pipeline applied to monocular video: VGGT for camera poses and rough pose-aligned depth → MoGE for accurate depth via least-squares depth alignment → Metric3D for scene-scale alignment to UE metric depth. The training corpus is RealEstate10K (74,766 clips), DL3DV (3,000 curated → 18,000 clips), and ~10,000 UE renders from 1,500 scene models.

Results

RealEstate10K image-to-video (150 clips): PSNR 18.751 / SSIM 0.715 / LPIPS 0.277, beating FlexWorld (18.278 / 0.693 / 0.281), See3D (18.189 / 0.694 / 0.290), ViewCrafter (16.512 / 0.636 / 0.332), SEVA (16.648 / 0.613 / 0.349) [Table 1].
Scene reconstruction (3DGS rendering quality): Voyager with VGGT post-hoc reaches PSNR 17.742 / SSIM 0.712 / LPIPS 0.404, best among all baselines; Voyager with its own depth jumps to PSNR 18.035 / SSIM 0.714 / LPIPS 0.381, the largest single improvement attributable to native depth output [Table 2].
WorldScore static (2,000 examples, average across 7 sub-metrics): Voyager 77.62 overall, vs. WonderWorld 72.69, WonderJourney 63.75, CogVideoX-I2V 62.15, Gen-3 60.71, Allegro 55.31, EasyAnimate 52.85 [Table 3]. Voyager is strongest on subjective quality (71.09) and competitive with 3D-based WonderWorld on camera control (85.95 vs. 92.98) while outpacing all video-based baselines on Camera-Object (66.92).
Ablations on WorldScore Style / Camera-Object / Object: RGB-only 74.98 / 48.92 / 68.86 → RGB-D 85.04 / 65.72 / 78.58 → +Control blocks 85.95 / 68.92 / 81.56 [Table 4]. Depth fusion is the dominant lever; Control blocks add ~3 points on top.
Point-culling reaches ~40% storage savings at parity with full-cloud rendering [§4.3, Fig. 8]; smooth sampling visibly removes clip-boundary inconsistency in qualitative examples.
Applications demonstrated: long videos composed of three clips with arbitrarily different per-clip camera trajectories, image-to-3D, video transfer, and depth estimation (qualitative only) [§6, Fig. 6].

Why it’s interesting

Voyager occupies a different point on the design-space than the other camera-controlled video diffusion papers on file. The recent cluster — SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time, NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos, VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control — is frozen Wan2.1 + lightweight geometric adapter; Voyager instead fine-tunes the full HunyuanVideo DiT and changes its output channels (RGB and depth jointly, height-axis concatenated). Three things make this paper worth tracking for worldbuilding: (i) the depth channel is generated rather than reconstructed, so 3DGS initialization from Voyager’s own depth strictly beats VGGT post-hoc on the same generated frames (a rare case of native depth measurably beating downstream MVS in the filed cluster); (ii) the autoregressive world cache with view-angle point culling is a different memory-management point from the KV-cache partitioning of Context Forcing: Consistent Autoregressive Video Generation with Long Context and PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference — the bounded representation here is a 3D point set, not a token cache, which removes the RoPE-related length-extrapolation pressures characterized in RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably; (iii) WorldScore puts Voyager above WonderWorld on overall score, which is the first filed datapoint of a generative-rollout WFM beating a 3D-first WonderWorld-style pipeline on the official static benchmark — a partial empirical answer to Sitzmann’s foundation-layer question raised in The flavor of the bitter lesson for computer vision (though Voyager keeps an explicit 3D world cache, so the answer is “yes, but with explicit 3D state retained”).