HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

TL;DR

HY-World 2.0 is Tencent Hunyuan’s open-source multi-modal world model that produces real 3D assets (3DGS, meshes, point clouds) rather than pixel videos, from text, single-view image, multi-view images, or casual video input. It splits cleanly into two stacks: World Generation — a four-stage pipeline (HY-Pano 2.0 panorama → WorldNav trajectory planning → WorldStereo 2.0 world expansion → WorldMirror 2.0 + 3DGS world composition) that turns a text/image prompt into a navigable, editable 3D world importable into Unity / Unreal / Isaac Sim; and World Reconstruction — WorldMirror 2.0, a unified feed-forward model that simultaneously predicts depth, surface normals, camera parameters, point clouds, and 3DGS attributes from multi-view images or video in a single forward pass. The release covers the technical report, model weights (HY-Pano-2 ~80B + 425M LoRA, WorldStereo-2 ~17B, WorldMirror-2 ~1.2B), and full inference code under one repo.

Key claims

3D-asset world models structurally avoid the durability/consistency/cost failure modes of video world models: outputs are editable and persistent, 3D-consistent by construction, render in real time on consumer GPUs, and incur generation cost only once rather than per-interaction [README §Why 3D World Models].
World Generation factors into four stages — Panorama Generation, Trajectory Planning, World Expansion, World Composition — that decouple appearance (panorama), camera schedule (navmesh-based trajectory), novel-view synthesis (stereo-style outpainting), and geometry consolidation (feed-forward 4D reconstruction + 3DGS optimization) [README §Architecture].
WorldMirror 2.0 is a single feed-forward model that jointly predicts depth, normals, camera intrinsics/extrinsics, point clouds, and 3DGS attributes — eliminating the per-task estimator stack typical of reconstruction pipelines [README §Introduction].
WorldMirror 2.0 supports flexible-resolution inference (50K–500K pixels) and prior injection (camera extrinsics + intrinsics + depth) — with prior injection cutting accuracy/completeness errors roughly in half on 7-Scenes/NRGBD and ~35% on DTU [README Point Map Reconstruction table].
WorldStereo 2.0 beats prior camera-controlled video-generation baselines (SEVA, Gen3C) on both camera-trajectory metrics (RotErr 0.492 vs SEVA 1.690 / Gen3C 0.944; TransErr 0.968 vs 1.578 / 1.580; ATE 1.768 vs 2.879 / 2.789) and visual quality (Q-Align 4.205 vs 3.232 / 3.353; CLIP-I 89.43 vs 77.16 / 82.33) [README Camera Control table].
On single-view → 3D reconstruction (Tanks-and-Temples + MipNeRF360), WorldStereo 2.0 — both the base model and a DMD-distilled variant — leads on F1 and AUC over SEVA, Gen3C, Lyra, and FlashWorld, with the DMD variant pushing Tanks F1 from 41.43 → 43.16 and AUC from 58.19 → 60.09 [README Single-View-Generated Reconstruction table].
The full HY-World 2.0 v2 release upgrades all four sub-modules over the v1 stack (HunyuanWorld 1.0, WorldMirror v1, WorldStereo v1) and adds end-to-end interactive character / first- and third-person navigation with physics-based collision into the same generated world [README §Highlights, Open-Source Plan].
The pipeline is built around a frozen-3DGS, learned-everything-else design: the 3DGS optimizer is the conventional differentiable renderer (a custom gsplat variant gsplat_maskgaussian), with all upstream stages providing pretraining-quality initialization (frames + aligned depth + normals + cameras from WorldMirror 2.0) [README §Code Usage — World Generation].

Method

The world-generation pipeline is a five-stage script chain (traj_generate.py → traj_render.py → video_gen.py → gen_gs_data.py → world_gs_trainer.py). Stage 1 (HY-Pano 2.0) produces a 360° panorama from a text prompt or input image; the model ships in two sizes (~80B base, plus a ~425M Qwen-LoRA variant). Stage 2 (WorldNav) runs VLM-guided trajectory planning over a navmesh built with recastnavigation — obstacle-aware camera paths over the panoramic scene. Stage 3 (WorldStereo 2.0) does memory-guided keyframe generation as the camera traverses the planned trajectory, with explicit camera-pose conditioning (the camera-control numbers in §Results are this stage’s). Stage 4 extracts per-frame depth, normals, and camera parameters via WorldMirror 2.0, producing aligned multi-view supervision. Stage 5 runs 3DGS optimization (the gsplat_maskgaussian custom kernel) to produce the final exportable Gaussian Splat world. WorldMirror 2.0 itself is a transformer-based feed-forward model that takes multi-view images or video and emits joint depth + normal + camera + point-cloud + 3DGS predictions; it accepts optional camera and depth priors that condition the output through a documented prior-injection interface. Reconstructed scenes are returned in a diffusers-style pipeline API (WorldMirrorPipeline.from_pretrained(...)) and via CLI with FSDP + bf16 multi-GPU support.

Results

The technical report (arXiv:2604.14268) is referenced from the README; the README itself surfaces three benchmark blocks:

Camera control (WorldStereo 2.0). Against SEVA / Gen3C / WorldStereo v1 on the camera-controlled video-generation benchmark: RotErr 1.690 / 0.944 / 0.762 → 0.492, TransErr 1.578 / 1.580 / 1.245 → 0.968, ATE 2.879 / 2.789 / 2.141 → 1.768. Visual quality: Q-Align 3.232 / 3.353 / 4.149 → 4.205, CLIP-IQA+ 0.479 / 0.489 / 0.547 / 0.544, Laion-Aes 4.623 / 4.863 / 5.257 → 5.266, CLIP-I 77.16 / 82.33 / 89.05 → 89.43.
Single-view-generated reconstruction. On Tanks-and-Temples, F1-Score: SEVA 36.73, Gen3C 31.24, Lyra 32.54, FlashWorld 22.29, WorldStereo 2.0 41.43, WorldStereo 2.0 (DMD) 43.16; AUC: 51.03 / 42.44 / 43.05 / 30.45 → 58.19 / 60.09. On MipNeRF360, F1: 28.75 / 35.26 / 36.05 / 42.60 → 51.27 / 50.52; AUC: 46.81 / 52.10 / 49.89 / 53.86 → 65.79 / 65.64.
WorldMirror 2.0 point-map reconstruction. On 7-Scenes / NRGBD / DTU with prior injection (camera + depth) at medium/high resolution, Acc/Comp drop to 0.012 / 0.016 (7-Scenes H), 0.013 / 0.013 (NRGBD M), 0.554 / 0.771 (DTU H) — roughly halving errors versus WorldMirror v1 at the same resolution + prior configuration, and dominating Pow3R / MapAnything across prior conditions on the averaged 7-Scenes/NRGBD/DTU metric per a separate figure.

No quantitative comparison is given against Genie 3, Cosmos Predict 2.5, or LingBot-World; the comparison set is camera-controlled video and feed-forward reconstruction baselines.

Why it’s interesting

This is the first filed open world model in the wiki that explicitly bets against pixel-video as the output modality and ships a working asset-export path (3DGS / mesh → Blender / Unity / UE / Isaac). It complements Lyra 2.0: Explorable Generative 3D Worlds — Lyra 2.0 also produces 3DGS/mesh and exports to Isaac Sim, but uses 3D state for retrieval/routing only (appearance from the DiT) while HY-World 2.0 uses 3DGS as the primary output substrate of an optimization stage at the end. It contrasts with Project Genie: Experimenting with infinite, interactive worlds (closed flagship, real-time pixel-video product surface, no asset export) and with Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory (real-time interactive video world model with long-horizon memory — the open-source video-output side of the same Tencent/Skywork-style product axis). The “3D-asset world model” framing aligns directly with Sitzmann’s Bitter-Lesson critique in The flavor of the bitter lesson for computer vision only at the human–machine interface layer (3D meshes / 3DGS as the export format), while the internal design still uses an explicit-3D-state intermediate (point cloud + multi-view depth/normals → 3DGS optimization) — so this is a working datapoint for “explicit 3D at the output, generative model upstream” as a design pattern.