LiteReality-Agent goes simulation-ready — Layout Agent, articulated objects, MuJoCo integration, GPT-6 small-object recovery

TL;DR

Zhening Huang (Cambridge) announces a “simulation-ready” upgrade to LiteReality-Agent — the agentic follow-up to the NeurIPS 2025 LiteReality pipeline that turns RGB-D room scans into graphics-ready 3D scenes. The update targets four gaps that blocked the original graphics-ready output from being directly usable in physics simulators: a Layout Agent that cleans noisy scan layouts into physically valid ones, an upgraded articulated-object reconstruction with comprehensive physical properties, a MuJoCo integration so reconstructed rooms drop straight into physics simulation, and a GPT-6-powered small-object recovery pass (via OpenAI’s “Astra” agent) that fills in objects the original pipeline missed. Code lives at github.com/LiteReality/LiteReality-Agent.

Key claims

• The original LiteReality pipeline produces graphics-ready scenes from real-world scans but the outputs are not simulation-ready without additional cleanup [tweet body].
• A dedicated Layout Agent can turn noisy scanned layouts into clean, physically valid, simulation-ready layouts automatically [tweet body].
• Articulated-object reconstruction has been upgraded with a comprehensive set of physical properties for simulation [tweet body].
• Reconstructed rooms now integrate directly with the MuJoCo physics engine [tweet body].
• GPT-6 (“Astra”) is used to detect and add small objects that the original perception pipeline missed [tweet body].

Method

The tweet describes four modular upgrades layered on top of the previously filed LiteReality pipeline (Huang et al., NeurIPS 2025) — the base pipeline already reconstructs a scanned indoor scene as individual retrieved CAD-quality objects with PBR materials placed inside a parsed room layout. The updates: (a) a Layout Agent replaces the deterministic layout-parsing step with an agentic pass that repairs geometrically invalid configurations (overlaps, floating objects, wall-clipping) into physically valid arrangements; (b) the articulated-object module — which produces URDF-style joint kinematics from the scan — is extended with mass, inertia, friction, and damping properties needed for physics rollouts; (c) an exporter to MuJoCo ships the assembled scene into a runnable physics simulation without a manual re-authoring step; (d) an Astra / GPT-6 small-object recovery pass examines the RGB scan and cued rendering of the reconstruction, and inserts small props (bottles, remotes, keys) that the original pipeline’s asset-retrieval step missed. The tweet does not disclose evaluation numbers, dataset sizes, or an ablation of the four components; the linked repo github.com/LiteReality/LiteReality-Agent presumably contains the details but was not readable at filing time.

Results

None reported in the tweet beyond the qualitative claim that reconstructed rooms are now “simulation-ready” in MuJoCo. No sim-to-real transfer, task-success, or layout-repair benchmarks are cited.

Why it’s interesting

This is a concrete step in the “real-to-sim as agentic composition” direction the wiki has been tracking. It complements Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents — Agentic Real2Sim targets per-episode video-conditioned twins with a VLM orchestrator, whereas LiteReality-Agent targets static room-scale scenes from RGB-D scans and uses a mix of a handcrafted graphics-ready pipeline plus targeted agentic passes (Layout Agent, Astra-driven small-object recovery). It also sits close to SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (SAGE) — SAGE composes 3D scenes from a text task specification via a critic-gated LLM loop; LiteReality-Agent is the scan-conditioned counterpart. The GPT-6 small-object recovery is another datapoint in the “frontier general MLLMs sidestepping domain-specific pipelines” pattern also seen in Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions (GPT-6 zero-shot real2sim) and Thomas Ricouard — Astra builds a Blender-to-UE5 walkable house end-to-end via Python 3D-as-code (Astra as end-to-end 3D-as-code agent). The MuJoCo target is notable — the concurrent NVIDIA line (SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation SimFoundry) targets Isaac Lab; convergence on simulator-agnostic output formats is still open.

See also

• Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents — VLM-agent-orchestrated per-episode video→sim twin; sibling agentic-real2sim recipe, video-conditioned rather than RGB-D-conditioned
• SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation — NVIDIA GEAR modular real-to-sim scene builder targeting Isaac Lab; contrasts on simulator target and on hand-composed vs agent-driven scene assembly
• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI — text-conditioned agentic 3D scene composer; the scan-vs-text-input contrast
• Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling — parse→generate→place composable real-to-sim pipeline; alternative decomposition of the same problem
• Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions — GPT-6 zero-shot real2sim from multi-view RGB; the frontier-model analogue of what LiteReality-Agent uses GPT-6 for at the small-object step
• Thomas Ricouard — Astra builds a Blender-to-UE5 walkable house end-to-end via Python 3D-as-code — Astra / GPT-6 as multi-tool 3D-as-code agent driving Blender + UE5; the same frontier-model tool-use pattern
• Synthetic Training Data — LiteReality-Agent as a scan-conditioned room-scale generator of physics-ready 3D environments
• Tool-Use Agents — Layout Agent + Astra small-object pass as domain-specific tool-use agents inside a graphics pipeline
• World Foundation Models — reconstructed simulator scenes as a substrate for policy training / eval alongside WFM rollouts