SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

TL;DR

SAGE is an agentic framework that takes a user-specified embodied task (e.g. “pick up a bowl and place it on the table”) and automatically generates a simulation-ready 3D scene to support it. Multiple generators propose layouts and object compositions; multiple critics evaluate semantic plausibility, visual realism, and physical stability; the agent loops until the scene meets both intent and physics constraints. The accompanying SAGE-10k release ships 10,000 diverse interactive scenes across 50 room types plus 565K unique 3D objects — large enough that policies trained purely on SAGE-generated environments show scaling trends and generalize to unseen objects and layouts in IsaacSim. The interesting contribution for Luma is the critic-gated generator loop as a recipe for producing simulation-ready 3D content at dataset scale.

Key claims

SAGE couples multiple generators (layout + object composition) with multiple critics (semantic plausibility, visual realism, physical stability) and iterates with adaptive tool selection until intent + physics validity are met [§Abstract].
The output environments are simulation-ready and directly deployable in modern simulators (IsaacSim shown) for policy training [§Abstract, §Embodied Policy Training].
The SAGE-10k dataset contains 10,000 scenes across 50 room types/styles and 565K uniquely generated 3D objects [§SAGE-10k Dataset].
Policies trained purely on SAGE-generated data exhibit scaling trends and generalize to unseen objects and layouts on pick-and-place and mobile manipulation tasks [§Abstract, §Embodied Policy Training].
The agent supports multi-room scenes (e.g. multi-bedroom apartments, mid-century family homes), image-conditioned generation from reference photos, category-level and layout-level scene augmentation, and articulated objects (open/closed states) [§Generated Multi-Room Scenes, §Image-Conditioned Generated Scenes, §Augmentation of Generated Scenes, §Articulated Objects].
Physical stability of generated scenes is demonstrated against Holodeck and SceneWeaver baselines via simulation-progress rollouts [§Physical Stability of Generated Scenes].
Action data collection is parallelized in IsaacSim (8 envs/GPU for pick-and-place; 2 envs/GPU for mobile manipulation) using 3 perspective cameras for manipulation and 2 fisheye cameras for navigation, with both RGB and depth fed to the policy [§Pick-and-Place Action Generation, §Mobile Manipulation Action Generation].

Method

SAGE is structured as an agentic loop: given a natural-language embodied task, the agent decomposes the requirement, calls layout and object generators to instantiate a candidate scene, then routes the candidate through three classes of critic — semantic (does it match the task intent?), visual (does it look like a real room?), and physical (is it stable under simulation?). Failures from any critic trigger targeted refinement: object replacement, layout perturbation, or full regeneration. The same loop produces single-room scenes (bedrooms, gyms, offices, themed/styled rooms), multi-room apartments, image-conditioned scenes (matching a reference photo), and augmentations of an existing base scene at either object-category or layout granularity.

For downstream policy training, SAGE feeds its generated scenes into IsaacSim and runs parallelized action collection — perspective + wrist cameras for pick-and-place, plus fisheye nav cameras for mobile manipulation — with RGB + depth as policy inputs. Both action generation code and 3D scene generation code are slated for release.

Results

The project page is qualitative; the headline numbers are dataset scale (10k scenes, 565K objects, 50 room types) and the demonstrated policy-training scaling trend. Physical-stability comparisons against Holodeck and SceneWeaver are shown as video rollouts rather than quantitative tables on this landing page. Successful and failure policy-inference cases are shown for both pick-and-place and mobile manipulation. A linked arXiv paper (cited as arXiv:2602.10116, 2026) presumably contains the quantitative scaling curves and ablations.

Why it’s interesting

SAGE is the third filed instance of a critic-gated agentic loop producing training data at scale, after Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (tool-use trajectories, three-role loop) and Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (video CoT traces, hindsight curation). The interesting twist here is that the artifact being generated is a 3D simulation environment — a much higher-dimensional, longer-validity-horizon output than text or tokenized reasoning — which forces the critic stack to span semantic + visual + physical (simulation-rollout) checks. That physics critic is the new ingredient: it can only be implemented by actually running the artifact through a simulator and reading out stability, which makes the “verification gate” of Synthetic Training Data far more expensive but also much harder to fool. For Luma’s interest in scalable 3D content pipelines and synthetic training data for spatial models, the SAGE pattern (multi-critic agentic loop → simulation-ready output) is the cleanest existence proof that the verification-gate recipe extends from text into full 3D scenes. Worth tracking alongside NVIDIA Unveils New Open Models, Data and Tools to Advance AI Across Every Industry (NVIDIA’s Cosmos/Isaac stack, which SAGE outputs feed into) as another instance of NVIDIA’s “open WFM + open dataset + open simulator” play in embodied AI.