SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

TL;DR

SimFoundry is NVIDIA GEAR’s modular system for zero-shot real-to-sim scene construction from a single video: it reconstructs a sim-ready digital twin of a real scene, then automatically generates digital cousins — affordance-preserving variations of objects, scene layout, and task — that act as training data for manipulation policies. The headline claim is twofold. (1) As an evaluator, SimFoundry-built sims predict real-world policy performance with mean Pearson r = 0.911 and mean maximum ranking violation (MMRV) of 0.018 over 7 tasks × 5 policy architectures, outperforming the prior SOTA PolaRiS [§Abstract]. (2) As a trainer, sim-only policies trained with object / scene / task cousins transfer zero-shot to real DROID-platform tasks with average success-rate improvements of 17%, 21%, and 40% respectively over a no-cousins baseline [§Abstract]. The pipeline’s modular design — foreground-removal video → 3DGS background, per-object 2D-to-3D mesh generation, physics-validated assembly — means each component swaps in better as foundation models improve.

Key claims

Zero-shot real-to-sim scene construction from a single video produces sim-ready digital twins that support object, scene, and task editing for automated cousin generation [§Abstract].
SimFoundry simulation evaluations strongly predict real-world policy performance: mean Pearson r = 0.911 and mean MMRV = 0.018 across 7 manipulation tasks × 5 policy architectures [§Abstract].
SimFoundry outperforms PolaRiS (Jain et al. 2025) as a sim-to-real performance predictor on the same evaluation suite [Project Page, Predictor Comparison].
Sim-trained policies transferred zero-shot to real-world DROID-platform tasks show +17% / +21% / +40% average success-rate improvements when trained on object / scene / task cousins respectively versus a no-cousins baseline [§Abstract].
SimFoundry recovers more accurate object meshes and poses than SAM3D on occluded, cluttered scenes, and generalizes across input modalities (real videos, open-source datasets, synthetically generated images) [Project Page, Scene Reconstruction].
Background reconstruction uses a foreground-erasure pipeline that produces a clean background video, refined and used to train a 3D Gaussian Splat for high-fidelity background [Project Page, Method].
Modular pipeline composes 2D foundation models (segmentation, depth, 2D-to-3D mesh generation) with physics-validated assembly: per-object segmentation masks + depth → 2D-to-3D mesh generation → annotated physical parameters + sanity check in a physics simulator [Project Page, Method].
Zero-shot transfer to challenging real tasks involves multi-step manipulation, articulated object interaction, and bimanual interaction [§Abstract].

Method

SimFoundry takes a video of a real scene as input and produces a fully-annotated, physics-validated simulation scene in three stages. (1) Foreground / background separation: foreground objects are erased to produce a background-only video, which is refined and used to train a 3D Gaussian Splat for the background. (2) Per-object reconstruction: each foreground object is independently extracted by segmentation + depth estimation; a 2D-to-3D generation model then produces a physics-ready mesh per object. (3) Scene assembly + validation: per-object meshes are placed back into the scene with annotated physical parameters (mass, friction, articulation joints) and the full configuration is sanity-checked in a physics simulator before export.

On top of the base twin, SimFoundry supports three classes of automated augmentation — object cousins (swap object identity while preserving affordances), scene cousins (perturb layout and background while preserving task feasibility), and task cousins (vary the task specification itself) — yielding affordance-preserving variations rather than arbitrary perturbations. The same exported sim is used both as training substrate (for sim-only policy training that transfers to real) and as evaluation substrate (sim rollouts predict real success rates per policy checkpoint).

Because each stage is a swappable foundation-model call (segmentation, depth, 2D-to-3D, physics validation), the pipeline is presented as improving with its components — no architectural redesign needed as upstream models advance.

Results

Predictor accuracy (7 tasks × 5 policy architectures including DreamZero, Gr00t N1.7, π0.5): mean Pearson r = 0.911 between SimFoundry-predicted and real-world success rates; mean MMRV = 0.018 [§Abstract]. Reported to outperform PolaRiS as the prior SOTA predictor on the same suite.
Training transfer (DROID platform, zero-shot real-world deployment of sim-trained policies): +17% average real success-rate improvement from object cousins, +21% from scene cousins, +40% from task cousins, each over the no-cousin baseline [§Abstract].
Reconstruction quality: claims more accurate object meshes and poses than SAM3D, particularly on occluded and cluttered scenes [Project Page, Scene Reconstruction].
Cross-modality input: pipeline accepts real videos, open-source datasets, and synthetically generated images out of the box [Project Page].

Per-task and per-architecture breakdowns, ablations on each cousin axis individually vs combined, and detailed scene-reconstruction metrics are deferred to the paper PDF (HTML preview unavailable at filing time).

Why it’s interesting

SimFoundry is the most concrete instantiation yet of the real-to-sim digital-twin + automated digital-cousin recipe for manipulation policy training and evaluation. Two prior threads on the wiki converge here. First, it stands as the third major position in the “what is simulation for?” debate playing out across the cluster: The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward deliberately uses simulation only for evaluation to keep training and eval distributions decoupled, NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI uses generative video models only for training data (long-tail synthesis), and SimFoundry uses the same physics-validated sim for both — and reports r = 0.911 sim-to-real prediction plus large cousin-driven training gains as evidence that the dual use is consistent. Second, it is the manipulation-domain sibling of SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (SAGE, also NVIDIA): SAGE generates 3D scenes from text via a critic-gated agentic loop, SimFoundry reconstructs them from video; both target the same downstream consumer (IsaacSim-style policy training) and both treat the physics simulator itself as the verification gate. The cousin axis adds an evaluation contribution that the Veo-as-evaluator work Evaluating Gemini Robotics Policies in a Veo World Simulator doesn’t have: Veo predicts policy ranking with image-edited OOD scenes inside a generative model, while SimFoundry predicts ranking with physics-simulated cousin scenes — direct comparison would help resolve whether closed-loop generative WFMs or explicit physics simulators are the better policy-evaluation substrate, and at what cost.