Evaluating a 3D Agent — Moonlake's Sim-Ready Asset Generator vs Astra, Lightwheel, Palatial, Tripo, and Humans

ListenThe Asset That Won't Let a Policy Cheat10:55

Show notes

Moonlake's "3D Agent" generates sim-ready articulated assets tuned for *physical* accuracy — collision geometry, masses, friction, joint dynamics — not just looks. In a 10-asset benchmark it posts a 65.2 rank score and 14 of 22 first-place slots against Astra, Lightwheel, Palatial, Tripo, and human artists, with a standout move: validating every asset with an idealized gripper in the target simulator and explicitly designing out the shortcuts a policy might exploit. We dig into what the metrics measure, which wins look real, and the caveats the post flags itself. Sources - Evaluating a 3D Agent — Moonlake's Sim-Ready Asset Generator vs Astra, Lightwheel, Palatial, Tripo, and Humans (The Moonlake Team)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Moonlake positions its “3D Agent” as a fully-automatic sim-ready asset generator that targets physical accuracy — collision geometry, part masses, friction, joint stiffness/damping, articulation dependency chains — rather than visual fidelity alone, and benchmarks it on 10 moderate-complexity articulated assets against Astra (frontier coding agent), Lightwheel, Palatial, Tripo, and human technical artists. Across 22 physics-authoring / articulation / render-geometry / collision-proxy / UV / materials metrics the agent takes a 65.2/100 rank score vs 47.0 (Astra), 40.9 (Lightwheel), 47.0 (Palatial), winning 14/22 first-place slots. The explicit framing is that a good training asset must preserve functional affordances, support idealized-gripper integration tests in the target simulator, and reject the “simulation shortcuts” that let policies succeed in sim for wrong reasons.

Key claims

Functional-affordance preservation is treated as a requirement (not a nice-to-have): visually-minor features that carry function — pin that reveals hinge location, graspable pot-lid rim, groove indicating force application point, state-indicator light — must be preserved; decorative detail is “as practical” [§Perception].
Validation is executed as integration tests in the target simulator with an idealized gripper, isolating asset quality from robot embodiment / planning / control: a gear train must mesh and produce the planned motion relationship, a folding multi-tool must fold and nest, buttons must traverse the correct path and produce the intended state transition [§Operation].
A “good” digital twin must distinguish a good policy from a bad one: simulation shortcuts (e.g. a plug seating despite off-axis insertion because collision geometry is too permissive) let a policy succeed for the wrong reasons; the agent is designed to anticipate the hacky uses a policy might attempt and close those loopholes [§Learning signal].
Assets are shipped as layered USD (geometry, articulation, collision representation, materials, physical properties, behavioral logic as inspectable layers) plus reproducible validation scripts, so the asset is an editable simulation object with evidence that it passes the author’s intended tests — the deliverable is “tested, editable simulation object together with evidence” rather than a 3D model [§Editability, §Reproducible validation].
Rank-score summary across 22 metrics: Moonlake 65.2, Palatial 47.0, Astra 47.0, Lightwheel 40.9; first-place finishes 14 / 7 / 11 / 6 respectively (Palatial result computed on the 5/10 assets where generation did not fail) [§Summary].
Physics-authoring: Moonlake is first on assets-with-complete-joint-limits-and-drives (100% vs 90%/87.5%/0% for Astra/Lightwheel/Palatial), and only method with meaningful velocity-limit coverage (90.1% vs 17.3%/0%/100%) and explicit armature coverage (42.6% vs 10%/0%/0%) [§Physics authoring].
Collision-proxy accuracy p95 / bounding-diagonal: Moonlake 0.0012 vs Astra 0.0108 / Lightwheel 0.0033 / Palatial 0.0195, and Moonlake is the only system that assigns SDF colliders where multipoint-contact concave surfaces require them [§Collision Proxies, §Collision proxy metrics].
Functional insertion test (plug ↔ socket in Isaac Sim): Moonlake plug inserts and withdraws under contact dynamics; Lightwheel’s convex-hull colliders block the socket recesses; Tripo’s socket has shallow recesses so the plug tilts; Astra plug seats but render mesh fails to display cleanly in Isaac Sim (vision-policy liability); Palatial failed to generate [§Functional Interactions].
Button-press interaction test under matched Isaac Sim conditions (door fixed, gravity disabled, identical light press): only Moonlake reaches full 1.5 mm travel; Astra under-travels, Lightwheel’s collisions block the press and there is no return-spring force, Palatial has no independent button joint [§Physical Parameters].
Baked-lighting rate (AI-reviewed, lower is better): Moonlake 0.0%, Astra 0.0%, Lightwheel 50.0%, Palatial 40.0%, Tripo 50.0%, Human 30.0% — material maps that bake lighting into albedo degrade later relighting / vision-policy transfer [§Materials].
UV quality: Moonlake first on seams-on-sharp-edges (80.1% vs 14.9%–41.6% for others and 35.9% for Human) and lowest seam color discontinuity p95 (0.83 vs 2.59–13.71 for competitors) [§UVs].

Method

The post describes a per-asset agentic pipeline that spans the whole asset stack (modeling, shading, physics, articulation, collision authoring, materials, UV mapping) and runs closed-loop testing suites at authoring time. Assets are authored at variable effort tiers — simple assets finish in minutes, difficult assets (e.g. a turbojet engine) can take hours — because the agent repeatedly runs integration tests in the target simulator and iterates. The validation substrate is an idealized-gripper rig in the destination simulator (Isaac Sim in the shown comparisons) so asset quality is isolated from robot-side perception / planning / control. The agent is described as “lifelong-learning”: prior-generation expert feedback is folded back into the agent’s knowledge so capability scales both with foundation-model competence and with the harness’s own improvements.

Delivery format is layered OpenUSD with the agent’s test scripts included — the explicit stance is that the deliverable is a tested editable simulation object rather than a mesh, so when a downstream RL policy discovers a new shortcut, the specific collision layer / joint drive / articulation chain can be revised without rebuilding the asset.

Results

Benchmark: 10 moderate-complexity articulated assets (plug/socket, pliers, button-door, wheelchair with tire+handrim, folding multi-tool, etc.) across five axes — collision proxies, physical parameters, functional interactions, editable geometry & materials, precise articulation, correct part decomposition, UV maps — plus 22 automated quantitative metrics grouped into physics-authoring (9), articulation (2), render-geometry (6), collision-proxy (3 scored), UV (2 scored), and materials (2 scored).

Overall rank score (out of 100): Moonlake 65.2 / Palatial 47.0 / Astra 47.0 / Lightwheel 40.9 [§Summary].
First-place finishes: Moonlake 14/22, Palatial 11/22, Astra 7/22, Lightwheel 6/22 [§Summary].
Collision proxy accuracy (p95 / bbox): Moonlake 0.0012 — ~2.8× sharper than Lightwheel (0.0033), ~9× sharper than Astra (0.0108), ~16× sharper than Palatial (0.0195) [§Collision proxy metrics].
Joint authoring: Moonlake 100% assets with complete limits+drives, 90.1% joints with explicit velocity limits — the only system with non-trivial velocity-limit coverage [§Physics authoring].
Mass-accuracy: exported total masses closer to real-product references than Astra / Lightwheel / Palatial on the compared assets (per-row tables with “Generation Failed” entries for competitors) [§Physical Parameters].
Interactions: Moonlake plug seats-and-withdraws; Lightwheel plug blocked by convex-hull socket colliders; Tripo plug tilts at shallow socket; Astra plug seats but render mesh breaks in Isaac Sim; Palatial failed [§Functional Interactions]. On a button-press test only Moonlake reaches full 1.5 mm travel [§Physical Parameters].
Baked-lighting (lower is better): Moonlake 0% / Astra 0% / Lightwheel 50% / Palatial 40% / Tripo 50% / Human 30% [§Materials].
UV mapping: Moonlake is the only method producing sensible UV-space textures across the test set; seams-on-sharp-edges 80.1% vs 14.9–41.6% for competitors and 35.9% for Human [§UVs].
Cost / speed: Moonlake is faster and cheaper than human simulation content engineers and reports higher quality than frontier agentic workflows at a fraction of the cost (no absolute $/time numbers disclosed in the post) [§Quality vs Cost].

Two caveats on the quantitative benchmark: (i) Palatial’s rank-score and tables are computed on the 5/10 assets that didn’t fail generation, (ii) the post explicitly notes these are proxy metrics — “stay tuned” for a correlation of metric success with policy training/eval success is promised but not yet delivered.

Why it’s interesting

This is the clearest head-to-head yet on the per-asset-physical-accuracy slice of the sim-ready pipeline that Introducing SimReadyGen — Agentic Simulation Generation for Physical AI staked out (“measured physics, not estimates”) — Moonlake directly measures against Lightwheel on the exact axes Lightwheel’s launch argued were load-bearing (collision authoring, joint drives, friction coefficients, articulation accuracy) and shows a sharp win on collision-proxy accuracy and joint-authoring completeness, while Lightwheel wins on some physics coverage dimensions (joint friction, drive damping). The result sharpens the “aligned-twin as reusable infrastructure” position developed on Synthetic Training Data by Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies (R2S2R) and SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation (SimFoundry) — namely that scene-composition and video-to-twin reconstruction aren’t the whole story; the assets inside the scene need per-part collision / articulation / friction fidelity or the resulting policy will exploit shortcuts.

The idealized-gripper validation in the destination simulator move is the methodological piece worth watching: it provides a crisp, embodiment-agnostic criterion (“does the asset admit the intended interaction when operated by a perfect controller?”) that neither pure visual-fidelity benchmarks (Tripo-style mesh generation) nor pure physics-coverage checklists (percent of joints with authored drives) capture. Pairing it with the explicit “distinguish a good policy from a bad policy” framing closes a loop that most synthetic-data recipes on the wiki leave open — Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies R2S2R, The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward Genesis World, and SimFoundry all argue the twin should be usable for policy evaluation, but Moonlake is the first filed artifact that operationalizes “the asset should not reward wrong policies” as a design criterion with a concrete test rig.

Also notable: the comparison includes a frontier coding agent (Astra) as a baseline on this task, which fits the broader “coding agents eat specialized tools” pattern on Agentic Software Engineering but shows a specific failure mode (Astra plug seats correctly but render mesh fails in Isaac Sim — fine for pure-physics eval, breaks any vision-conditioned policy). Suggests the asset-authoring stack has non-trivial multi-objective structure that generic coding agents don’t yet get right across all target simulators and renderers.