Skip to content
Bud's Wiki

Simulator-in-the-Loop Refinement

TL;DR

Section titled “TL;DR”

Robot policies (or VLM planners) that at inference time — not just training time — refine each candidate action by rolling it out in a physics simulator or learned world model, then re-selecting the action using cost, reward, or in-context VLM evaluation. The pattern lifts a brittle base proposal (a distilled base policy, a VLM’s first-pass symbolic plan, or nothing at all) into robust real-world behavior on contact-rich or physically-underdetermined tasks that end-to-end VLA scaling has repeatedly failed on. Distinct from end-to-end VLA training (which pushes physics understanding into weights), from test-time gradient guidance of the policy itself (which stays in one forward-pass regime), and from RL-environment platforms (which are sim-for-training rather than sim-for-control). Load-bearing design axes: (a) the rollout backend — hand-built physics simulator vs learned drifting video WFM vs VLM-authored multi-physics simulator; (b) the selection mechanism — sampling-based MPC (MPPI) vs best-of-N with a reward function vs VLM in-context optimization over failure modes; (c) whether there is a base policy being refined or whether the sampler is the planner.

Key claims

Section titled “Key claims”
  • Inference-time simulator-in-the-loop refinement can carry a brittle base policy from 10-30% real-world success to 80-90% on tasks where end-to-end scaling has failed: SILR reports 1/10 → 9/10 on single-arm diagonal cloth folding and 1/10 → 8/10 on dual-arm symmetric folding by wrapping an offline-distilled base policy with prior-guided MPPI over parallel FLASH simulator rollouts (Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement Results, real-hardware).
  • The rollout backend is a first-class design decision, not a swappable primitive: SILR’s ablation shows FLASH beats Newton (40%) and Isaac Sim (55%) at K=64 on the same MPPI harness with 100% simulator-side success and 1.61 cm keypoint error, and scaling Newton or Isaac Sim to K=256 or 512 does not close the gap (Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement MPPI-harness table). DriftWorld’s answer is orthogonal — swap the physics simulator for a one-step drifting video WFM to unlock 30+ FPS rollout on a single H100 at 17× the throughput of diffusion-based WFM baselines (DriftWorld: Fast World Modeling through Drifting Abstract, per-task FPS: Bridge-V2 33.3, RT-1 38.8, Language Table 36.6, Robomimic 100.0, Push-T 270.3).
  • The selection mechanism splits along a reasoning-vs-search axis: SILR uses classical sampling-based MPC (MPPI reweighting), DriftWorld uses simple best-of-N reward-based rollout selection, and SIMPACT introduces VLM in-context optimization where the same VLM that sampled the action sequences also reasons over the failed rollout state trajectories to propose refined sequences — the ablation isolates this VLM-optimizer role as worth 20-50 pp on tasks that need iterative refinement (SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models Table 3, §3.2, Fig. 4).
  • Simulator-in-the-loop refinement is a direct empirical answer to end-to-end VLA failure on contact-rich out-of-distribution tasks: SIMPACT reports 0% success across five real-world tasks (non-toppling push, bowl stacking, pivoting, shape rope, shape Play-Doh) for π₀.₃, versus 60-90% for the full SIMPACT loop with the same hardware and single RGB-D observation (SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models Table 2). Consistent with SILR’s per-hardware finding that base-policy execution reaches only 10-30% real-world success on cloth folding before refinement (Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement Results).
  • The real-to-sim front-end is load-bearing when the world model is a physics simulator: SILR’s RGB-native DINOv2 + canonical-token real-to-sim module (7-12 ms per frame) beats point-cloud diffusion by ~40× on latency and achieves the lowest reconstruction error on every metric on both towel and shirt assets (Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement real-to-sim table). SIMPACT delegates this to a VLM-orchestrated stack (GroundedSAM2 + image-to-3D + FoundationPose) and reports single-view reconstruction quality as the dominant perception failure mode (SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models §4.4).
  • The rollout-vs-ground-truth correlation claim for learned WFMs is a rank-level claim, not a dynamics-preservation claim: DriftWorld reports Pearson r = 0.9916 on Robomimic Lift, 0.9250 on Robomimic Can, and 0.9515 on Push-T between simulator-inside policy scores and real-world ground truth (DriftWorld: Fast World Modeling through Drifting Policy Evaluation), similar to RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation RoboWorld’s r = 0.989 on the same evaluation axis. Whether the same learned WFM would preserve causal action-observation dynamics closely enough for in-loop control (not just offline ranking) is an open question the current filings do not settle.

Recent contributions

Section titled “Recent contributions”
  • [2026-11-03] Do robot policies transfer between physics engines?: Manda Robotics’ cross-engine study bounds how trustworthy any inference-time sim refinement loop can be when solver settings are uncontrolled: a single PhysX solver swap (TGS 100/1 → PGS, both read back from the live stage) moves the worst insertion policy from 5/25 to 21/25 success at Holm-corrected p=0.001 without touching weights. If a sim-in-the-loop refiner silently selects between candidate rollouts on an engine configured one way vs another, the ranking can flip entirely. Also provides a concrete negative-transfer envelope for refiners whose candidate rollouts leave the training engine: validated Isaac Lab grasp-and-lift recipes still lose 20–29 points on backend swap, and MuJoCo’s contact time constant is the one knob whose variation reproduces PhysX’s failure mode on specific policies.
  • [2026-11-02] 3D World Simulator: Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation: 3D World Simulator extends the learned-WFM cell staked out by DriftWorld with an explicit 3D latent and co-generated tactile: §6 reports qualitative policy-in-the-model evaluation on Push-T, cloth folding, and plug insertion where the predicted rollout is detailed enough to distinguish success from failure (two policies compared from the same plug-insertion start). §7 pushes further than DriftWorld or SILR by claiming policies trained purely on model-generated data run on real robots (shown at 20× speed), making this the first filed instance of a learned WFM serving as both an in-loop evaluator and a training-data generator for real-robot transfer in one system. No quantitative success rates or correlation numbers reported.
  • [2026-10-28] Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents: RPG sits adjacent to this concept on the “sim is in the loop” axis but amortizes the sim work across the training-time self-improvement loop rather than running sim rollouts per episode at inference time. Like Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation Recova, RPG uses the simulator at training time to diagnose failures and author corrective artifacts (symbolic skills + system-prompt edits), then deploys a frozen artifact at test time with the multimodal LLM coordinating perception + control. Distinct from SILR / DriftWorld / SIMPACT (per-episode inference-time refinement) in loop timing, and distinct from Recova in that the authored artifacts are a persistent skill library + prompt rather than per-capability task/recovery policy pairs. 30/30 real-robot success across three tasks after calibration — concrete datapoint for the “amortize sim work across capability authoring” pole of the design space.
  • [2026-10-27] AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra: AHa-3D (Xu, Bian, Gao) is the scene-reconstruction-and-animation cell of the agent-authored-Real2Sim pattern: GPT-6 Astra orchestrates Pi3X / SAM3 / GVHMR / Kimodo as a 3D vision tool library to turn an indoor room video into an editable Blender scene with reusable furniture + materials, estimated per-person motion, foot-ground and object contact refinement, and video re-rendered at the original camera and timing. Unlike SIMPACT and RoboSimStudio, the downstream payload is video re-rendering + new-action generation rather than control — same agent-authors-the-world cell, different consumer. LiteReality-Agent (NeurIPS 2026) is the formal benchmark; the blog is the system write-up.
  • [2026-10-26] Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation: Recova (NVIDIA + UCSD + UT Austin) sits adjacent to this concept on the “agent authors programs in a physics simulator” axis but amortizes the authoring work across deployments: the reconstructed digital twin is used at training time to let the agent diagnose failures, write corrective programs, and collect twin-authored rollouts for separately-trained task and recovery policies, rather than at inference time per episode like SILR, DriftWorld, or SIMPACT. Closest neighbor is SIMPACT (same VLM-authors-simulator shape) but with the loop timing flipped from inference-time per episode to training-time once per capability. Reports 78.8% / 64.9% LIBERO-Pro / MolmoSpaces vs 71.7% / 38.0% baselines, DAgger 23.8% → 77.5% → 87.5% with recoveries, and 87.5% → 0% human-intervention rate over four collection rounds — data for whether twin-authored training-time refinement amortizes better than per-episode inference-time refinement on the same contact-rich-failure regime.
  • [2026-10-15] RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents: RoboSimStudio (UC Berkeley + NVIDIA) extends the “agent-authored multi-physics simulator” cell staked out by SIMPACT by moving the authoring agent from a VLM to a frontier coding agent (Claude Code / Codex) and inverting the loop timing — the agent-authored Newton rigid + SuperDex FEM simulation is used at training time to produce varied rollouts that update a Graph-as-Policy (GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks) offline, rather than at inference time for action refinement. Project-page teaser only; no quantitative sim-to-real numbers disclosed yet.
  • [2026-10-13] Gemini 4 Argon for Robotics: Argon implements “build the digital twin, practice there, then run on real hardware” as a reusable pattern: for Spot’s foot-pedal task, Argon constructs an articulated MuJoCo digital twin of the pedal bin first and practices before stepping on the physical pedal; for the digital-twin-from-teleop prompt, it iteratively tunes static object assets + offline-planned robot commands (±0.55 rad arm / ±0.65 rad hand deviation ceiling) under strict no-sim-cheating constraints (object motion only via mj_step contact, no welds, no feedforward torques, no qpos edits, no rolled-back state) until MuJoCo rendering matches real head/left-wrist/right-wrist cameras and simulated fingertip force/torque matches logged tactile in timing/magnitude/direction.
  • [2026-10-07] FLASH: Fast Learning via GPU-Accelerated Simulation for High-Fidelity Deformable Manipulation in Minutes: FLASH (Luo, Zhou, Zhang, Liu, Huang, Zhou, Han, Hu, Yang, Yu, Zeng, Shi) is the simulator-side companion to SILR — the full technical write-up of the GPU-native deformable-object physics engine that SILR used as its MPPI rollout backend. Core design move is an inertia-dominated approximation to the non-smooth Newton contact system (Eq. 10) that keeps the Schur complement sparse under block-diagonal multi-env assembly, so the dense coupling that bottlenecks fully-implicit contact solvers at parallel-env scale is eliminated. Scales to 3M+ DoF at 30 FPS on a single RTX 5090. Validates the full analytic-physics-simulator cell of the design space end-to-end: zero-demonstration pure-sim training in 50–600 min on a single GPU transfers zero-shot to 85.8% (91/106) over a one-hour continuous AdamU dual-arm towel-folding evaluation. Sharpest datapoint for why the “specialized hand-built physics simulator” cell wins on deformable contact-rich manipulation specifically, where general-purpose GPU simulators (Newton VBD, Isaac Sim FEM, Genesis PBD) have known fidelity gaps.
  • [2026-10-02] EMPIRIC — agents that write physics-engine extensions as code to learn unknown dynamics: EMPIRIC (Yichao Liang) adds a fourth cell to the rollout-backend axis: keep a conventional physics engine fixed and have the agent write only the missing physics as Python subclass code (forces, constraints, hidden state like glue cure time), with explicit Bayesian posterior over the program’s parameters. 25/25 across 5 simulated domains × 5 seeds matches oracle dynamics, vs 14-16/25 for WorldCoder-/SimFoundry-/direct-coding-agent baselines on the same Claude Opus 5 backbone; sim-to-real transfer on a real robot infers a decaying post-fan wind force and two domino masses from two gusts (predicted 10.1 ± 2.3 cm vs observed 9.9 cm knock distance). Sharpest datapoint on the “write physics deltas as code with uncertainty” vs “regenerate the whole model” split.
  • [2026-09-30] SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models: SIMPACT (Liu, Yao, Chen, Gao, Mao, Huang, Du) is the first filed instance of the “VLM authors the simulator + the same VLM reasons over rollouts” cell of the design space — VLM prompts choose per-object physics engine (MuJoCo for rigid, projective-dynamics for stiff deformable, MPM for soft), VLM samples hierarchical symbolic actions (MOVE / GRASP / RELEASE + continuous params), VLM optimizes over failed rollouts by reasoning over the state trajectory, VLM evaluates the final rollout. Ablations: without VLM sampler 0-30%, without simulation rollout 0-30%, without VLM optimizer 30-70%, full SIMPACT 40-90% across five contact-rich real-world tasks. Sharpest available datapoint on the “sampling-based MPC vs VLM in-context reasoning” split on the selection-mechanism axis — the VLM optimizer is not restricted to local action perturbations and can propose structural corrections after observing failure modes (Fig. 4).
  • [2026-07-21] DriftWorld: Fast World Modeling through Drifting: DriftWorld (Yilun Du et al.) stakes out the learned-WFM cell of the design space — one-step drifting video WFM at 30+ FPS enables MPC-style action search with dozens of parallel rollouts inside the control loop, without paying the multi-step-denoising cost that keeps diffusion WFMs from serving as fast policy rollout backends. Doubles as an offline policy evaluator with rank correlation to ground truth up to 0.9916 on Robomimic Lift, connecting simulator-in-the-loop refinement to the fast-WFM-as-evaluator thread.
  • [2026-07-02] Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement: SILR (Fan Shi group) is the analytic-physics-simulator cell of the design space — custom sparse-mesh cloth simulator (FLASH) tuned for stable parallel MPPI rollouts, RGB-native real-to-sim module trained entirely on FLASH-generated synthetic data (7-12 ms/frame, ~40× faster than DPM), prior-guided MPPI around an offline-distilled base policy. First filed evidence that inference-time simulator-in-the-loop refinement can carry cloth manipulation from 1-3/10 to 8-9/10 real-world success on the same hardware.

Open questions

Section titled “Open questions”
  • Is the load-bearing lever the rollout backend (analytic physics vs learned dynamics vs VLM-authored simulator) or the selection mechanism (MPPI vs best-of-N vs VLM in-context reasoning)? SILR’s MPPI + FLASH and SIMPACT’s VLM-optimizer + VLM-authored simulator are two-variable-swaps apart, so no filed paper isolates the two axes cleanly.
  • Does the real-to-sim / observation-to-latent-state module generalize across manipulation domains, or does each new physical regime (cloth, rope, dough, rigid contact) require a domain-specific reconstruction pipeline? SILR’s DINOv2 + canonical-token approach is cloth-specific; SIMPACT relies on a general VFM stack but reports single-view reconstruction as its dominant perception failure mode.
  • For learned WFMs (DriftWorld): does the drifting formulation actually preserve causal action-observation dynamics closely enough for in-loop control, or does it just produce visually-plausible rollouts that happen to correlate at rank level with ground truth on offline evaluation? The 0.99 correlation is a rank-level claim; RoboWorld’s Step Forcing was designed to address the dynamics-preservation failure mode that this framing does not test.
  • What is the relationship between simulator-in-the-loop refinement and end-to-end VLA scaling? The current filings all report large lifts over VLA baselines on contact-rich tasks (SIMPACT 0-90% delta vs π₀.₃, SILR 1-9/10 delta vs distilled base policy), but no paper measures whether the same simulator-in-the-loop lift persists as the base VLA scales up along the VLA Models recipe-lever axes (action pretraining, clean teleop, unified-VLM pointing, WFM+action-expert).
  • Is there a natural extension of the pattern from single-episode planning to persistent skill acquisition — i.e. does the refined action sequence get folded back into the base policy via imitation, closing an outer loop, or does each episode start fresh?

Papers

Section titled “Papers”

2025-12

Section titled “2025-12”

2026-07

Section titled “2026-07”