SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models

TL;DR

SIMPACT is a test-time, zero-shot framework that augments a frontier VLM (Gemini 2.5 Pro) with a physics simulator constructed from a single RGB-D image so the model can propose action sequences, observe simulated rollouts, and iteratively refine its plan before executing on a real Franka Research 3. The simulator itself is assembled per-scene by prompting the VLM to select segmentation + 3D reconstruction + pose-estimation + physics-engine choices (MuJoCo for rigid bodies, projective-dynamics for stiff deformables, MPM for soft), and the VLM plays three roles in the planner: action-sequence sampler, in-context optimizer over failed rollouts, and success evaluator. On five real-world tasks spanning rigid-body and deformable manipulation (non-toppling push, bowl stacking, pivoting, rope shaping, Play-Doh shaping), SIMPACT reaches 80/60/40/90/80% success vs 0% for π₀.₃ across the board and 0–20% for VoxPoser / MOKA — direct evidence that embedding physics into the VLM reasoning loop at test time closes the physical-grounding gap.

Key claims

A single RGB-D frame is sufficient to construct a scene-specific multi-physics simulator: VLM selects object labels + physics engine, GroundedSAM2 segments, an image-to-3D model reconstructs meshes, FoundationPose gives 6-DoF pose for rigid bodies, and back-projected + volume-sampled particles handle deformables [§3.1].
The VLM performs three distinct roles inside the planner — Sample (hierarchical symbolic actions: MOVE / GRASP / RELEASE + continuous parameters), Optimize (in-context refinement over failed rollout state trajectories), TaskSuccess (evaluation from final simulator observation + state) — with a separate system prompt per role [§3.2].
SIMPACT outperforms all baselines on all five real-world tasks: 80% / 60% / 40% / 90% / 80% on non-toppling push / bowl stacking / pivoting / shape rope / shape Play-Doh, versus 0% across the board for π₀.₃ (VLA baseline), 0–20% for VoxPoser, and 0–20% for MOKA [Table 2].
Ablations show all three components are load-bearing: without VLM sampler (Gaussian sampling instead), success drops to 0–30%; without simulation rollout (proposer-verifier without physics), 0–30%; without VLM optimizer (naive best-of-N over initial proposals), 30–70% [Table 3].
VLM optimization is not restricted to local updates — it reasons over failed attempts and produces successful action sequences through internal reasoning, unlike CEM-style local numerical optimizers [§3.2, Fig. 4].
Hierarchical action generation (symbolic MOVE / GRASP / RELEASE with continuous parameters, deterministically translated to 6-DoF EE trajectories by Action2Pose) is more effective than direct 6-DoF pose prediction, exploiting semantic-action priors in the VLM’s pretraining data [§3.2].
Named failure modes: perception failures dominated by single-view 3D reconstruction errors, planning failures most frequent on the pivoting task, execution failures from sim-to-real dynamics mismatch [§4.4].
Named limitations: single-view reconstruction struggles with occlusion; VLM-estimated physical parameters can deviate from reality; execution is open-loop and susceptible to compounding errors [§4.5].

Method

SIMPACT constructs a physics simulator from a single RGB-D observation via an automated visual-foundation-model pipeline. A VLM enumerates object labels from the user instruction; GroundedSAM2 segments each object; the VLM assigns each object to a physics engine (MuJoCo for rigid bodies, a PyTorch projective-dynamics variant for stiff deformables, an MPM Warp solver for soft objects). For rigid bodies, meshes are reconstructed by a pretrained image-to-3D model, scaled to match the real-world bounding box diagonal, and 6-DoF poses are estimated with FoundationPose. For deformables, surface points from the depth mask are augmented with volume-sampled points bounded by the table plane. Physical parameters (mass, friction, elasticity, plasticity, center of mass) are inferred by the VLM from commonsense reasoning.

The planner is a three-role VLM loop (Alg. 1). Sample generates initial action sequences using a compact hierarchical action vocabulary — high-level symbolic types (MOVE / GRASP / RELEASE) parameterized by continuous variables, deterministically translated to 6-DoF EE trajectories by Action2Pose. Each sequence is rolled out in the simulator to obtain a state trajectory. Optimize takes both prior sequences and their rollouts as in-context evidence — subsampled per-timestep rendered simulator observations plus numerical action + state — and produces a refined sequence, reasoning over failure modes rather than performing local perturbation. TaskSuccess evaluates the final rollout state via a rendered image + simulator state; if the VLM judges success, the sequence executes on the real robot. Otherwise the loop iterates up to a maximum bound (10 initial proposals, 5 optimization iterations in the paper’s default). The default VLM is Gemini 2.5 Pro across all three roles.

Results

Real-world success across five tasks (Franka Research 3, RealSense D435i single-view RGB-D):

Task	π₀.₃	VoxPoser	MOKA	SIMPACT
Non-toppling push	0%	0%	0%	80%
Bowl stacking	0%	20%	10%	60%
Pivoting	0%	0%	0%	40%
Shape rope	0%	0%	20%	90%
Shape Play-Doh	0%	0%	0%	80%

π₀.₃ generates actions that approach the target but fails to complete manipulation, confirming that VLAs generalize poorly to out-of-domain contact-rich tasks not seen during training [§4.2, Table 2].

Ablations across the same five tasks: removing the VLM sampler yields 0/10/0/0/30%, removing simulation rollout yields 20/0/0/30/30%, removing the VLM optimizer yields 30/50/30/40/70% — full SIMPACT wins across the board and the largest gaps are on non-toppling push and shape rope where initial proposals need iterative refinement [Table 3].
Fig. 4 documents the VLM optimizer learning from failed attempts: initial pushes near the carton top cause toppling in simulation; the VLM identifies this in-context and produces a stable low-push action that succeeds [§4.2].
Failure analysis: perception failures mainly from occluded single-view reconstruction; planning failures peak on pivoting (dominant regime) even after multiple rounds; execution failures from kinematic / dynamic sim-to-real mismatch [Fig. 7, §4.4].

Why it’s interesting

SIMPACT is the sharpest published instance of the Simulator-in-the-Loop Refinement pattern for VLM-driven manipulation, and it slots directly next to Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement SILR and DriftWorld: Fast World Modeling through Drifting DriftWorld on the concept’s design-axis split: SILR uses a custom cloth simulator with MPPI; DriftWorld uses a learned drifting video WFM with reward-based rollout selection; SIMPACT uses a VLM-authored multi-physics simulator with VLM-based in-context optimization over failed rollouts. The load-bearing difference from CEM-style MPC or Prompting-with-the-Future (which uses rigid-body sim + VLM as reward only) is that the same VLM does sampling, optimization, and evaluation — the optimizer can perform structural reasoning over failure modes rather than being confined to local action perturbations, and the paper’s ablations isolate this VLM-optimizer role as worth 20-50 pp on tasks that need iterative refinement.

The framing is a companion piece to Guava: An Effective and Universal Harness for Embodied Manipulation Guava from the same first author (Haowen Liu) — Guava distills a frontier VLM’s harness behavior into a compact 4B model that runs autonomously, while SIMPACT keeps the frontier VLM in the loop and hands it a physics simulator as the reasoning substrate. Both papers stake out the “no VLA, physics-grounded VLM” cell of the VLA Models recipe-lever board and both use the Tool-Use Agents ReAct pattern, but they differ on where the physical understanding lives: in the model’s post-trained weights (Guava) or in an externally-constructed simulator queried at test time (SIMPACT). The 0/0/0/0/0% π₀.₃ column is the sharpest quantitative evidence to date that current large-scale VLAs fail on precise contact-rich tasks outside their training distribution — direct empirical support for the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position that VLA scaling has not solved contact-establishment tasks.