Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

ListenFixing a Bad Fold at Inference Time10:16

Show notes

A look at SILR, a real-world cloth-manipulation stack that takes a brittle base policy and refines it *at inference time* by rolling out candidate actions inside a fast deformable-object simulator called FLASH. We dig into the three pillars — the sparse-mesh simulator, an RGB-native real-to-sim module built on frozen DINOv2 plus learnable canonical tokens, and prior-guided MPPI — and what the numbers actually show, including the jump from roughly 1-to-3 out of 10 folds to 8-to-9 out of 10. We also draw out the "wrap the policy with test-time compute" parallel for video and world-model folks. **Sources** - Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

SILR is a real-world cloth manipulation stack organized around three coupled pieces: a custom sparse-mesh deformable simulator (FLASH) tuned for stable parallel rollouts, an RGB-native real-to-sim module trained purely on FLASH-synthesized data that lifts a single image to a simulation-compatible cloth mesh state, and a prior-guided MPPI controller that runs inside the simulator at each control step to refine the actions of an offline-distilled base policy. The bet is that inference-time simulator-in-the-loop refinement can carry brittle cloth manipulation policies from base-policy execution (1/10 – 3/10 real success on folding) to 8/10 – 9/10, and that FLASH beats Newton and Isaac Sim as the rollout backend under a shared MPPI harness on both accuracy and speed.

Key claims

Method

At training time, FLASH — a triangular-mesh deformable-object simulator that resolves frictional contact via non-smooth Newton iterations reduced to GPU-friendly sparse operations — generates synthetic (RGB image, deformed 3D mesh vertices) pairs at scale. The real-to-sim module fuses frozen DINOv2 patch features with N learnable canonical tokens (one per mesh vertex), applies self-attention over the tokens to exchange structural context, and decodes each token to its 3D vertex position via a shared MLP. At inference time, the module runs on a single RGB frame in 7–12 ms per frame, producing a simulation-compatible cloth state that seeds parallel FLASH rollouts. Prior-guided MPPI then samples K candidate action sequences around an offline-distilled base-policy trajectory, rolls each out in FLASH, reweights by cost, and executes only the first refined action in a receding-horizon loop.

Results

Under a fair MPPI harness (fixed cost, initial state, seed, cloth asset; per-simulator physics tuning), FLASH at K=64 reaches 100% MPPI success rate on the reported cloth task with 1.61 cm KP error and 3.7 cm² MSE, versus 40% for Newton at K=64 (8.4 cm KP err) and 55% for Isaac Sim at K=64 (9.78 cm KP err). Even scaling Newton and Isaac Sim to K=256/512 doesn’t close the gap on keypoint error.
The RGB-native real-to-sim module attains the lowest reconstruction error on every metric versus point-cloud diffusion (DPM) and depth-based (DeFM) baselines: on the towel asset, 2.33 mm² MSE / 1.94 mm CD / 1.98 mm EMD versus DPM’s 7.17 / 3.89 / 4.01 and DeFM’s 43.36 / 5.51 / 6.53. On long-sleeve shirt, 24.23 mm² MSE versus DPM’s 45.74. Latency is 7.4 ms (towel) / 12.4 ms (shirt) per frame versus 318.5 / 512.9 ms for DPM — ~40× faster.
On real hardware, the full pipeline reaches 9/10 on single-arm diagonal folding and 8/10 on dual-arm symmetric folding, versus 3/10 and 1/10 for the base policy without refinement. Ablations show every component is load-bearing: swapping DPM in for the RGB real-to-sim drops single-arm success to 6/10 and dual-arm to 3/10; swapping Isaac Sim in for FLASH drops to 5/10 and 4/10 respectively.

Why it’s interesting

SILR is a cleanly-articulated counter-recipe to the end-to-end-VLA framing that dominates the recent VLA Models filings: rather than scaling a large pretrained policy to absorb deformable-object physics, it treats the base policy as a nominal trajectory and burns inference-time compute on parallel physics rollouts to fix it up. This is the same “wrap the policy with test-time refinement” pattern Playful Agentic Robot Learning and Learning What to Say to Your VLA: Mostly Harmless VLA Steering argue for on the language-instruction side, but on the action side and with a hand-designed physics simulator rather than a learned world model — a sharper alternative to the ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing? / µ₀ (μ₀: A Scalable 3D Interaction-Trace World Model) camp that reuses generative world models as the rollout backend. The real-to-sim ablation is also the cleanest filed evidence that a frozen DINOv2 + learnable canonical tokens beats point-cloud diffusion on speed and accuracy for structured mesh reconstruction, which is a transferable primitive beyond cloth. Directly answers the Jitendra Malik counter-position (Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level) that CV-heavy robotics work skips the sensorimotor level: SILR’s contribution is precisely that the sensorimotor loop (contact dynamics in FLASH) is where the manipulation lift comes from, not the perception encoder.