Skip to content
Bud's Wiki

Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts

TL;DR

Section titled “TL;DR”

Bronars, Park, Kedia & Agrawal (MIT) train assembly policies that observe only joint positions, velocities, and recent actions — no vision, no F/T, no tactile, no joint-torque sensing — and show these blind proprioception-only policies, trained purely in simulation with PPO over randomized parts/grasps/poses, solve peg insertion (zero-shot to real at 73% across 8 peg shapes/sizes including 0.5 mm clearance), gear meshing, and nut threading. Matched visual-policy controls in simulation show blind policies generalize better to unseen part shapes and sizes while being somewhat slower (median 0.76 s vs 0.44 s for the vision-equipped baseline when the latter works). The paper also releases mjfit, an open-source procedural generator that builds MuJoCo assembly scenes from exact convex pieces with clearance preserved — reportedly more accurate and faster than off-the-shelf convex decomposition (V-HACD, CoACD) or MuJoCo’s SDF collision for tight-clearance parts.

Key claims

Section titled “Key claims”
  • Blind policies — joint position + velocity + recent-action history only — learn peg insertion, gear meshing, and nut threading in simulation and reach near-perfect success, matching matched visual-policy baselines trained the same way [§Abstract, Fig. 2].
  • The peg insertion policy transfers zero-shot to the real world and inserts pegs of different shapes and sizes in 73% of trials across 8 pegs including three never seen during training (0.5 mm tighter clearance, larger peg, taller handle) [§Abstract].
  • On unseen part shapes and sizes, blind policies generalize substantially better than matched visual policies with a wrist camera that were trained the same way; both are reliable on parts resembling the training distribution [§Abstract, Fig. 6].
  • Vision costs less, not more, when it works: median completion time 0.44 s with vision vs 0.76 s blind (vision sees the hole and goes straight in; blind policy must find it by touch first) [Fig. 7].
  • A linear probe fit on the blind policy’s latent in simulation and applied unchanged to real-robot rollouts recovers interpretable task-relevant features — evidence the policy reasons through contact rather than finding an open-loop mechanical pattern that happens to work [Fig. 5].
  • mjfit procedural generator builds assembly scenes from exact convex pieces for MuJoCo at both higher accuracy and higher speed than V-HACD / CoACD convex decomposition (which fills tight clearances so parts jam) or MuJoCo’s SDF collision (slow and hard to tune) [§Abstract, released at github.com/younghyopark/mjfit].

Method

Section titled “Method”

The policy is a small history-conditioned network that takes only a short history of joint positions, joint velocities, and its own recent actions — no visual, force, torque, or tactile input, no privileged state (fixture pose, grasped-part pose, grasp parameters are all unknown to the policy). Training is pure RL in simulation with randomization over part shapes/sizes, grasps (substantially randomized within the gripper), fixture poses, and physics. Downstream of a higher-level system or VLA that gets the grasped part roughly near the hole, the blind policy takes over for the contact-rich “last mile” from “roughly there” to “assembled.”

The MuJoCo simulation uses the authors’ mjfit procedural asset generator: parts are built from exact convex pieces that keep the designed clearance intact at collision time, so a 0.5 mm clearance remains 0.5 mm after import. Off-the-shelf convex-decomposition pipelines (V-HACD, CoACD) fill the clearance with convex hull overshoot and cause tight parts to jam; MuJoCo’s SDF collision keeps the geometry but is slow and difficult to tune. mjfit sidesteps both failure modes.

Three task families are learned under the same framework: peg insertion, gear meshing, and nut threading. For deployment, the peg insertion policy is run zero-shot on the real robot across 8 physical pegs — 5 from the training distribution and 3 held out (tighter 0.5 mm clearance, larger peg, taller handle).

Results

Section titled “Results”

Simulation: blind policies reach near-perfect success on all three tasks (peg / gear / nut), matching visual-policy baselines trained with the same recipe but with a wrist camera.

Real-world peg insertion: 73% success across 8 pegs with the single policy; held-out pegs (including the tighter clearance) are included in the 73%.

Vision-vs-blind generalization: both reliable on parts similar to training; on unseen part shapes and sizes, visual policies fail far more often than blind ones [Fig. 6]. Vision’s advantage is speed when it works — median 0.44 s to finish with a camera, 0.76 s blind (the blind policy locates the hole by touch first).

Linear-probe diagnostic: a probe fit on the policy latent in simulation and used unchanged on real rollouts recovers task-relevant features — the paper’s argument against an “open-loop mechanical pattern” explanation and in favor of real contact reasoning.

Why it’s interesting

Section titled “Why it’s interesting”

This is a counter-position paper to the sensorimotor framing most of the VLA Models cluster has been consolidating around. The Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level Malik thread and the recent tactile-rich recipes (Tactile-Reactive Dexterous Hand: High-Frequency Physical Interaction Tactile-Rex, VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation VTAP, FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT, ReForce — force-aware retargeting via closed-loop residual force tracker ReForce) argue instrumented tactile / force sensing is what current VLAs are missing for contact-rich tasks like 0.57 mm DIMM insertion. Bronars et al. argue the opposite narrow point: joint sensing itself is a high-resolution contact sensor that gets you to 73% zero-shot on multi-mm-clearance peg insertion with no F/T, no tactile, no vision — and in fact generalizes better than vision to unseen parts because vision overfits to appearance. The sharpest comparison on file is to A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control SGS: both are pure-sim PPO sim-to-real distillation pipelines targeting NIST-style assembly without any real-world data or human demonstrations, but SGS distills into an RGB-conditioned student and gets 6/20 NIST board assets; Bronars et al. drop vision entirely and get 73% on 8 pegs — making this the sharpest “no vision, no demos, no F/T” datapoint on the recipe-lever board. Sibling in sim-to-real deformable-manipulation lineage to FLASH: Fast Learning via GPU-Accelerated Simulation for High-Fidelity Deformable Manipulation in Minutes FLASH (vision+proprioception student, 85.8% real-world cloth folding, no real data) but with proprioception alone as the input and tight-clearance rigid assembly as the target. The mjfit release is also independently valuable — it addresses the specific “exact-convex-decomposition for tight-clearance assembly simulation” gap that Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks Tactile Genesis and Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map Tacmap do not solve.

See also

Section titled “See also”