Skip to content
Bud's Wiki

Assemble Bench and CG-DAgger — NIST small-parts assembly benchmark on Isaac Lab Arena + code-guided DAgger with zero teleop

TL;DR

Section titled “TL;DR”

@hud_evals (HUD, YC-backed RL-environments-and-evals shop) announces Assemble Bench, a robot-model benchmark built on top of the NIST small-parts assembly standard, plus CG-DAgger (Code-Guided DAgger), a synthetic post-training recipe used to fine-tune inside it. The benchmark runs on the DROID robot platform (Franka arm + Robotiq gripper) in a photorealistic Isaac Lab Arena simulation for real-to-sim transfer, with 4 task families and 14 contact-rich manipulation tasks. Frontier DROID checkpoints (pi0.5-DROID, Cosmos Nano-DROID) fail zero-shot; even after fine-tuning, pi0.5 reaches only 1.7–11.1% success across task suites. CG-DAgger replaces the human in HG-DAgger with code that detects failure plus a scripted expert that demonstrates the recovery — so 1,355 demonstrations were generated with zero teleoperation.

Key claims

Section titled “Key claims”
  • Assemble Bench targets the NIST small-parts assembly standard on DROID (Franka + Robotiq gripper) inside a photorealistic Isaac Lab Arena environment for real-to-sim transfer, with 4 task families totalling 14 contact-rich manipulation tasks [tweet 2/6].
  • Frontier DROID checkpoints fail zero-shot on the benchmark: pi0.5-DROID and Cosmos Nano-DROID both fail; even after fine-tuning, pi0.5 reaches only 1.7% – 11.1% success depending on task suite [tweet 3/6, attached figure].
  • Alignment for these tasks is too fine for teleoperation to produce clean trajectories, so the entire fine-tuning corpus — 1,355 demonstrations — was generated by scripted experts, with zero teleoperation [tweet 4/6].
  • Standard RL algorithms (PPO, GRPO, DPO) are not compatible with most VLA and WAM architectures; model-agnostic alternatives (PA-RL, PLD) were tried and none got far [tweet 5/6].
  • CG-DAgger (Code-Guided DAgger) is proposed as the working recipe: instead of a human in HG-DAgger’s loop, code detects failure, a scripted expert demonstrates the recovery, and those corrections become training data [tweet 5/6].
  • Framing claim: robotics will shift from pre-training to post-training over the next few years, and real progress will require physical RL environments as the substrate [tweet 6/6].

Method

Section titled “Method”

The benchmark itself is a photorealistic Isaac Lab Arena scene of NIST-standard small-parts assembly, exposed via the DROID interface (single Franka arm with a Robotiq parallel-jaw gripper), organized into 4 task families with 14 total tasks. The synthetic-post-training loop is the load-bearing methodological contribution: for each task, a scripted expert (written in a domain-specific language for the assembly primitives) generates a base demonstration; when a rollout fails, code — not a human — detects the failure condition and dispatches the scripted expert to demonstrate the recovery from the current state. The recovery trajectories are appended to the DAgger dataset for the next fine-tuning iteration. All 1,355 demonstrations in the final fine-tuning corpus come from this pipeline, with no teleoperation at any point. Prior attempts at model-agnostic RL adaptation (PA-RL, PLD) are reported as not working; CG-DAgger is the recipe that produced the reported pi0.5 fine-tune numbers.

Results

Section titled “Results”

Zero-shot: pi0.5-DROID and Cosmos Nano-DROID both fail on Assemble Bench (specific numbers not shown in the announcement). Fine-tuned: pi0.5 with the CG-DAgger corpus reaches success rates in the 1.7% – 11.1% range depending on which of the 4 task suites is evaluated, per the attached figure. No numbers are given for the RL baselines that “got nowhere” (PA-RL, PLD). No real-hardware transfer numbers are reported in the thread; the platform is positioned as sim-only with real-to-sim as the design intent.

Why it’s interesting

Section titled “Why it’s interesting”

Assemble Bench is a new datapoint on the RL Environment Platforms board specifically for the contact-rich fine-alignment regime that recent VLA benchmarks (LIBERO, RoboCasa365, RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs), RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies) don’t stress hard — the pi0.5 fine-tune ceiling of 11.1% is the sharpest filed evidence yet that frontier DROID checkpoints have not solved the sub-millimeter-tolerance small-parts assembly regime, complementing the ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm finding that VLAs collapse to ~0% on cable_clip while a from-scratch Diffusion Policy beats them on precise-force tasks. CG-DAgger stakes out a distinct row on the VLA Models recipe-lever board next to FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space‘s FlowDAgger — both are DAgger-style corrective-data pipelines for frozen or fine-tuned VLAs, but FlowDAgger keeps the human and pushes corrections into noise space via action inversion, while CG-DAgger removes the human entirely and generates corrections from scripted experts triggered by code failure detectors. It is also the sharpest filed evidence for the “PPO/GRPO/DPO don’t fit generative VLA action heads” position that π*0.6: a VLA That Learns From Experience (RECAP) built RECAP around — HUD independently tried PA-RL and PLD and reports they got nowhere on this regime.

See also

Section titled “See also”