Skip to content
Bud's Wiki

ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm

TL;DR

Section titled “TL;DR”

ArmnetBench v0.1 is a real-world manipulation benchmark from the Armnet team that runs 2,518 core rollouts of seven manipulation policies across 12 tasks (8 single-arm, 4 bimanual) on a low-cost arm farm — evolution cells with a BOM under $500 each. Each policy is trained or fine-tuned per task under a fixed 50-demonstration budget and evaluated for 30 rollouts per task–policy pair with on-site operator labels, then the raw success rates are published as a leaderboard alongside all 3,718 released episodes (26 hours, 3 camera views, ~1,290 successful / 89 suboptimal / 1,739 failed). π0.5 leads both embodiments (45.4% single-arm strict success, 52.1% bimanual), the newest GR00T is close behind on the overall table, but per-task rankings shift: Diffusion Policy from-scratch wins tool_removal (63%) and open_lamp_door (73%) — precision/force tasks — beating every VLA. cable_clip remains unsolved at 0% for all seven policies.

Key claims

Section titled “Key claims”
  • π0.5 leads the overall v0.1 leaderboard on both embodiments at the shared 50-demo-per-task budget: 45.4% strict success across the 8 single-arm tasks and 52.1% across the 4 bimanual tasks [§Leaderboard, per-task table].
  • The overall ranking is not the whole picture: on the two low-variance / precise-force tasks (tool_removal, open_lamp_door), a Diffusion Policy trained from scratch beats every VLA under test — 63% vs GR00T 30% / π0.5 13% on tool_removal, and 73% vs π0.5 23% / GR00T 13% on open_lamp_door [per-task table].
  • On the two eye-drops single-arm tasks the ranking is very different: π0 (76%) and π0.5 (70%) dominate eye_drops_to_shelf while ACT-from-scratch collapses to 0%; on eye_drops_to_basket seven policies cluster between 23% and 70%, so the task is not saturated but is also not the discriminator the eye-drops-to-shelf task is [per-task table].
  • Cable-clipping remains completely unsolved: every one of the seven policies scored 0% on cable_clip at 30 rollouts each, and the sibling cable_unclip task varies wildly (π0.5 70% down to Diffusion Policy 7%) — suggesting the failure is contact-establishment-under-clip-geometry rather than a general contact-perception limitation [per-task table].
  • The benchmark’s fixed-budget design (each policy is trained or fine-tuned with the same 50-demonstration budget per task) is the load-bearing methodological choice: it isolates the policy-recipe axis from data-scale confounds, at the cost of favoring policies that are structured to be data-efficient at small demo counts [§Task suite, §Evaluation protocol].
  • The full release includes both the scored core benchmark rollouts and a large corpus of failed and suboptimal rollouts (1,739 failed + 89 suboptimal out of 3,118 labeled core episodes) — the launch thread claims it is one of the largest open suboptimal manipulation datasets to date [§Released corpus, launch tweet].
  • The v0.1 protocol has known measurement holes the authors call out: scene resets are manual (no logged initial-state randomization), lighting is uncalibrated ambient office light, and every rollout label is assigned on-site by operators rather than by an automatic scorer [§Limitations and next steps].

Method

Section titled “Method”

Each cell is a small physical rig — a single arm or a bimanual pair on a shared table with an inference workstation co-located per farm. v0.1 ran on two single-arm cells and one bimanual cell (three cells total), which lets the same physical fleet run different tasks and policies in parallel. Each cell’s BOM is under $500 — the low-cost claim is what lets the benchmark run thousands of rollouts without a cluster of production robot arms.

The protocol is a fixed-budget behavior-cloning bake-off. For each of 12 tasks (8 single-arm, 4 bimanual), the team collected 50 teleoperated reference demonstrations (600 total across the suite), then used those demos as the sole training / fine-tuning budget for seven policies: three from-scratch policies (ACT, Diffusion Policy, SmolVLA) and four VLA fine-tunes (π0, π0.5, GR00T, MolmoAct 2). Each task-policy pair is then rolled out 30 times on the corresponding cell, and every rollout is manually labeled by an on-site operator as successful, suboptimal, or failed — 3,118 labeled core episodes total (2,518 counted toward the strict success rate, with the remaining ~600 spread across parallel labeling runs).

The reported metric is strict success rate: only clean successes count; both suboptimal and failed rollouts are treated as failures for the leaderboard. All 3,718 released episodes (including 600 labeled episodes from parallel runs beyond the core benchmark) come with three camera views. The benchmark is built on the Armnet software stack, and the launch thread promises the next Armnet generation is targeted at automating this pipeline so operators can supervise many more cells and drive real-hardware evaluations at “millions of evaluations” scale.

Results

Section titled “Results”

The public leaderboard on the project page reports strict success rates for each of the seven policies × 12 tasks. Headline numbers:

  • Single-arm overall: π0.5 45.4% > π0 41.7% > GR00T 34.6% > ACT 27.5% > Diffusion Policy 22.0% > MolmoAct 2 20.0% > SmolVLA 15.0% (computed from the per-task table).
  • Bimanual overall: π0.5 52.1% > π0 32.8% > Diffusion Policy 35.8% > MolmoAct 2 17.3% > GR00T 19.0% > SmolVLA 15.3% > ACT 2.5% (computed from the per-task table).
  • Per-task inversions: on tool_removal Diffusion Policy 63% beats π0.5 13% by 50 pp; on open_lamp_door Diffusion Policy 73% beats π0.5 23% by 50 pp; on fold_tea_towel Diffusion Policy 50% ties π0 50% and loses only to π0.5 70%.
  • Unsolved: cable_clip at 0% for all seven policies.
  • Language-following-adjacent asymmetry: on the two eye-drops single-arm tasks, π0 tops eye_drops_to_shelf at 76% (vs π0.5 70%, GR00T 43%, ACT 0%), while on eye_drops_to_basket the field compresses to 23–70% — the shelf variant is a much sharper discriminator.

The launch thread separately reports a training-vs-real-world diagnostic — training flow loss and validation action error correlate with real-world success across model classes, while validation flow loss does not — evidence that action-error tracking is the more informative offline signal for predicting real-world performance across policy families [launch thread reply].

Why it’s interesting

Section titled “Why it’s interesting”

For the recipe-lever debate this wiki has been tracking on VLA Models, ArmnetBench is a fixed-demo-budget, real-hardware datapoint that sharpens two ongoing questions. First, it strengthens π*0.6: a VLA That Learns From Experience (RECAP)‘s recipe (action pretraining + RECAP-style refinement) as the current overall winner at low-demo budgets on both single-arm and bimanual embodiments — π0.5 tops both leaderboard sides. Second, it undercuts the “one recipe dominates everywhere” reading: on precise-force tasks (tool_removal, open_lamp_door) a from-scratch Diffusion Policy beats every VLA by wide margins, and cable-clipping remains 0% for all seven policies. That’s a direct empirical answer to the tactile/contact counter-position in Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level and adjacent Malik-line arguments: VLA scaling has not solved contact-establishment tasks, and structured from-scratch policies remain competitive when force application is the bottleneck.

For the RL Environment Platforms surface, ArmnetBench is the third real-world manipulation benchmark filed alongside RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies and Claude Opus 5 driving robot arms zero-shot via Inspect Robots (Jay Chooi / Robocurve) (Inspect Robots), and picks a distinct point on the design space. RoboArena externalizes evaluation to crowd-sourced evaluators on shared DROID hardware, does not fix tasks, and uses pairwise A/B under a task-aware Bradley-Terry model — the trust argument is decentralization. ArmnetBench is the opposite: run by one team, fixed 12 tasks, fixed 50-demo budget, absolute strict success rates on a low-cost arm farm — the trust argument is that everything is auditable and cheap enough to replicate. Sibling to RoboLab — Task-Based Evaluation Benchmark for Robot Manipulation Policies on Isaac Lab (NVLabs) on the “fixed task set + fixed protocol” axis but real-hardware rather than Isaac Lab, and complementary to RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies which pairs sim tasks with a smaller real-hardware track on remote standardized hardware.

See also

Section titled “See also”