Skip to content
Bud's Wiki

Reality Check — Pokeandwiggle's Real-World Robot Manipulation Leaderboard on an FR3 Duo Farm

TL;DR

Section titled “TL;DR”

Reality Check is a public leaderboard + benchmark for real-world robot manipulation policies, running on a bimanual Franka FR3 Duo farm hosted at the Deutsches Museum in Munich. The initial PAW-GEN-10 release ships 14,400 rollouts across four open-source VLAs (π0.5, MolmoAct 2, GR00T N1.7, DiT-Flow) over 10 environments × 3 data-scaling tiers (D10/D100/D300) × 2 spatial-generalization conditions (nominal / interpolation), with every object placement physically fixed per-rollout (over 40,000 object placements across the benchmark). Each episode is double-reviewed by two independent operators with a QA engineer sampling for third-pass audit; grading is per-subtask (Excellent / Ok / Failed) with eight metrics beyond success rate (task progress, execution quality, time, speed, SPARC smoothness, jerk, safe-failure rate, max contact force) reported with 95% CIs. Half the task-specific training data and all evaluation rollouts are open; the other half (e02, e06–e08, e10) is held out. Positions itself as a third-party, non-crowd-sourced counterweight to RoboArena’s distributed A/B protocol and to any lab’s own cherry-picked demos — a remote policy server is the only integration point.

Key claims

Section titled “Key claims”
  • The benchmark fixes a bimanual FR3 Duo embodiment and specifies each environment down to allowed controls, success criteria, constraints, and per-subtask steps — e.g. e01 “Sort screws into bins” requires 3 black screws in red bin + 2 silver screws in black bin, max 120 s, one screw per grasp [§Environments / e01].
  • Four open-source VLAs are evaluated under matched fine-tuning: π0.5, MolmoAct 2, GR00T N1.7, and DiT-Flow — all started from their official checkpoints (except DiT-Flow which has no public default and starts from an ImageNet vision encoder), then fine-tuned by the Reality Check team on the target tasks and data regimes [§Training Protocol].
  • Eval axes are explicit and multiplicative: 10 environments × 3 data tiers × 2 spatial conditions × 2 mid-training tiers (M100 marked “coming soon”) × 30 rollouts per combination = 3,600 rollouts per model and 14,400 across the four-model release [§Eval Procedure, table].
  • Scene reproducibility is the load-bearing design choice: every object pose per rollout is set to a prescribed initial pose, scenes are built from scratch between rollouts, and every model faces the same 30 scenes per combination — more than 40,000 placed-exactly objects across the benchmark [§Scene Reproducibility].
  • Spatial generalization is split into three regimes — Nominal (training placements), Interpolation (unseen placements between training placements), and Extrapolation (outside the convex hull, marked “coming soon”) — explicitly framed as a memorization-vs-understanding diagnostic [§Spatial Generalization].
  • Data scaling is nested rather than independent: D10 ⊂ D100 ⊂ D300 demonstrations per environment, so the across-tier curve measures sample-efficiency on the same trajectory distribution rather than confounding with distribution drift [§Task-Specific Demonstrations].
  • Grading is per-subtask with three ratings — Excellent (one fluid motion), Ok (meets objective, minor wobble), Failed (misses objective, recoverable by later subtask) — and the headline success rate counts Excellent + Ok, while Execution quality reports the Excellent share of successes [§Eval Procedure, ratings table].
  • Every evaluation is reviewed twice (operator who ran it rates first, second operator verifies placements + annotates subtasks + can correct the rating or request re-evaluation) with a QA engineer sampling for third-pass audit that can send a series back for re-annotation or re-evaluation [§Eval Procedure].
  • Reported metrics span eight dimensions beyond success rate — task progress, execution quality (Excellent-vs-Ok share of successes), avg execution time (lower better), execution speed (steps per active minute), SPARC smoothness (closer to -1 is better), avg jerk at 10 Hz (lower better), safe-failure rate (both arms OK at end), and max contact force on either arm (lower better) — all with 95% CIs [§Per Environment].
  • Dataset release policy is explicitly asymmetric for benchmark integrity: five environments (e01, e03, e04, e05, e09) ship all task-specific training data, eval rollouts, and object placements; the other five (e02, e06, e07, e08, e10) are fully held out [§Environments].
  • Mid-training is a declared future axis (M0 = no mid-training, direct task-specific fine-tune from the developer’s checkpoint; M100 = first mid-train on 100 hours of embodiment data from the farm, then fine-tune), designed to measure what cross-task experience on the specific embodiment is worth [§Mid-Training].
  • Operational integration is remote-only: policies connect via a remote policy server to the FR3 Duo stations and receive fully annotated rollouts within 30 minutes of each run’s end — the lab shipping the model never physically touches the hardware [§Nicolas Keller launch thread, tweet 3/10].
  • Changelog discipline is explicit: nine evaluation episodes have been re-annotated post-release across DiT-Flow, GR00T N1.7, MolmoAct 2, and Pi 0.5 (e.g. step rating failed→excellent on DiT-Flow e08 D10 Interpolation), with the net effect reported as <1% change in overall execution quality for GR00T N1.7 / Pi 0.5 and <0.1% change in progress for every model — no success rate or overall ranking changed [§Export changelog].

Method

Section titled “Method”

PAW-GEN-10 is a leaderboard-first benchmark running at the Deutsches Museum in Munich on bimanual Franka FR3 Duo stations with a 3D scene-understanding + object-tracking pipeline that physically verifies every randomly-sampled object pose for every training and evaluation episode. The evaluation axes are environment × data tier × spatial placement × mid-training tier × rollouts, with 30 rollouts per combination; the initial release freezes the mid-training axis at M0 and the spatial axis at Nominal + Interpolation, yielding 10 × 3 × 2 × 30 = 1,800 per model-condition pair and 3,600 rollouts per model × four models = 14,400 total.

Each environment specifies allowed controls (full arm + gripper control at 30 Hz for all 10 current environments), success criteria, hard constraints (max episode duration, no-touch rules on obstacles, order constraints), and a decomposed step list. Policies are evaluated by running the specified task; subtasks are annotated and rated (Excellent / Ok / Failed) by the operator, verified by a second operator, and sampled by a QA engineer. The scoring protocol collapses the per-subtask ratings into eight reported metrics — success rate, task progress (share of steps completed), execution quality (Excellent-vs-Ok share), avg execution time, execution speed (steps per minute), SPARC smoothness (dimensionless ≤ -1), avg filtered jerk, safe-failure rate, and max contact force from FR3 joint-torque sensors — with 95% confidence intervals derived from the 30 rollouts per combination.

Training protocol fine-tunes each VLA from its developer’s public checkpoint and recipe on D10 / D100 / D300 demonstrations per environment, with the nested-set property (D10 ⊂ D100 ⊂ D300) by construction. All training uses only Excellent-rated teleop demonstrations with no recoveries or suboptimal episodes — a design choice that aligns with the Spirit v1.5 “clean teleop is the lever” position (see Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models) and against the on-the-robot failure-data substrate position of RW-RL Dataset: Real-World Reinforcement Learning for Robots / OopsieData — a multi-lab dataset of real robot manipulation failures. DiT-Flow is initialized from an ImageNet vision encoder (no public checkpoint for the full model); the other three start from their public foundation checkpoints.

Results

Section titled “Results”

No headline leaderboard numbers were readable from the page at filing (the per-model + per-environment cells render client-side), but the structural release numbers are concrete: 14,400 rollouts across 4 models × 10 environments × 3 data tiers × 2 placement conditions × 30 rollouts per condition. Four models are unveiled in this round (π0.5, MolmoAct 2, GR00T N1.7, DiT-Flow) with M100 (100 hours of embodiment mid-training) marked as “coming soon” alongside the extrapolation placement regime. The post-release re-annotation audit found that correcting 9 rollouts across the four models shifted overall execution quality by less than 1% for GR00T N1.7 / Pi 0.5 and progress by less than 0.1% for every model — the first filed quantitative stability number for small-N real-hardware per-rollout human annotation.

Why it’s interesting

Section titled “Why it’s interesting”

Reality Check sits in the real-hardware pole of the RL Environment Platforms cluster that RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies RoboArena and Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) Inspect Robots occupy — but with an opposite protocol discipline on both ends: centralized (one farm in Munich, not seven academic labs) and fixed-task-with-fixed-placements (not pairwise crowd-sourced A/B), while ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm ArmnetBench stakes out the low-cost multi-arm fleet at a different hardware tier (SO-100/-101 vs FR3 Duo). The three-way protocol split — RoboArena (crowd-sourced), Reality Check (centralized FR3 farm with physics-verified placements), Inspect Robots (ship-your-own-hardware harness) — gives the field a triangulation surface for real-hardware eval that didn’t exist until mid-2026.

The design decision worth engaging with for VLA Models work is the strict Excellent-only training-data policy paired with per-rollout physics-verified placement reproduction: it isolates the “generalization to unseen poses” axis cleanly from the “robust to messy demonstrations” axis, letting the D10/D100/D300 × Nominal/Interpolation grid report scaling-under-held-constant-data-quality numbers that complement StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models‘s and Empirical results fine-tuning π0.5 on a real manufacturing task‘s on-the-rig fine-tuning sweeps.

See also

Section titled “See also”