Skip to content
Bud's Wiki

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks

TL;DR

Section titled “TL;DR”

BiDex-100 is a simulation benchmark for bimanual dexterous manipulation with 106 tasks organized along five capability axes (generalization, instruction following, limited demonstrations, long horizon, memory), two high-DoF hand embodiments, and 8,080 success-validated teleoperated demonstrations totaling 6.25 million frames. Each task has a paired out-of-domain variant with controlled visual and spatial changes so robustness can be scored directly against an ID baseline. The authors evaluate seven recent Vision-Language-Action models under a unified protocol to establish per-axis capability baselines plus robustness and data-efficiency measurements. Positions itself as the first benchmark deliberately tailored to the dexterous-hand regime where existing manipulation suites (LIBERO, RoboTwin, RoboCasa) under-characterize current VLAs.

Key claims

Section titled “Key claims”
  • Few existing benchmarks are tailored for dexterous manipulation, leaving dexterous capabilities of current robotic action models insufficiently characterized [Abstract].
  • 106 tasks are organized along five explicit capability axes — generalization, instruction following, limited demonstrations, long horizon, and memory — rather than aggregated into a single success-rate ranking [Abstract].
  • Each task is paired with an out-of-domain variant featuring controlled visual and spatial changes, enabling systematic robustness assessment relative to in-domain performance on the same task [Abstract].
  • The benchmark supports two embodiments and ships 8,080 success-validated teleoperated demonstrations totaling 6.25 million frames [Abstract].
  • Seven recent Vision-Language-Action models are evaluated under a unified protocol to establish baselines for capability-specific performance, robustness, and data efficiency [Abstract].

Method

Section titled “Method”

The benchmark exposes 106 bimanual simulation tasks factored by five capability axes with two high-DoF hand embodiments. Per-task evaluation uses success-validated teleoperated demonstrations (8,080 trajectories, 6.25M frames) as the training substrate, and each task’s ID eval is paired with an out-of-domain variant that applies controlled visual and spatial changes to the same underlying task — so a per-task robustness drop is directly measurable without introducing a new task definition. Seven existing VLAs are run under one unified evaluation protocol to produce the headline baseline table across the five axes plus robustness and data-efficiency slices. Detailed task list, per-axis evaluation protocol, specific hand embodiments, and the identity of the seven VLA baselines are not retrievable from the abstract alone at filing time.

Results

Section titled “Results”

Abstract reports that the authors establish baselines for capability-specific performance, robustness, and data efficiency across seven recent VLA models [Abstract]. Specific per-task, per-axis, and per-model numbers are not retrievable from the abstract at filing — flagged for /bud refresh once the full PDF or project page is reachable.

Why it’s interesting

Section titled “Why it’s interesting”

Fills a benchmark gap on the VLA Models recipe-lever board — current filed VLA evaluations lean on LIBERO / RoboTwin / RoboCasa / RoboLab-120 / real-robot farms, none of which cover high-DoF bimanual dexterous manipulation with factored capability axes. Complements DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations DONIC (first generalist bimanual dexterous tracker) and The Cartesian Hand: In-Hand Manipulation with All-Linear Fingers on the policy side by moving the lever to the shared evaluation substrate: a factored-axis + paired-OOD design that scores robustness directly, with pre-collected 8K-demo supervision so data-efficiency is a first-class axis rather than a side ablation. Sits adjacent to the sim-only evaluation lineage (RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies RoboDojo) but specializes to dexterous bimanual rather than general manipulation, and to The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction The Imitator Game on the “level-grade the capability axis rather than aggregate” methodological axis.

See also

Section titled “See also”