ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation

TL;DR

ABC is a fully open-source bimanual-manipulation behavior-cloning stack from a Berkeley/Stanford/Toyota-affiliated team, anchored on ABC-130K — 3,553 hours / 134,806 episodes / 195 tasks of bimanual teleoperation data, the largest such dataset on file. Alongside the data it ships two policy architectures (ABC-DiT, a 2B Diffusion Transformer paired with a small DINOv3 backbone; ABC-VLA, a 4.3B Gemma-3 backbone with a 45M action head), the ABC Sim platform (400h sim-teleop / 20 tasks, MuJoCo + Blender re-rendering) with empirically validated sim-to-real correlation (r=0.85 success, r=0.91 progress across 12 checkpoints), a fast distributed dataloader (abcdl), a passive-leader-arm DAgger intervention recipe, and over 100 hours of physical-robot evaluation logs. The headline architectural findings are that DiT heads benefit from scaling more than vision encoders do (DiT-xL beats DiT-S at every compute budget), and that the VLA’s 4.3B-VLM / 45M-head split allows amortizing one VLM forward pass across many diffusion targets per batch.

Key claims

ABC-130K contains 134,806 episodes / 3,553 hours / 195 bimanual tasks across 7 primitive categories (pick-and-place, folding, sorting, tool use, etc.), with episodes ranging 7s–469s — the largest filed open bimanual teleoperation dataset [§Data].
ABC-DiT (2B) deliberately allocates 1.93B to the DiT action head and only 85.7M to a frozen DINOv3 vision encoder, on the rationale that the encoder is compute-heavy per parameter (3 camera views) so scaling the head is cheaper per-FLOP than scaling the encoder [§ABC-DiT 2B].
Larger DiT heads reach lower training loss both per step and per FLOP across a four-size sweep (S, B, L, xL); at 300K steps DiT-xL reaches 0.040 loss vs DiT-S’s 0.059 [§ABC-DiT 2B, scaling figure].
ABC-VLA (4.3B Gemma-3 backbone + 45M action head) amortizes one expensive VLM forward over k diffusion targets per batch by replicating hidden states k times and pairing each with an independent (noise, timestep) draw — yielding lower-variance gradients and higher SNR on the VLM-side gradient signal [§ABC-VLA 4.3B].
Sim-eval is empirically predictive of real-eval: across 12 checkpoints spanning architectures, batch sizes, and training durations on three matched tasks (throw-bottles-in-bin, load-plates-in-dishrack, turn-mugs-right-side-up), sim and real correlate at r=0.85 on strict success and r=0.91 on task progress [§Sim].
Validation action error and training error both correlate strongly with real-world performance across checkpoints (r=−0.84 training error, r=−0.89 validation action error), while validation loss is essentially uncorrelated (r=−0.04) — the standard cross-entropy/MSE loss is the wrong offline metric [§Evaluation].
Passive-leader-arm DAgger: a method for collecting intervention data with cheap GELLO-style passive leaders, using the delta of leader pose since intervention added to the follower’s current end-effector and IK-solved onto the followers — sidesteps the active-leader requirement standard in DAgger literature [§Collecting Interventions].
Concrete DAgger result on box-folding: ABC-DiT finetuned on 10h of curated data reaches 24% mean progress; two rounds of passive-leader DAgger (≈1–1.5h each at 30% then 15% intervention rates) raise it to 85% [§Collecting Interventions].
Progressive compilation stack (eager → per-block torch.compile → fullgraph → CUDA graphs) reaches 27.6 Hz for ABC-DiT and 57.2 Hz for ABC-VLA inference on a single 5090 [§Inference speed].
abcdl encodes each episode as a single stacked-view MP4 plus a binary state/action file with deterministic frequent keyframes, allowing analytical frame-index computation that bypasses torchcodec’s default whole-file index scan — near-free random frame access [§Fast data loading].
Single-task finetuning performance improves with the size of the pretraining dataset on the 1K/3K/10K-hour ablation; ABC-VLA additionally benefits from larger batch size where ABC-DiT shows less sensitivity [§Scaling].

Method

ABC factors the bimanual-BC stack into four pieces: data (ABC-130K + ABC Sim teleop), policies (ABC-DiT, ABC-VLA), evaluation (matched sim+real, 100h physical), and infrastructure (passive-leader DAgger, abcdl dataloader, compiled inference). The two policies share an input shape (3 camera views + proprioception + language) and output an action chunk via flow-matching/diffusion, but differ in where the parameter budget lives. ABC-DiT is conditioning-light: a small DINOv3 backbone (85.7M) supplies visual tokens to a heavy 1.93B DiT denoiser that does the bulk of the work. ABC-VLA is conditioning-heavy: a Gemma-3 4.3B VLM backbone produces hidden states that feed a deliberately tiny 45M action head; per-batch the same VLM hidden states are reused across k independent diffusion samples so the cost of the VLM forward is amortized.

ABC Sim is MuJoCo-based with an offline Blender re-rendering pipeline that takes any saved trajectory and produces ray-traced frames at higher visual fidelity. The dataset includes 400h of sim-teleop across 20 tasks, and the team validates sim-as-cheap-eval-proxy with a paired sim+real evaluation on three matched tasks. For data collection, ABC swaps the active leader arms standard in teleoperation rigs for passive GELLO-style encoder-only readouts; this same hardware is then reused for DAgger interventions, where the intervention’s pose delta is applied to the follower’s current end-effector and IK-solved, removing the active-leader requirement.

Results

ABC-130K: 134,806 episodes / 3,553h / 195 tasks, released on HuggingFace at XDOF/ABC-130k [§Data].
ABC-DiT scaling: DiT-xL reaches 0.040 training loss at 300K steps vs DiT-S 0.059; larger models win at fixed step and fixed FLOP budget [§ABC-DiT 2B].
Sim-real correlation across 12 checkpoints: r=0.85 strict success, r=0.91 task progress [§Sim].
DAgger on box-folding: 24% → 85% mean progress after ~3h of intervention collection [§Collecting Interventions].
Inference: 27.6 Hz (DiT) and 57.2 Hz (VLA) on a single 5090 after the compile + CUDA-graph stack [§Inference speed].
Single-task finetuning: monotone gains as pretraining hours scale 1K → 3K → 10K on the held-out validation action error and on real-world rollouts [§Scaling].
Cross-checkpoint correlation with real performance: training error r=−0.84, validation action error r=−0.89, validation loss r=−0.04 [§Evaluation].

Why it’s interesting

ABC is the first filed open-weights bimanual VLA paper that actually releases the entire stack — data, sim, real-eval logs, training code, hardware design, dataloader — at the scale and packaging this concept page has been tracking for closed releases. It lands as a direct counter-bet to Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models (Spirit v1.5, Qwen3-VL + DiT action head, “scale the demo distribution, don’t filter”) in two specific ways: ABC additionally argues you can use simulation as eval rather than only as training data, and it ships clean DiT-scaling ablations on robot-policy heads that Spirit declined to publish. On the architectural-recipe axis, ABC-VLA’s “tiny action head + amortize VLM forward across k diffusion draws” is a concrete and unusually pragmatic alternative to the π*0.6: a VLA That Learns From Experience (RECAP) (π*0.6 / RECAP) “advantage-conditioned flow-matching head” recipe and to the Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (Embodied-R1.5) “make the VLM do all the work via pointing/grounding” recipe.

The cross-checkpoint finding that validation loss is uncorrelated with real performance (r=−0.04) while validation action error and training error correlate strongly (~−0.85) is a direct echo of the RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies / RoboArena rolls back evaluations after benchmark hacking observed since April (Pranav Atreya announcement) concern that offline robot-policy metrics are unreliable — ABC supplies a concrete worked example with 12 paired checkpoints. And the passive-leader DAgger trick is a useful counterweight to the active-leader-rig dependency that has limited DAgger uptake in the open community.