Skip to content
Bud's Wiki

We Looked at the Data — Diagnosing Open-Source Robotics Datasets for WAM Training

TL;DR

Section titled “TL;DR”

Pantheon Inc. (Sambhav Gupta et al.) audited several major open-source UMI and bimanual teleoperated robotics datasets — FastUMI_100k_lerobot, MolmoAct2-BimanualYAM, 10Kh-RealOmin-OpenData, XDOF/ABC-130k — plus inspected another 8 (HiFi-UMI-2K, AgiBotWorld-Beta, WEB-Dataset, RealSource-World, HIW-500, Hy-Embodied-0.5-VLA-Data, RoboMIND, Galaxea-Open-World). They normalize every dataset into a unified episode schema (14-D bimanual state + SE(3)-consistent relative EE actions + per-frame timestamps), then diagnose a taxonomy of 15+ common anomalies — camera disconnections, freezes, cuts, dropped-frame fast-forwards, dark exposure, action-video misalignment by several frames, missing gripper actions, spiky actions, ultra-slow actions, actionless tails — with per-episode flag conditions and severity assessments. Downstream framing is explicitly world-modeling-first: each anomaly is scored by how much it poisons WAM / world-action model training objectives (unlike VLA / behavior-cloning where some anomalies are less damaging). They release per-episode manifests with anomaly flags and boolean masks + smoothed action data for four of the datasets on HuggingFace under sambhav-pantheon/*_pantheon_annotated.

Key claims

Section titled “Key claims”
  • Many published open robotics datasets contain silent data corruption at rates high enough to raise the training noise floor — camera disconnections/freezes, cuts, dropped-frame fast-forwards, misaligned actions, non-uniform frame intervals — and the impact is largest on world-modeling objectives where the causal relationship between action and next observation is directly supervised [§Anomalies].
  • Stored gripper open/close actions can lead the actual gripper motion by several frames while position/rotation are correctly aligned — a subtle per-episode misalignment that would train WAMs to predict state transitions too early or too late [§Anomalies — Actions shifted by several frames].
  • The single-episode ceiling case: a stored action can register a 10.46 cm translation in one 50 ms interval with no corresponding motion in the wrist camera — a large action with no state change directly breaks the WAM’s action→observation causal signal [§Anomalies — Action spike with no visual analogue].
  • Action stops-recording-mid-episode is detectable via state-change vs action-change correlation dropping below 0.04 on one arm while the other stays correlated — a per-arm asymmetric failure mode that a global episode-quality flag would miss [§Anomalies — Actions stopped recording].
  • Missing intermediate frames (dropped 40–100 ms windows) manifest visually as fast-forwarded footage with smoother-than-expected state changes; XDOF/ABC-130k and (to a lesser extent) 10Kh-RealOmin-OpenData are the most-affected datasets [§Anomalies].
  • Some anomalies are dataset-uniform rather than per-episode — spiky per-frame action patterns and non-uniform camera brightness intervals appear consistently across entire datasets, requiring per-dataset correction (smoothing, downsampling) rather than per-episode filtering [§Anomalies — Camera freeze, Spiky actions].
  • Per-embodiment idiosyncrasies matter: HiFi-UMI-2K exhibits systematic mutual gripper pixelation in the opposing wrist camera during bimanual manipulation, a data-quality axis that penalizes WAMs specifically for accurately predicting one gripper’s view from the other’s [§Anomalies — One gripper is pixelated].
  • Raw actions — especially in UMI datasets not constrained by robot arm kinematics — contain jerky spikes and abrupt accelerations that harm both safety and policy variance; smoothing must minimize deviation from the global SE(3) trajectory (not per-frame states) to avoid compounding drift [§Action Processing and Smoothing].
  • Actions can only be treated as SE(3) group operations for delta computation — naïve subtraction of states to find deltas is incorrect. The verification protocol is to round-trip original states → relative actions → SE(3)-cumulative reconstructed states and confirm trajectory preservation [§Preprocessing and Unified Episode Schema, §Action Processing and Smoothing].
  • Static flag thresholds (visual-change magnitudes, action-magnitude ratios, action-state correlation cutoffs) inherently produce false positives and false negatives; a small IDM fit per-dataset or globally is proposed as a smarter data-quality filter that generalizes better than hand-tuned thresholds — but is significantly more opinionated [§Beyond Our Current Static Checks].
  • Bias-class anomalies (unnecessary handoffs, extremely slow non-uniform action segments, nonuniform task descriptions) are hard to automatically flag but represent additional axes of contamination that a VLM-based pipeline could plausibly detect [§Anomalies — Unnecessary handoffs, Extremely slow actions, Nonuniform task descriptions].

Method

Section titled “Method”

The team defines a dataset-agnostic unified episode struct with 14-D bimanual robot states (7 per arm), video streams for at-most-one overhead + one-per-wrist camera downscaled and center-cropped to 224², SE(3)-computed relative EE delta actions in both local and global frames, and per-frame globally-synced timestamps. Each source dataset gets a bespoke adapter that respects its file format, camera count, resolution, and quirks. Episodes are trimmed so every stream has exactly T states and T-1 actions with equal counts. This normalization stage already surfaces many malformed episodes (non-overlapping video streams, missing frames, extremely short episodes).

Anomaly detection follows an iterative loop: (1) form a prior about what well-formed data looks like from spot checks; (2) encode the prior as an explicit programmatic flag condition (e.g. luma MAD threshold for camera-freeze detection, action-state correlation floor of 0.04 for action-recording dropout); (3) run flags across all episodes and spot-check triggered examples; (4) assess flag severity from full-reject through false-positive. A visualization dashboard shows synced camera views, action graphs, and 3D reconstructions of EE trajectories per episode, extended per-flag as needed. Action smoothing is formulated as minimizing acceleration + global-trajectory-deviation loss terms in opposition, correcting spiky UMI-style actions without compounding end-to-end drift.

Results

Section titled “Results”

Per-episode manifests released for 4 datasets on HuggingFace (sambhav-pantheon/IPEC-COMMUNITY_FastUMI_100k_lerobot_pantheon_annotated, sambhav-pantheon/allenai_MolmoAct2-BimanualYAM-Dataset_pantheon_annotated, sambhav-pantheon/XDOF_ABC-130k_pantheon_annotated, sambhav-pantheon/genrobot2025_10Kh-RealOmin-OpenData_pantheon_annotated) with per-episode flags, frame-level boolean masks where flags are frame-assignable, missing-frame indices, and raw + smoothed frame-to-frame action data. No headline aggregate-quality-loss percentage is reported per dataset; the framing is that flag rates and severity vary by dataset (missing frames dominate ABC-130k, brightness spikes appear uniformly in certain datasets, gripper-pixelation appears specifically in HiFi-UMI-2K).

Why it’s interesting

Section titled “Why it’s interesting”

This sits alongside Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models (Spirit AI’s “clean data is the enemy of great robot foundation models”) but takes the opposite structural position — Spirit argues large-scale diversity beats aggressive filtering for VLA scaling, while Pantheon argues that for WAMs specifically many anomalies directly poison the action→observation causal signal and must be filtered or masked. Also complements Instance Labs — Verifying Robot Learning Episode Success (Instance Labs, episode-level success verdicts as an upstream data-verification service) and Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs (Kite ML, frontier-VLM success-detection calibration) by attacking a distinct layer of the data pipeline: not episode-success labeling but per-episode signal integrity across camera, state, and action streams. The three together (Pantheon → Instance Labs → Kite ML) sketch a data-QA pipeline that the field has largely left unaddressed.

The list of audited datasets overlaps most of the substrate this wiki tracks under Synthetic Training Data and Human-to-Robot Retargeting — HiFi-UMI-2K (HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone), MolmoAct2 Bimanual YAM (Pareto (Hebbian Robotics) indexes AllenAI's MolmoAct2 Bimanual YAM dataset — clusters + prompt normalization (Brandon Ong)), Hy-Embodied-0.5-VLA-Data (Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack), Galaxea Open World, HIW-500, AgiBot World, RoboMIND — making Pantheon’s manifests an immediately-useful upstream filter for the majority of open UMI + bimanual-teleop pretraining stacks. The claim that missing frames concentrate in XDOF/ABC-130k is directly relevant to any lab building on that corpus (Shotwell’s 415h annotation layer notwithstanding, Shotwell releases 415+ hours of annotations on ABC-130K).

See also

Section titled “See also”