Argus: An Open-Source Annotator for Robotics Data

TL;DR

Pantheon releases Argus, an open-source VLM-based annotation and data-quality pipeline for robotics datasets. Argus ingests any episode in its raw format (LeRobot v2/v3, MCAP, mp4/mkv archives) along with recorded states, actions, and publisher labels, then uses a frontier VLM (benchmarked on GPT-6 Astra, Claude Opus 5.5, GPT-6 and 6.1 Sol, DeepSeek v4.1 Flash) to emit dense per-episode timelines, operator-mistake flags, goal-frame identification, and instruction-mismatch checks, supplemented by deterministic checks for sped-up recordings and swapped camera streams. Across 3,546 audited episodes from nine datasets, 27% have at least one medium-or-higher-severity issue, and the full ~150K annotations are released on pantheon.inc/data-board. This is the direct follow-up to We Looked at the Data — Diagnosing Open-Source Robotics Datasets for WAM Training — same audit targets, but moving beyond deterministic checks into the context-dependent errors (mislabeled instructions, labelled-success-but-undone goals, operator meandering) that only a VLM harness can catch.

Key claims

27% of 3,546 labelled episodes across nine datasets have at least one Argus-flagged issue of medium severity or higher, with 46 distinct problem types found across the nine datasets (25 in the data and 21 in operator demonstrations) [§Summary of results].
20% of episodes across the eight datasets with instructions or labels have an instruction or label that does not fully match the footage (14% at medium or high severity); OpenAoE-2000h is the most-affected at 79% of 107 labelled episodes [§Summary of results, §02 Task instructions do not match footage].
11% of MolmoAct2-BimanualYAM episodes (3,536 of all 32,246) are sped up: the control loop records at <30 Hz but episodes are timestamped at 30 Hz, invisible to format inspection but detectable via the fixed ~125 ms leader-follower delay collapsing into too few frames [§04 Recording plays faster than real time].
1.5% of FastUMI-100K episodes (14/964) have camera streams swapped between left and right grippers; recorded motion alone cannot find all swaps because 64 other FastUMI episodes also have broken motion data, so Argus uses the VLM to detect swaps directly from footage [§05 Camera files swapped].
GPT-6 Astra labels much more densely than peer VLMs — 44 events per minute on tele-op/UMI vs 1.9× fewer for the densest of Claude Opus 5.5 / GPT-6 Sol / DeepSeek v4.1 Flash; the gap is widest on human ego video (26 vs ≤15 key events per episode) [§How Argus performs across models].
In-context learning from Astra’s traces does not close the density gap: adding one fully-annotated Astra example to peer-model prompts shifts outcome agreement only marginally and lifts annotation density by 4–9%, reaching at most 59% of Astra’s rate [§How Argus performs across models].
GPT-6.1 Sol is the first non-Astra model to match Astra on density (96% as many events per minute, 95% outcome agreement on tele-op/UMI) at $5 per hour of footage vs Astra's$ 27 — a 5× cost reduction with near-parity quality [§How Argus performs across models].
Out-of-the-box Astra systematically mislabels wrist-camera scenes (misreporting a block as turned over in 9/9 runs on episode 1346 because the wrist camera moves with the gripper) and gets biased by incorrect task instructions (reporting “chopsticks picked up” when the gripper holds a fork); Argus’s harness fixes both by describing camera mount type and treating instruction-named objects as claims to verify against frames [§Astra out of the box].
Severity ratings are not binary — an episode that completes the task but then undoes it (10% of Galaxea Open-World, 1.9% of MolmoAct2-BimanualYAM, 1.8% of 10Kh-RealOmin) is a clean trajectory for behavior cloning if trimmed at the goal frame but a corrupting trajectory for goal-conditioned models that read the last frame as the goal [§01 Goal-undone episodes, §Downstream Impact].
Operator mistakes vs recording defects have opposite downstream implications: operator mistakes should be filtered or trimmed for imitation learning but kept as recovery-learning data for world models — Argus flags both but recommends different treatments [§08 Labelling operator mistakes as clean demonstrations].

Method

Argus is structured as a VLM-driven annotation harness with a deterministic-check sidecar, designed to run on any sufficiently capable VLM but anchored to GPT-6 Astra for the benchmark. The pipeline (a) adapts resolution per task — GPT-6 Sol pre-reads the task spec, downstream footage goes out at 224px for confirmed low-detail tasks and 448px for high-detail or uncertain tasks, with gripper-open/close frames always sent high-res; (b) describes each camera’s mount (fixed, wrist-mobile, head-mobile) and prompts the VLM to infer each camera’s direction from scene geometry so that wrist-camera motion isn’t confused for scene change; (c) presents instruction-named objects as claims to verify rather than ground truth, requiring the VLM to describe identifying features before accepting a name; (d) reconstructs raw timelines from observed frames (asking the VLM whether an object was dropped vs placed based on its landing geometry) rather than assuming optimal action semantics.

Deterministic checks exploit the physics of the recording: UMI pose annotations are validated against camera motion (fixed camera-to-gripper geometry means a stationary pose during camera motion is a contradiction); sped-up recordings are detected via the leader-follower delay collapsing into too few frames; camera swaps are flagged when motion-change correlation (left stream vs right gripper motion = 0.68 vs 0.28 for the left gripper) inverts. Argus runs these checks before and alongside the VLM pass. Per-episode output includes action-phase timelines, object state changes, key events with outcomes, goal-frame identification (including goal-then-undone), completion outcome with reason, instruction-check verdict, operator-mistake log with timing and severity, failed-attempt-plus-recovery pairs, every data problem with severity rating, and (for human ego video) per-activity annotations with hand-visibility flags.

Results

Argus audits 3,546 episodes (66.5 hours of footage) across nine datasets: MolmoAct2-BimanualYAM, Galaxea Open-World, 10Kh-RealOmin, FastUMI-100K, HABIT, ABC-130k, Egocentric-100K, Gen-HumanEgo, and OpenAoE-2000h. 27% carry at least one medium-or-higher-severity issue; 46 distinct problem types are documented. Labelling costs about $26/hour on tele-op,$ 30/hour on UMI, $19/hour on human ego video.

Dataset-level findings:

MolmoAct2-BimanualYAM: 25% instruction mismatch (177 of 202 “failed” demonstrations carry an instruction that doesn’t match), 11% sped-up recordings (3,536 of all 32,246 episodes), 5.5% unlabeled operator mistakes [§Every data issue].
OpenAoE-2000h: 79% instruction mismatch — the highest in the audit [§02].
FastUMI-100K: 1.5% swapped camera streams, 6% recorded-motion-disagrees-with-video, 1.7% recorded gripper openings that never actually happen [§Every data issue].
10Kh-RealOmin: 7.1% long idle stretches, 6.8% motion-disagrees-with-video, 1.8% success-then-undone [§Every data issue].
Egocentric-100K: 12% of 112 clips have the head camera taken off mid-recording, 12% have long idle stretches, 0 operator mistakes [§Every operator mistake].
HABIT: 3.8% have privacy blurring that masks task objects (donuts, not just faces) [§06].

Cross-model benchmark on 193 episodes: outcome agreement with Astra ranges 73%–96% across Claude Opus 5.5, GPT-6 Sol, DeepSeek v4.1 Flash, and GPT-6.1 Sol. Annotation density ranges from 23 events/min (GPT-6 Sol) to 43 events/min (GPT-6.1 Sol) vs Astra’s 44/min. Human-ego video key-event density: Astra 26 per episode, all three mid-tier peers ≤15, GPT-6.1 Sol 28.

Why it’s interesting

This is the direct Phase 2 of the Pantheon audit — We Looked at the Data — Diagnosing Open-Source Robotics Datasets for WAM Training used deterministic flag conditions on normalized SE(3) actions and uniform episode schemas, which caught camera disconnects, action-video misalignments, and spiky actions but could not detect context-dependent errors (mislabeled instructions, labelled-success-but-undone goals, operator meandering). Argus closes exactly that gap by swapping in a frontier VLM for the semantic checks while retaining the deterministic sidecar for the camera-swap and playback-speed detection that are hard to do visually. It also sharpens the open question raised across the VLM-as-Evaluator concept page about judge-backbone sensitivity: Argus publishes the first head-to-head annotation-density comparison across five frontier VLMs on the same 193 robotics episodes, with Astra and GPT-6.1 Sol clearly ahead of Claude Opus 5.5 / GPT-6 Sol / DeepSeek v4.1 Flash — a cleaner apples-to-apples datapoint than the per-paper judge selection studies filed elsewhere (IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation on infographics, Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs on robot success detection).

Argus also complements the service-layer plays this wiki tracks: Instance Labs — Verifying Robot Learning Episode Success (per-episode success verdicts), Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs (frontier-VLM success-detection calibration), and Shotwell releases 415+ hours of annotations on ABC-130K (415h of ABC-130k annotations) each attack a different data-pipeline layer. Argus is the broadest of the set — dense timeline + goal + mistake + instruction-check + recording-defect, all in one pass — and the only one where the full annotation corpus is released, letting third parties A/B whether downstream WFM / VLA quality actually improves after Argus-filtered training. The split-recommendation framing (filter operator mistakes for imitation learning, keep them for world-model recovery learning) is the clearest filed articulation of why “data quality” is not a single-objective problem for robotics.