Kalyan — dense Argus-format annotations for >5,000 hours of public robot data, labeled by GPT-6.1 Sol

TL;DR

Kalyan announces a release of dense per-episode Argus-format annotations covering more than 5,000 hours (~290k episodes) of public robot data — ABC-130k, MolmoAct2 YAM, DROID, GM-100, and ABC-Sim — labeled by GPT-6.1 Sol with per-hand subgoals and actions, outcomes, mistakes, recoveries, and object-state changes. The dataset ships on Hugging Face at nkalyanv/argus-robot-annotations and is pitched as both a data-cleaning gate (used to mask out idle stretches and mistakes while keeping recoveries, retraining on ABC-130k lifts task progress +14.9 pts in sim) and the training substrate for a hierarchical VLA factorization where the VLM holds memory and planning while the action head only learns a small set of atomic skills across many scenes. Work is with Tobias Hoeppe, Shashank Reddy, and Stefan Bauer.

Key claims

Release scope: dense Argus-format annotations over >5,000 hours and ~290k episodes drawn from ABC-130k, MolmoAct2 YAM, DROID, GM-100, and ABC-Sim, with per-hand subgoals + actions, outcomes, mistakes, recoveries, and object-state changes [tweet §opening].
Downstream data-cleaning gain: using the Argus labels on ABC-130k to mask idle stretches and mistakes (keeping recoveries) and drop episodes whose video shows the wrong task, then retraining the same model/recipe, improves sim task progress by +14.9 pts [tweet §2].
Positional thesis: naively scaling the VLM backbone alone does not translate to better VLA, citing VLM4VLA (arxiv:2601.03309) and LAP (arxiv:2602.10556); the hypothesis is that VLMs encode task intent while the action head still absorbs the complex sensorimotor mapping — in a scarce-robot-data regime this is intractable [tweet §3-4].
Proposed factorization: let the VLM own memory and planning across the episode and break tasks into fine-grained steps so the action head reduces to a small set of atomic skills seen across many more scenes, objects, and embodiments [tweet §5].
Format choice: labels are emitted in Pantheon’s open-source Argus schema and pipeline; hindsight-annotated subtasks (annotator sees the whole episode) work well, but models asked to predict the next subtask without hindsight remain poor — citing MicroPlanBench as the external evidence [tweet §6-7].
Dataset availability: shipped at huggingface.co/datasets/nkalyanv/argus-robot-annotations [tweet §opening].

Method

The annotation pipeline reuses the Pantheon Argus harness (Argus: An Open-Source Annotator for Robotics Data) with GPT-6.1 Sol as the backbone VLM — a cost-parity near-Astra choice Argus had benchmarked at ~ $5/hour vs Astra's ~$ 27/hour with 96% event-density parity. Annotations are per-episode and dense: per-hand subgoals and actions, task outcomes, operator mistakes, recoveries, and object-state changes. For the data-cleaning experiment, Argus labels on ABC-130k are used as a filter: idle stretches and mistake segments are masked out (recoveries are retained as a distinct supervisory class), and episodes showing the wrong task are dropped entirely before retraining with an unchanged model and recipe.

Results

+14.9 pts task progress improvement in simulation on ABC-130k after Argus-driven cleaning (idle+mistake masking + wrong-task-episode drop), model and recipe held fixed [tweet §2].
One community reply (@Anishfishhh) reports GPT Sol success-labeling accuracy at only 60–70% on their own tests — flagged as a counter-datapoint not addressed by the tweet thread [tweet §reply].
No numbers released for the hierarchical VLM+atomic-skills policy architecture Kalyan proposes as the follow-up direction.

Why it’s interesting

This is the first filed instance of the Argus annotator (Argus: An Open-Source Annotator for Robotics Data) being used as a cross-dataset public data-cleaning gate at >5k-hour scale, applied to the exact substrates most open-VLA/WAM recipes draw from (ABC-130k, MolmoAct2 YAM, DROID). The Pantheon Argus release characterized what the harness detects and benchmarked VLM backbones (Astra, Sol, 6.1 Sol); this tweet converts that into a public annotation corpus plus a measured downstream training lift. The +14.9 pt sim-task-progress number from idle/mistake masking is the sharpest filed quantitative evidence yet that the “data quality splits by downstream objective” argument Argus staked (Argus: An Open-Source Annotator for Robotics Data) actually translates into VLA training gains.

The accompanying positional thesis places this release in direct tension with the backbone-scaling bet: if VLMs mostly encode task intent, then dense per-episode subgoal + outcome labels are precisely the signal needed for the hierarchical VLM-plans-atomic-head-executes factorization already prototyped by Agent as Policy for Robotic Manipulation AGP and by Sunday Robotics’ ACT-2 preview (ACT-2 Preview: Generalizing Reliability) — this release is the public data substrate for that recipe. The community reply flagging 60–70% Sol success-labeling accuracy sharpens the open question about judge-backbone sensitivity (VLM-as-Evaluator) and sits in productive tension with Argus’s own cross-model density benchmark where Sol tracked Astra within ~95% outcome agreement at 5× cost reduction. First-contact-rate parity on human-ego video (28 vs Astra’s 26 key events/episode) does not necessarily transfer to robot-episode success labeling — this is the first independent data point on the specific sub-task.