Segmenting Robot Video into Actionable Subtasks (WGO-Bench)

TL;DR

Macrodata Labs benchmarks VLM-based subtask annotation of egocentric and robot videos with WGO-Bench, a 100-episode (743-segment, 62-task, 71.5-minute) test set spanning HomER egocentric, DROID third-person, and Galaxea head-camera footage. Across 60+ experiments they find Gemini 3.5 Flash on a contact-sheet representation with visual-timestamp cues is the dominant recipe — reaching 0.306 F1 on segmentation, 61.0% accuracy on labeling (gold boundaries given), and 0.168 F1 end-to-end — while running at $2.64 per hour of video, roughly 19× cheaper than human annotation. Subtasks are a growing supervision signal in VLA work (RT-H, π0.5, SARM), and the open pipeline is shipped in Refiner.

Key claims

Best end-to-end pipeline: contact sheets of 20 frames in a 4×5 grid sampled at 0.5 s, 224 px tiles, visual timestamps drawn directly on each tile, Gemini 3.5 Flash with a GEPA-tuned “completed-events” prompt — 0.306 segmentation F1, 61.0% labeling accuracy, 0.168 end-to-end F1 [§“Final segmentation recipe”; §“Measuring Progress: WGO-Bench”].
Visual timestamps engraved on the contact sheet outperform text timestamp maps: F1 climbed from 0.201 (contact sheets with text-map timestamps) to 0.263 (in-tile visual timestamps); combining both, ID-style {A-Z}_{1-20} codes, or alternate render locations (side strip, large bottom box) all underperformed [§“Timestamp Cues”].
Contact sheets are not just cheaper but slightly better than per-frame timestamped images: $0.0158 vs$ 0.188 per minute of video (~12× cheaper) and 0.201 vs 0.193 F1; cost win comes from Gemini billing image tiles at 258 tokens regardless of frame count [§“Contact Sheets: Better and Cheaper”].
Gemini dominates frontier alternatives on this task — Gemini 3.5 Flash beats the best non-Gemini model (GPT-5.5) by 24.5%; Gemini Robotics ER unexpectedly does not outperform general Gemini despite its spatial/robotics tuning; open-source models tend to over-predict segments [§“The Model Matters”].
Embedding-similarity boundary detection underperforms: caption-embedding cosine-drop (the prior Suzuki et al. 2026 recipe) reaches only 0.081 F1; skipping captions and embedding frames directly with Gemini Embedding 2 collapses to 0.007 F1 — direct VLM prompting on the raw video beats embedding heuristics by 3-4× [§“Splitting Videos by Similarity”].
GEPA prompt search tightens granularity without changing event vocabulary: the best prompt predicts 550 segments vs. the base prompt’s 645 by suppressing approach/grasp-adjustment/retreat boundaries, raising F1 by removing incidental-motion splits [§“Prompt Search with GEPA”].
Input decomposition hurts: splitting the episode into per-contact-sheet calls with overlap and last-segment context made the model treat artificial split points as real boundaries, despite the expectation that smaller contexts would help [§“Input Decomposition Experiments”].
Cost: $2.64 per hour of video at Gemini batch pricing, ~19× cheaper than human annotation, with the full pipeline shipped in the open-source Refiner repo [§TL;DR].
Benchmark composition is heterogeneous on purpose: 25 HomER egocentric (1280×720+) episodes / 470 segments, 50 DROID third-person robot-camera (320×180) / 150 segments, 25 Galaxea head-camera (1280×720) / 123 segments — letting the same recipes be tested across both human-hand egocentric and robot-camera distributions [§“Benchmark composition”].
LLM-as-judge label evaluation, Scale-style: Gemini-3.5-Flash judges whether a predicted label matches the gold label on action / object / source-destination correctness, with a strict JSON rubric (no exact-string matching) [§“Grading”; §“Prompt: Label judge rubric”].

Method

Three task formulations are decoupled and graded separately: (a) segmentation, predicting timestamped boundaries, scored by Segment F1 with IoU ≥ 0.75 and snapping the outermost predicted endpoints to the gold endpoints; (b) labeling, naming each gold segment, scored by an LLM-as-judge rubric; (c) end-to-end, where a predicted segment must both align temporally and pass the labeling check.

The dominant representation is a contact sheet: 20 frames from a 10-second window arranged in a 4-row × 5-column grid at 224 px per tile, with the timestamp drawn into the top-left of each tile. Gemini receives all sheets in a single call (up to ~18 sheets / ~343 frames / ~20k tokens for the longest episodes) with an instruction that defines what counts as a completed manipulation event and what should not be split (approach, grasp adjustment, retreat, hesitation, tiny repositioning). The prompt was refined via GEPA on a separately-annotated 15-episode validation set to a “completed-events with duration prior” variant.

The labeling stage runs essentially the same setup on the already-bounded segment. Human annotators worked through a self-built Codex-driven custom UI (replacing CVAT/Labelbox); the protocol defines atomic manipulation events bounded by object-state changes (held / released / new location / lid-or-door change), with labels self-contained (object + target location, no dependence on past actions).

Results

Segmentation F1 progression across 54 experiments: fixed-length 5.77 s baseline 0.070 → caption-embedding cosine-drop 0.081 → direct Gemini-Embedding-2 frame embeddings 0.007 (worse) → per-frame timestamped images 0.193 → contact sheets with text timestamps 0.201 → contact sheets with visual timestamps 0.263 → contact sheets with visual timestamps + GEPA-found prompt 0.306 [§“Final segmentation recipe”].
Labeling: best gold-boundary accuracy 61.0% (Gemini 3.5 Flash) [§TL;DR].
End-to-end: best F1 0.168 [§TL;DR].
Cross-model gap: best Gemini > best non-Gemini (GPT-5.5) by 24.5%; Gemini Robotics ER does not exceed the general Gemini model in this setup; open-source models over-segment [§“The Model Matters”].
Cost: $2.64 per hour of video (batch pricing), ≈19× cheaper than human annotation [§TL;DR].

Why it’s interesting

This is the cleanest public example so far of the labels-on-real-video pattern (Synthetic Training Data) applied to the boundary problem rather than the caption problem — Scale’s dense-video-captioning lineage assumes pre-segmented clips, and so does Action100M: A Large-scale Video Action Dataset‘s 147M-annotation HowTo100M pipeline. Macrodata’s WGO-Bench numbers say boundary discovery is the harder half (0.306 F1 vs 61.0% labeling accuracy) and identify a non-obvious recipe — visual timestamps printed onto a contact sheet beat textual timestamp maps even though VLM literature has consistently warned against relying on rendered text.

It also gives the VLA Models cluster a concrete data-engine answer to a question several filed papers leave open: π0.5, RT-H, and the SARM reward-modeling work all use subtask labels but don’t say how to mine them at scale, and the wiki’s filed VLA pages (e.g. What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?, HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining) emphasize that automated labeling of in-the-wild human video is the actual bottleneck for human-to-robot pretraining. A $2.64/hour subtask pipeline that transfers across egocentric and robot-camera views is a direct input to those recipes. The use of Gemini-as-judge for label evaluation also lands cleanly in VLM-as-Evaluator — same Scale-style rubric pattern as BaseReward: A Strong Baseline for Multimodal Reward Model and EdiVal-Agent: An Object-Centric Framework for Automated, Scalable, Fine-Grained Evaluation of Multi-Turn Editing, now applied to per-segment subtask labels rather than image-edit quality.