Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs

ListenThe Judge That Cracks on Video10:49

Show notes

Kite ML puts five frontier VLMs head-to-head as auto-judges for robot-policy episode success, across sixteen real-robot datasets and two input forms — four keyframes versus full video. Two surprises: accuracy inverts by model family (Gemini crushes keyframes but craters on video; reasoning models hold steady), and cost inverts too (native video is cheapest for Gemini, ~10× pricier for models that read it as stacked frames). We dig into the manufactured-failures trick, the bigger-isn't-better result, and why this is a cheap recipe for building evals when you only have success demos. **Sources** - Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs — Raul Romero (blog)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Kite ML benchmarks five frontier VLMs (Gemini 3.1 Pro, Gemini 3.6 Flash, Claude Opus 5, GPT-5.6 Sol, Kimi 3) as auto-judges for robot-policy episode success across 16 real-robot manipulation datasets. Episodes are graded in two input forms — four evenly-spaced keyframes vs full native video (or 16 stacked frames for models that don’t accept video) — against a human-reviewed ground truth mixing genuine successes with deliberately manufactured failures (truncated, shuffled, reversed, or task-swapped demos). Two surprises: (1) accuracy inverts by model family — Gemini judges are strong on keyframes (Flash 0.94, Pro 0.88) but crater on video (Flash 0.63, Pro 0.59), while reasoning models (GPT-5.6 Sol 0.90 → 0.92, Kimi 3 0.86 → 0.84, Claude Opus 5 0.76 → 0.73) are steady or improve; (2) cost also inverts — Gemini is cheapest on video ($0.0005/episode) since it ingests natively, while reasoning models pay ~10× more on video because they consume it as ~16 separate frames. On the keyframes efficient frontier only Kimi 3 (cheapest reasoning judge) and Gemini 3.6 Flash (highest accuracy) survive; Claude Opus 5 is strictly dominated.

Key claims

Real robot-policy evaluation today is bottlenecked by humans-with-spreadsheets, and open-source robot datasets are almost entirely success demonstrations — so a judge benchmark has to manufacture failures by truncating, shuffling, reversing, or task-swapping genuine demos to generate any negatives to grade against [§What the judge sees].
Some public open-source dataset task labels are wrong (specifically lerobot/aloha_static_towel labels its episodes as “cleaning up a spill” when the video shows no spill and the robot is placing a paper towel next to a tipped-over can); grading a judge against the shipped label makes it fail every episode, so labels were re-derived from the video before benchmarking [§What the judge sees].
There is no single best judge across input forms — Gemini 3.6 Flash is the strongest keyframe reader at 0.94 accuracy but drops to 0.63 on video; GPT-5.6 Sol is 0.90 on keyframes and 0.92 on video (the best of anyone on video) [§The benchmark, Table].
Bigger isn’t better within a family: Gemini 3.6 Flash beats Gemini 3.1 Pro on both keyframes (0.94 vs 0.88) and video (0.63 vs 0.59) — the smaller newer model is simply the better judge, so paying for “Pro” bought worse evaluations [§The benchmark].
Reasoning models (GPT-5.6 Sol, Kimi 3, Claude Opus 5) stay within a few points across keyframes and video and can even improve on video, while Gemini judges lose the thread when handed dense video and drop 25–30 accuracy points [§The first surprise].
Cost direction depends entirely on native video ingestion: Gemini charges ~ $0.0005/episode for a single native MP4 vs ~10× more for a few full-resolution keyframes, while Claude Opus 5 jumps from$ 0.0095 (keyframes) to $0.0551 (video) and GPT-5.6 Sol from$ 0.0078 to $0.0538 because both consume video as ~16 separate frames; Kimi 3 is the cheapest reasoning judge either way [§The second surprise].
On the keyframes accuracy-vs-cost plot only two judges sit on the efficient frontier — Kimi 3 (cheapest) and Gemini 3.6 Flash (most accurate); everything else is dominated, and Claude Opus 5 pays the most to score the worst [§The second surprise].
Practical routing: Kimi 3 for most cases (near-best accuracy at fraction of the cost), GPT-5.6 Sol when highest accuracy matters and video ingest cost is acceptable, Gemini 3.6 Flash for cheap keyframes-only workflows [§The second surprise].

Method

The benchmark hands each of five frontier VLMs a single yes/no question — “was the task accomplished?” — paired with the exact instruction the underlying dataset shipped (post-relabel where labels were wrong). Ground truth was assembled by hand-reviewing 16 open datasets (LeRobot / HuggingFace) covering coffee-making, tape-cutting, paper-towel placement, and similar tabletop manipulation tasks. Because the source data is almost pure success demos, failures were manufactured by truncating episodes partway through, shuffling frames, reversing time, or swapping the task instruction to something the robot did not do — giving the judge a balanced set to grade against. Each episode is presented to the judge in one of two forms: (a) keyframes — four evenly-spaced stills including the first and last frame; (b) video — a single natively-encoded MP4 for models that accept video (Gemini) or a dense ~16-frame stack for the reasoning models. Same battery, both forms, five judges, scored on accuracy against the human ground truth and on API cost per episode.

Results

Headline accuracy per (judge × input form):

Judge	Keyframes	Video
Gemini 3.1 Pro	0.88	0.59
Gemini 3.6 Flash	0.94	0.63
Claude Opus 5	0.76	0.73
GPT-5.6 Sol	0.90	0.92
Kimi 3	0.86	0.84

Best keyframe judge: Gemini 3.6 Flash at 0.94; best video judge: GPT-5.6 Sol at 0.92; strongest cross-form all-rounder: GPT-5.6 Sol averaging 0.91 across the two [§The benchmark].
Cost per episode: Gemini native video ≈ $0.0005; Claude Opus 5$ 0.0095 (keyframes) → $0.0551 (video); GPT-5.6 Sol$ 0.0078 → $0.0538; Kimi 3 cheapest reasoning judge either form [§The second surprise].
Efficient frontier on keyframes: only Kimi 3 and Gemini 3.6 Flash; Claude Opus 5 strictly dominated (highest cost, lowest accuracy) [§The second surprise].

Data, code, per-episode judgments, and the cost model are open-sourced at github.com/kite-ml/robot-eval-bench.

Why it’s interesting

This is the first filed benchmark that (a) puts frontier VLM judges head-to-head on a robot-policy episode success task with a balanced positive/negative set, (b) varies the input form (keyframes vs native video vs stacked frames) as an explicit axis, and (c) reports both accuracy and dollar cost per episode with an efficient-frontier analysis. It directly extends VLM-as-Evaluator into the robot-manipulation setting — sibling to Instance Labs — Verifying Robot Learning Episode Success (productized per-episode success verdicts, no calibration numbers) and TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics (open-source VLM dense-reward via affirmative-token log-prob, also finds backbone-sensitive rankings). The bigger-isn’t-better and family-inverts-by-input-form findings both sharpen IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation‘s judge-backbone-sensitivity result: Gemini’s dense-video collapse mirrors that page’s observation that most open-source MLLMs cluster below r = 0.5 against humans. The manufactured-failures methodology (truncate/shuffle/reverse/task-swap real success demos) is also a cheap general recipe for grading any success detector when only positive demonstrations exist — worth borrowing when building evals for VLA Models rollouts under RL Environment Platforms.