EmbodiedEvalKit — A Unified Evaluation Framework for Embodied AI Benchmarking

TL;DR

EmbodiedEvalKit is a unified evaluation framework for embodied-AI vision-language models — bundling 25+ benchmarks (ERQA, CV-Bench, EmbSpatial, SAT, RoboSpatial, BLINK, COSMOS, VSI-Bench, OpenEQA, BOP-Ask-adjacent pose/grasp datasets, Where2Place, RefSpatial-Bench, VABench-Point, VABench-Visual-Trace, PointBench, PIO-Bench, EgoPlan2, Pixmo Points, etc.) under a single CLI, with three pluggable inference backends (vLLM, HuggingFace Transformers, hosted-API) and 14+ supported model families (GPT, Gemini, Gemini Robotics, Qwen2.5-VL / Qwen3-VL, InternVL, Molmo, RoboBrain2, VeBrain, Mimo-Embodied, Pelican-VL, Magma, and the authors’ own Embodied-R1 / Embodied-R1.5). All benchmark data is reformatted into a standardized HuggingFace Parquet layout, and a backbone abstraction handles per-model coordinate-system conventions (0–1000 normalized for Qwen3-VL / Gemini-2.5; absolute pixels for Qwen2.5-VL / MiMo; 0–100 in XML for Molmo; 0–1 normalized for GPT / Pelican / InternVL / Magma) and prompt formats so one eval_<benchmark>.py script runs against any backend.

Key claims

Single unified pointing-evaluation interface across 25+ embodied benchmarks, with coordinate-system handling built into the --backbone flag rather than each benchmark’s eval script [README §Supported Benchmarks, §What is Backbone].
Three inference backends switchable via config: vLLM with tensor parallelism for locally-deployed open models, HuggingFace Transformers for arbitrary HF checkpoints, and a generic OpenAI-compatible API client for GPT/Gemini/custom endpoints [README §Backend Description, §Quick Start].
14+ model families officially supported out of the box, including frontier closed APIs (GPT-5, Gemini 2.5, Gemini Robotics), generalist open VLMs (Qwen2.5/3-VL, InternVL, Molmo), and embodied specialists (RoboBrain2-7B, VeBrain, MiMo-Embodied, Pelican-VL, Magma-8B, Embodied-R1, Embodied-R1.5) [README §Model List].
All bundled benchmarks have been reformatted into HuggingFace Parquet and re-uploaded under the IffYuan/ namespace (or pointed at upstream copies when feasible — FlagEval, nyu-visionx, chanhee-luke, BLINK-Benchmark, MINT-SJTU, VLABench) [README §Supported Benchmarks].
Adding a new benchmark is documented as a 3-step recipe: subclass BaseDataset with seven required methods (get_default_instruct, load_dataset, prepare_dataset, process_raw_output, evaluate_results, compute_statistics, save_results), copy an existing eval_<x>.py script, and register the benchmark [README §Adding New Benchmarks].
Paired with the Embodied-R1.5 model release (9B image-text-to-text checkpoint + SFT-Dataset + RFT-Dataset + VLA-SIMPLER + VLA-LIBERO splits) in the IffYuan/Embodied-R1.5 HF collection — the eval kit is the substrate against which R1.5 was tuned and reported [HF collection IffYuan/embodied-r15].
Sits in the lineage of Embodied-R1 (ICLR 2026, arxiv 2508.13998), which introduced the “pointing-as-unified-embodiment-agnostic-intermediate-representation” thesis and the two-stage RFT curriculum that R1.5 extends — citations include Yuan et al. 2025 From Seeing to Doing and Yuan et al. 2026 Embodied-R1.5 [README §Citation].

Method

The framework is organized around three orthogonal axes — Model × Backend × Benchmark — connected by a backbone abstraction that handles the messiest part of cross-model evaluation: coordinate-system conventions. The author’s note that Qwen3-VL and Gemini-2.5 emit [500, 500] (0–1000 normalized), Qwen2.5-VL emits [256, 256] (absolute pixels), Molmo emits <point x="50" y="50"> (0–100 in XML), and GPT/Pelican/InternVL/Magma emit (0.5, 0.5) (0–1 normalized) — and the same benchmark scoring code has to handle all four without per-model branching. The backbone field auto-detects from model_name and routes to the appropriate prompt template, output parser, and metric extractor.

Each benchmark subclass implements a thin set of methods: load from HF, transform into a standardized {'question', 'image', 'metadata'} record, parse raw model output into a structured prediction, and aggregate per-sample metrics. Inference happens via a single batch_inference call to whichever backend was selected. The CLI accepts a common set of flags for tensor parallelism (--tensor_parallel_size), context length (--max_model_len), image budget (--max_images_per_prompt), and sampling (--temperature, --top_p, --top_k, --repetition_penalty) and exposes a --thinking_model flag for models that emit reasoning between special tags.

The benchmark roster covers the major axes of embodied VLM evaluation: embodied QA (ERQA, OpenEQA), spatial reasoning (EmbSpatial, SAT, RoboSpatial, VSI-Bench, CV-Bench), pointing (Where2Place, RefSpatial-Bench, VABench-Point, PointBench, Pixmo Points, PIO-Bench), affordance and grasping (Part-Affordance-2K, RoboRefit, RoboAfford-Eval, ShareRobot Trajectory), planning and long-horizon (EgoPlan2, RoboVQA, VLABench), physical commonsense (COSMOS), failure analysis (RoboFAC), and general perception (BLINK).

Results

No headline benchmark numbers are reported in the repo itself — EmbodiedEvalKit is the infrastructure, not the result. The numbers it produces show up in the Embodied-R1 paper (arxiv 2508.13998, ICLR 2026: 56.2% SIMPLEREnv success, 87.5% across 8 real-world XArm tasks, +62% over baselines) and presumably in the forthcoming Embodied-R1.5 paper (Yuan et al. 2026, currently arxiv preprint). The repo’s value is reproducibility: a third party can run bash scripts/eval_embodied_r1.5.sh on 2 GPUs and reproduce the headline numbers against the same 25 benchmarks, against the same coordinate conventions, on the same Parquet-standardized data splits.

Why it’s interesting

The closest filed cousin is RL Environment Platforms — SETA, Toolathlon-GYM, OpenReward — but those are training env packages with automated graders, while EmbodiedEvalKit is purely an evaluation substrate for VLMs that emit pointing / grasping / spatial outputs. Both share the “env-as-package” pattern (Docker-or-pip-installable, single CLI, standardized config files) but EmbodiedEvalKit’s verifier asymmetry is different: the action is cheap (one forward pass per question) and the coordinate-system normalization is where most of the engineering went, not Docker sandbox sizing.

The benchmark roster is also the most concrete map yet of what VLM Perception Failures consumes: BLINK, EmbSpatial, BOP-Ask-adjacent pose/grasp datasets, VSI-Bench, ERQA, and SAT are exactly the benchmarks where frontier VLMs collapse on pixel-precise 3D-grounded output formats (cf. BOP-Ask: Object-Interaction Reasoning for Vision-Language Models, Cambrian-S: Towards Spatial Supersensing in Video, Solving Spatial Supersensing Without Spatial Supersensing). The kit makes those failure modes pluggable: one CLI run gives 25 numbers across all 14+ supported model families. It complements CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (CaP-X, a coding-agent-for-robot-manipulation benchmark) by sitting one rung up the stack — evaluating perception, pointing, and spatial reasoning rather than full agentic robot-code generation. And it’s a natural eval target for the next wave of pointing-centric VLMs that MolmoPoint: Better pointing architecture for vision-language models (MolmoPoint) and Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding (Molmo2) point toward.