BOP-Ask: Object-Interaction Reasoning for Vision-Language Models

TL;DR

BOP-Ask is a 150k-image, 33.8M-QA dataset for object-interaction reasoning in VLMs, derived from the BOP family of 6D pose-estimation datasets (HOPE, HANDAL, YCB-V, LineMOD). Where prior spatial-reasoning benchmarks test high-level relations (“left of”, “behind”) with multiple-choice or yes/no answers, BOP-Ask asks for pixel-precise outputs across six skills: 3D cuboid pose, parallel-jaw grasp keypoints, RRT-planned collision-free trajectories, declutter-rearrangement order, plus binary spatial and relative-depth questions. Frontier VLMs (GPT-5, Gemini Robotics-ER 1.5, RoboRefer, Molmo, Qwen-VL 2.5, NVILA) score far below the human baseline on the grounded tasks, and a 2B NVILA fine-tuned on BOP-Ask reaches 77.4 3D-IoU and 50.8% trajectory success — beating the proprietary Gemini Robotics-ER 1.5 baseline at a fraction of the size.

Key claims

BOP-Ask defines object-interaction reasoning as a six-skill set: 3D object pose, parallel-jaw grasp pose (5-keypoint representation), inter-object collision-free motion trajectories, object rearrangement / decluttering order, binary spatial relations, and binary relative depth [§3.1].
The data pipeline grounds annotations in BOP 6D poses rather than monocular-depth approximations: a planar support surface is fit via RANSAC after pointing-based localization with Molmo, the plane normal sets the world up-axis, and all geometry is expressed in a consistent world frame [§3.2.1].
Trajectories are synthesized by an RRT planner in 3D Cartesian space with 10% goal-biasing, mesh-collision-filtered against neighboring objects, then simplified by the Ramer-Douglas-Peucker algorithm [§3.2.2].
Grasps come from M2T2 (transformer parallel-gripper grasp predictor) with dual global+object-centric sampling; the top-5 grasps per object are retained, and objects whose all-predicted-grasps collide are labeled “fully cluttered” — this label is the basis for the rearrangement task [§3.2.2].
The final dataset is 33.8M QA pairs split as 32.0% relative-depth / 16.0% grasps / 16.0% trajectories / 16.0% spatial reasoning / 12.4% poses / 7.6% rearrangement [Table 2].
Two held-out benchmarks accompany the training corpus: BOP-Ask-core (688 manually-verified VQAs from held-out BOP scenes) and BOP-Ask-lab (240 VQAs from 15 new lab images outside BOP, for OOD generalization) [§4].
GPT-5 collapses on the grounded tasks: 9.0 3D-IoU on pose, 0% trajectory success rate, infinite distance error on trajectories, infinite NCE on grasps; Gemini Robotics-ER 1.5 is the strongest proprietary model at 24.4 3D-IoU / 43.0% trajectory success / 138.4 distance error / 4.2 NCE [Table, §5].
Humans achieve 54.2 3D-IoU on pose, 67.3% trajectory SR, 84.9% spatial, 87.3% relative depth, 44.1% object rearrangement recall — non-trivial but not ceiling, confirming the benchmark is hard even for humans on cluttered scenes [§5.2].
Fine-tuning NVILA-2B on BOP-Ask lifts pose from 6.5 → 77.4 3D-IoU, trajectory SR from 0 → 50.8%, grasp NCE from 8.2 → 1.69, spatial from 65.0 → 94.2%, relative depth from 52.5 → 94.6%, and rearrangement recall from 8.3 → 56.4% [Table].
The fine-tuned NVILA-2B beats Gemini Robotics-ER 1.5 on every metric and beats human pose accuracy by 23.2 IoU points — a small open model on a vertical synthetic dataset surpasses both the frontier proprietary baseline and humans on geometric grounding [Table].
Fine-tuning on BOP-Ask transfers to out-of-domain spatial benchmarks (RoboSpatial-Home, SpatialBench, CV-Bench) — the dataset is not narrowly overfit to BOP scenes [§1].
Real-world robot deployment validates the VLM-as-grounded-perception use: a Franka arm with a ZED2 stereo camera uses the fine-tuned model for visual grounding, grasp selection, and motion planning across multi-step manipulation tasks [Fig. 3, §5].

Method

The contribution is fundamentally a data pipeline. Inputs are RGB-D images with 6D ground-truth poses from BOP datasets and camera intrinsics; outputs are (image, question, answer, task) tuples. The pipeline has three stages. First, world-frame reconstruction: a VLM (Molmo) points at the table surface, RANSAC fits a plane to the corresponding 3D points, the plane normal defines the world up-axis (via Rodrigues’ rotation formula), and the translation aligns the plane with the world origin. Second, geometric-prior generation: 3D cuboid boxes from object pose + model size, RRT-planned 3D motion paths between object pairs (Ramer-Douglas-Peucker simplified, mesh-collision filtered), and M2T2-predicted parallel-jaw grasps with top-5 retention. Third, QA-pair generation: rendered 3D models go through a VLM captioner to get per-object shape/color/size/utility descriptions (human-verified); per-task templates {TASK_TYPE} {OBJECT A} {OBJECT B} are filled by an LLM with in-context examples; multi-instance disambiguation uses computed positional attributes (“leftmost”, “rightmost”, “topmost”, “bottommost”) on the 2D bounding-box centroids.

Outputs are ordered 2D pixel-keypoint lists for the geometric tasks (poses, grasps, trajectories, rearrangement) and binary yes/no answers for spatial reasoning and relative depth. The grasp representation departs from rectangle-based or single-point affordance formulations: five keypoints (grasp center, left/right finger bases, left/right finger tips) capture both position and orientation, and the normalized coordinate error (NCE) metric averages the gripper-width-normalized Euclidean distance over those five points. For training, the team fine-tunes Qwen-VL 2.5 (3B) and NVILA (2B, 15B) with each model’s official codebase on 8×A100s using default hyperparameters.

Results

Pose 3D-IoU: Human 54.2 / GPT-5 9.0 / Gemini Robotics-ER 1.5 24.4 / RoboRefer 34.3 / Qwen-VL 2.5-SFT 48.2 / NVILA-2B-SFT 77.4 / NVILA-15B-SFT 73.5 [Table].
Trajectory success rate: Human 67.3 / GPT-5 0 / Gemini Robotics-ER 1.5 43.0 / NVILA-2B-SFT 50.8 / NVILA-15B-SFT 64.2; distance error: Human 112.4 / Gemini 138.4 / NVILA-2B-SFT 78.5 / NVILA-15B-SFT 77.4 [Table].
Grasp NCE (lower better): Human 1.1 / Gemini Robotics-ER 1.5 4.2 / GPT-5 ∞ / NVILA-2B-SFT 1.69 / NVILA-15B-SFT 1.40 [Table].
Spatial reasoning SR: Human 84.9 / GPT-5 68.3 / Gemini 84.2 / RoboRefer 81.7 / NVILA-2B-SFT 94.2 / NVILA-15B-SFT 95.8 [Table].
Relative depth SR: Human 87.3 / GPT-5 74.6 / Gemini 88.0 / NVILA-2B-SFT 94.6 / NVILA-15B-SFT 94.6 [Table].
Object rearrangement recall: Human 44.1 / GPT-5 14.8 / Gemini 48.9 / NVILA-2B-SFT 56.4 / NVILA-15B-SFT 57.7 [Table].
1037 human responses gathered from 40+ contributors on all 688 BOP-Ask-core VQAs, with each annotator covering 10-25 questions [§5.2].

Why it’s interesting

BOP-Ask sharpens the diagnosis already on file at VLM Perception Failures: it isn’t that VLMs are bad at “spatial reasoning” in the abstract — they’re bad at the pixel-precise, 3D-grounded, multi-step version of it, which is exactly what robotics needs. The GPT-5 “0% / ∞ / ∞” failure across the geometric subtasks (pose / trajectory / grasp) is the strongest published evidence on the wiki that frontier proprietary VLMs degrade not just on out-of-distribution spatial perception but on the specific output format (ordered 2D keypoint sequences with millimeter-level precision) that downstream embodied stacks consume. The result complements Cambrian-S: Towards Spatial Supersensing in Video‘s diagnosis at the temporal/scene-scale end of spatial cognition — Cambrian-S found long-horizon spatial supersensing untouched by data scaling; BOP-Ask finds short-horizon pixel-grounded interaction reasoning is at least partially trainable from synthetic 6D-pose-grounded data, with a 2B model beating the Gemini Robotics-ER 1.5 baseline.

It is also a clean public template for the “BOP-style precise 3D annotations → automated VQA synthesis” pattern that complements UrbanLLaVA’s per-modality recipe (UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding) and SpatialVID’s annotated-video approach (SpatialVID: A Large-Scale Video Dataset with Spatial Annotations) — three different vertical recipes for grounding general MLLMs in domain-specific geometric signal. For Luma, the takeaway is that pixel-keypoint outputs from VLMs are a non-trivial skill that does not come for free from frontier scale; if any pipeline asks a closed-source VLM for precise 2D/3D coordinates, BOP-Ask’s “GPT-5 = ∞” results are the calibration data point worth keeping in mind.