Evaluating Gemini Robotics Policies in a Veo World Simulator

TL;DR

Google DeepMind fine-tunes Veo into an action-conditioned, multi-view robotics video simulator and uses it as a policy evaluator across the full spectrum of evaluation use cases — in-distribution performance ranking, OOD generalization probing, and predictive red-teaming for safety — for Gemini Robotics On-Device (GROD) VLA checkpoints on a bimanual manipulator. The Veo (Robotics) model takes a current image plus a sequence of future robot poses and predicts a tiled 4-camera future (top-down, side, left/right wrist), and generative image editing is used to synthesize OOD variants (novel objects, backgrounds, distractors) of real scenes. Across 1600+ real-world evaluations of eight policy checkpoints on five tasks, predicted success rates correlate with and correctly rank both policies (in-distribution) and OOD conditions, and rollouts in safety-edited scenes surface policy vulnerabilities that reproduce when re-staged with physical props.

Key claims

Veo is fine-tuned for action conditioning: given a current image observation and a sequence of future robot poses, the model predicts the corresponding sequence of future image frames [§ “Action Conditioning”].
Veo is fine-tuned for multi-view generation: the four cameras (top-down, side, left wrist, right wrist) are tiled into a single frame and Veo predicts the joint tiled future, mitigating partial-observation ambiguity [§ “Multi-view generation”].
OOD variants of real scenes are synthesized by generative image editing along three axes — novel interaction objects, novel visual backgrounds, novel distractor objects — so the same upstream Veo prior covers the OOD evaluation surface without separate domain-specific data collection [§ “Overview”].
Veo (Robotics) ranks 8 VLA checkpoints in-distribution: predicted success rates from video rollouts correlate strongly with real-world success rates, and the predicted ranking matches the real-world ranking [§ “Nominal Evaluations”].
Veo (Robotics) ranks OOD generalization axes correctly: for a fixed policy, predicted success rates over object / background / distractor edits correlate with real-world success rates, and the relative impact of each axis on policy performance is recovered [§ “OOD Evaluations”].
Predictive red-teaming surfaces real safety failures: rolling out policies in scenes edited to include safety-critical elements exposes potentially unsafe behaviors that reproduce on hardware when re-staged with physical props, without requiring the corresponding hardware safety-violation events [§ “Red Teaming”].
The system is validated at scale: 1600+ real-world evaluations across 8 Gemini Robotics policy checkpoints and 5 tasks on a bimanual manipulator [Abstract].
Limitations are explicit: multi-view consistency and realistic physical interactions remain open challenges in the current Veo (Robotics) model [§ “Limitations”].

Method

The base architecture is Google DeepMind’s Veo (the same family as the closed-source Veo 3 Tech Report), fine-tuned on a large-scale robotics dataset of diverse manipulation tasks. Two capabilities are added on top of the base T2V prior: (1) action conditioning, where future robot poses are rendered as pose-overlay images and concatenated with the initial observation as conditioning, so the network learns observation × pose-sequence → future-observation; and (2) multi-view consistency, achieved by tiling the four cameras (top-down, side, two wrist views) into a single tiled frame and fine-tuning Veo to generate the tiled future jointly. Multi-view tiling sidesteps cross-camera consistency losses by collapsing the four views into one image-grid the diffusion model can co-denoise.

For OOD evaluation the same fine-tuned Veo is composed with generative image editing: given a real scene captured by the robot, image edits introduce novel interaction objects, novel backgrounds, or novel distractors; the edited image is then fed back into Veo (Robotics) as the new initial observation, and the policy under test is rolled out in the resulting synthesized environment. The simulator and the policy interact in closed loop — the policy emits pose targets, Veo predicts the resulting observations, and the policy ingests them as its next input. Predictive red-teaming is the same loop with safety-critical scene edits (props or hazards inserted via image editing) used as the OOD condition.

Policies under test are vision–language–action (VLA) checkpoints trained on top of Gemini Robotics On-Device (GROD). Eight checkpoints × five tasks are evaluated for the ranking experiments. No parameter counts, training compute, or architectural details for the underlying Veo backbone are disclosed; the project page commits only to the action-conditioning + multi-view fine-tuning recipe at a block-diagram level.

Results

The headline result is a predictor claim, not an absolute-quality claim. The system reports:

In-distribution: across 8 VLA checkpoints, the Veo-predicted success rate strongly correlates with the real-world success rate and the predicted ranking matches the real ranking (correlation plot in the “Nominal Evaluations” section of the project page; specific R² not stated on the project page).
OOD: for a given policy, predicted success rates under object / background / distractor edits correlate with real success rates and rank the axes of generalization correctly by their impact on policy performance.
Red-teaming: examples of policy failures discovered by rolling out in image-edited scenes that re-occur on hardware when staged with physical props — qualitative case studies rather than aggregate metrics.
Scale: 1600+ real-world evaluations are used as the ground truth that the simulator is calibrated against.

The project page does not include FVD / VBench / human-preference numbers for the Veo (Robotics) outputs themselves, and explicitly flags multi-view consistency and physical-interaction realism as ongoing limitations.

Why it’s interesting

This is the closed-loop policy-evaluation counterpart to the latent-predictive WFM-as-reward pattern the wiki has been tracking. Inference-time Physics Alignment of Video Generative Models with Latent World Models uses VJEPA-2’s predictive surprise as a frozen reward signal to steer video generation toward physically plausible outputs; this paper inverts the relation — the WFM is the environment, not the reward, and the thing being scored is a downstream VLA policy, not a video sample. Both rely on the same underlying bet (the WFM’s learned dynamics is faithful enough to drive a downstream decision), but they apply it to different layers of the stack.

It also makes the Gemini Robotics 1.5 brings AI agents into the physical world / SIMA 2: A Generalist Embodied Agent for Virtual Worlds / Veo trio coherent as a single Google DeepMind robotics program: Gemini Robotics ER + 1.5 is the planner+action model, SIMA 2 puts a Gemini-derived agent inside Genie-3-generated open worlds, and this work puts Gemini Robotics policies inside Veo-generated manipulation scenes to evaluate them. The same Veo-family backbone documented at policy-light, architecture-light disclosure in Veo 3 Tech Report gets re-used here as a robotics foundation model — a concrete instance of the “post-train the closed flagship into a vertical foundation model” pattern that The Waymo World Model: A New Frontier For Autonomous Driving Simulation showed for autonomous driving (Genie 3 → Waymo camera+lidar simulator). World-model-as-driving-sim and world-model-as-manipulation-sim are now both publicly demonstrated specializations from the Veo / Genie family.

Finally, the synthetic-OOD-via-image-editing recipe is the policy-evaluation analogue of NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos‘s on-the-fly synthetic degradation pipeline: rather than collecting OOD evaluation scenes, the system manufactures them by editing real captures and rolling the policy out inside the resulting world model. This makes evaluation a generative problem, not a data-collection problem, and the validation against 1600+ real evaluations is what currently distinguishes the claim from extrapolation.