Skip to content
Bud's Wiki

Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA)

TL;DR

Section titled “TL;DR”

DOMINO is a large-scale simulation benchmark and dataset for dynamic robotic manipulation — 35 tasks with moving targets and unpredictable environmental changes at three complexity levels, ~110K expert trajectories, and a multi-dimensional evaluation suite (Success Rate + a Manipulation Score that penalizes collisions/out-of-bounds). PUMA is the paper’s proposed dynamics-aware VLA architecture that couples historical optical flow with learnable “world queries” implicitly forecasting object-centric future states, and reports a +6.3% absolute success-rate gain over baselines. The authors also show that training on DOMINO’s dynamic data transfers back to standard static manipulation tasks, arguing dynamic data is a general-purpose lever rather than a domain-specific fix. ECCV 2026; dataset, benchmark code, PUMA checkpoint, and StarVLA integration are all released Apache-2.0.

Key claims

Section titled “Key claims”
  • Mainstream VLAs are single-frame observers and fail in dynamic environments where targets move or the scene changes mid-execution — the paper attributes the gap to (a) scarcity of dynamic-manipulation datasets and (b) reliance on single-frame perception restricting spatiotemporal reasoning [Abstract].
  • DOMINO ships 35 tasks organized by “hierarchical complexity” with over 110K expert trajectories collected via a RoboTwin 2.0-based automated pipeline; dynamic behavior is controlled by four config knobs (use_dynamic, dynamic_level ∈ {1,2,3}, dynamic_coefficient, check_render_success) so static and dynamic evaluation share the same task shells [README §1.2].
  • The evaluation suite adds Manipulation Score (MS) — a route-completion metric with penalties for collisions/out-of-bounds — alongside Success Rate, plus strict success conditions (out-of-bounds detection before grasping; a lifting-height threshold to prevent false positives from accidental touches) [README §1.3].
  • PUMA integrates two dynamics-aware components on top of a Qwen3-VL-4B-Instruct-Action base: (i) scene-centric historical optical flow consumed as visual input, and (ii) learnable world queries that implicitly forecast object-centric future states inside the VLA — history-aware perception paired with short-horizon prediction [Abstract, README §“Key Idea”].
  • PUMA reports +6.3% absolute success rate over baselines on DOMINO and claims SOTA on the benchmark [Abstract].
  • Training on dynamic data fosters robust spatiotemporal representations that transfer back to static tasks — the paper positions dynamic data as a general-purpose training substrate, not a domain patch [Abstract, README §“Key Idea”].
  • DOMINO has been adopted as an OOD dynamic-manipulation benchmark by the Qwen-VLA technical report (§ News 2026/05/29), and is compatible with the StarVLA training/evaluation codebase.

Method

Section titled “Method”

The DOMINO simulator is built on RoboTwin 2.0 (SAPIEN backend, Vulkan rendering, CuRobo motion planning) and layers dynamic-environment configuration on top: at data-collection time each task template can be run with use_dynamic=true, one of three dynamic_level complexity tiers, and a dynamic_coefficient speed scale that drives moving-object trajectories. Expert trajectories are collected in HDF5 via an automated pipeline (bash collect_data.sh <task> <config> <gpu>), producing 110K+ trajectories across 35 tasks. Evaluation composes with any policy that speaks the RoboTwin 2.0 interface — the paper’s evaluation harness runs the policy as a WebSocket-connected server against the simulator.

PUMA is a predictive VLA on the Qwen3-VL-4B-Instruct-Action backbone (from StarVLA). Two additions distinguish it from a vanilla VLM+action-head recipe: (i) a Grounded-SAM-2 stack (SAM 2.1 Large + GroundingDINO Swin-T) supplies object masks that feed a scene-centric historical optical-flow branch, injecting explicit motion history into the perception path; (ii) learnable world queries attend to the language + visual context and are supervised to implicitly forecast object-centric future states without an autoregressive video head, so the coupling of history-aware perception and short-horizon prediction stays inside a single forward pass. Both training and evaluation run through the StarVLA/LeRobot-derived launch scripts in policy/PUMA/scripts/run_scripts/.

Results

Section titled “Results”
  • Headline: PUMA reports a +6.3% absolute success-rate improvement over baselines on DOMINO and claims SOTA on the benchmark [Abstract].
  • Cross-benchmark scale reference (from the InternVLA-A1.5 report, filed as InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization): DOMINO is one of six sim benchmarks where InternVLA-A1.5 reports leaderboard numbers, at 27.7 overall, indicating DOMINO is a hard suite relative to LIBERO/RoboTwin variants where numbers cluster above 80.
  • Dynamic-to-static transfer: training on DOMINO’s dynamic data is reported to fortify spatiotemporal representations that transfer to standard static tasks [Abstract] — no static-task numbers are quoted in the abstract or README at filing.
  • Ablation surface exposed to reviewers: dynamic complexity levels 1/2/3 are separate columns in the released config, so per-level Success Rate + Manipulation Score can be reported; a training-config vs evaluation-config matrix is baked into the eval CLI (ckpt_setting and task_config are separate flags).

Why it’s interesting

Section titled “Why it’s interesting”

DOMINO occupies a specific gap in the VLA benchmark landscape tracked in VLA Models: LIBERO / RoboTwin 2.0 / SimplerEnv are largely quasi-static tabletop suites where the scene is fixed at policy-rollout start, so success rates on those benchmarks have climbed into the 80–99 band while the underlying models remain single-frame observers. DOMINO’s contribution is to inject scene-native motion — moving targets, timed events — as a first-class task axis with three difficulty tiers, and the immediate consequence visible in already-filed papers is a sharp performance cliff: InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization‘s SOTA number on DOMINO (27.7) is roughly a third of its 98.9 on LIBERO. That gap is the paper’s core claim materialized in a third-party number.

PUMA itself sits alongside several recipes on World Foundation Models and VLA Models that pair a VLA with future-frame prediction — RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation co-generates RGB+depth+optical-flow via a tri-branch DiT then reads the diffusion internals into an inverse-dynamics head; Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio adds a Temporal-Ratio diagnostic on top of a flow-matching head that reads a Cosmos-Predict video prediction; Next Forcing: Causal World Modeling with Multi-Chunk Prediction trains a causal-AR world-model core with multi-chunk prediction heads; and World Pilot: Steering Vision-Language-Action Models with World-Action Priors uses a separate WFM as a dual-path steering peer. PUMA’s positioning against these is deliberately lightweight: no separate video model, no denoising loop, no external WFM — historical optical flow (Grounded-SAM-2 masks + flow) as input and a fixed set of learnable world queries as an implicit prediction bottleneck inside the VLA. It’s the closest filed analogue to the “predict the future implicitly inside the policy” strand while explicitly targeting dynamic rather than static generalization. The dataset’s dynamic-to-static transfer claim, if it holds, would also be a data-side argument for dynamic pretraining that complements the egocentric-vs-teleop debate on VLA Models.

See also

Section titled “See also”