ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

TL;DR

ZeroDex (Seoul National University) is a zero-shot, training-free framework for long-horizon dexterous manipulation that bypasses end-to-end VLA policies entirely: a VLM produces reference-frame task grounding and primitive-level 2D keypoints from calibrated multi-view RGB images, and a multi-view fusion step lifts those keypoints into 3D using triangulation plus reference-view ray voting. The lifted 3D keypoints drive two execution modes: pick-and-place, where a task-conditioned affordance region around the grasp keypoint feeds an arm-hand motion generator; and tool-use, where the inferred skill category retrieves an object-centric atomic action and aligns its stored 6D tool trajectory to the scene. Long-horizon execution is closed via status verification and replanning. Real-world experiments report improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines, on unseen objects and tool-use tasks in novel scenes.

Key claims

• A pretrained VLM can ground long-horizon dexterous manipulation tasks into executable 3D plans zero-shot, with no policy training, by emitting reference-frame task grounding plus primitive-level 2D keypoints that are lifted into 3D via multi-view fusion [§Abstract].
• 3D lifting combines triangulation of view-wise VLM groundings with reference-view ray voting — searching along a semantic camera ray for geometrically consistent candidates across neighboring views — which is reported to improve grounding robustness over single-view RGB-D pipelines [§Abstract].
• A single 3D-keypoint representation supports both pick-and-place (lifted grasp keypoint expanded into a task-conditioned grasp affordance region, then handed to an arm-hand motion generator) and tool-use (inferred skill category retrieves an object-centric atomic action whose stored 6D tool trajectory is aligned to the scene) [§Abstract].
• Real-world experiments report improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines on unseen objects and tool-use tasks in novel scenes [§Abstract].
• Closed-loop status verification with replan enables long-horizon execution on multi-step household tasks (throw-away, sort-by-category, place-on-stove, pour, sweep, toss-in-wok) [§Abstract; project page demos].

Method

ZeroDex takes calibrated multi-view RGB images and a high-level language instruction and outputs an arm-hand execution plan via four stages:

1. Reference-view semantic grounding. A VLM picks a reference view and produces task grounding plus primitive-level 2D keypoints describing what to grasp and where to act.
2. Multi-view fusion 3D uplifting. View-wise VLM groundings are triangulated; in parallel, reference-view ray voting walks along the semantic camera ray from the reference view and votes for candidates that are geometrically consistent with neighboring views. This yields 3D keypoints without depth sensors.
3. Object-centric atomic-action alignment (tool-use path). For tool-use skills, the inferred skill category indexes into a library of object-centric atomic actions, each storing a 6D tool trajectory; the retrieved trajectory is aligned to the scene using the lifted 3D keypoints.
4. Affordance-guided dexterous grasp and motion generation (pick-and-place path). The lifted grasp keypoint is expanded into a task-conditioned affordance region, and an arm-hand motion generator emits feasible grasp-motion pairs for a multi-fingered hand.

Long-horizon tasks are run as a sequence of these primitives with closed-loop status verification: after each step, the system checks whether the intended sub-goal was achieved and replans if not. There is no policy training, no in-domain fine-tuning of the VLM, and no requirement for action-labeled robot data.

Results

Headline numbers are not surfaced in the abstract or project page; the paper claims qualitatively that ZeroDex (a) improves 3D grounding accuracy over single-view RGB-D grounding baselines, and (b) improves execution reliability over fine-tuned VLA baselines, both on unseen objects and tool-use tasks in novel scenes [§Abstract]. The project page demonstrates qualitative long-horizon execution on instructions like “Throw away Pepsi to a basket,” “Put the object outside the baskets into the matching basket,” “Put the pot on the stove,” “Place the apple on the mug,” “Hit something on the cutting board with hammer,” “Pour water into the dripper,” “Sweep the object on the cutting board toward the matching category object,” and “Toss the food in the wok.” Per-task quantitative success rates against named VLA baselines are deferred to a future refresh once the paper body is parsed.

Why it’s interesting

ZeroDex stakes out a fifth lever in the VLA Models design-space debate the wiki has been tracking: rather than picking between action-pretraining at scale (π*0.6: a VLA That Learns From Experience (RECAP)), clean teleop data (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), unified-VLM pointing (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models), or frozen-trace world models with small action experts (μ₀: A Scalable 3D Interaction-Trace World Model), it argues you can drop all policy training and rely on a frozen VLM plus calibrated multi-view geometry — with the claim that this beats fine-tuned VLA baselines on novel scenes. This is the closest cousin to VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation (VoLo) on the wiki — both wrap a VLM around dexterous manipulation, but VoLo treats a trained VLA/WAM as an interruptible tool inside the orchestrator’s loop, whereas ZeroDex removes the VLA entirely and substitutes multi-view 3D grounding plus a hand-coded atomic-action library. It also complements Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (DO AS I DO) by attacking the opposite end of the dexterous pipeline: DO AS I DO produces robot-complete data from human RGB videos to train policies, while ZeroDex argues you may not need to train one. The reference-view ray-voting trick directly addresses a VLM Perception Failures concern — single-view VLM groundings drift in depth — by exploiting calibrated multi-view geometry without falling back to depth sensors, which is the same failure-mode wedge GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation (GPS) attacks with a learned articulated-axis abstraction.

See also

• VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation — VoLo wraps a trained VLA/WAM as an interruptible tool under a VLM orchestrator; ZeroDex removes the VLA and substitutes multi-view 3D grounding + atomic-action library
• VLA Models — VLA design space; ZeroDex is a training-free counter-recipe to every position on that page
• Do as I Do: Dexterous Manipulation Data from Everyday Human Videos — DO AS I DO produces dexterous training data from human videos; ZeroDex argues you can skip the training step entirely on novel scenes
• GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation — GPS uses a learned 3-keypoint articulated-axis abstraction + heuristic policy as a perception-rich counter-recipe to end-to-end VLAs; ZeroDex pushes the same heuristic-policy idea further by deriving keypoints zero-shot from a VLM
• μ₀: A Scalable 3D Interaction-Trace World Model — µ₀ factors VLAs as a frozen 3D-trace world model + small action expert; ZeroDex factors further into VLM-grounding + classical motion generation with no learned action expert at all
• Tool-Use Agents — VLM-as-controller pattern over an atomic-action library; embodied counterpart to text-tool-use ReAct loops
• VLM Perception Failures — single-view VLM grounding drifts in depth; reference-view ray voting is the geometric fix