Skip to content
Bud's Wiki

VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances

TL;DR

Section titled “TL;DR”

VLAff (JSK Lab, University of Tokyo) is a vision-language model that emits unified actionable affordances — visual heatmaps of where to interact, 6-DoF grasp poses for how to grasp, and trajectory waypoints for how to move — from a single RGB observation and language instruction. It is trained on EgoAffordance, a new large-scale dataset built by running state-of-the-art 3D Structure-from-Motion + hand-mesh reconstruction over egocentric human videos to auto-extract 204K episodes with 5.6M visual affordances and 11.6M grasp+trajectory affordances. At deployment, VLAff’s outputs are converted to directly-executable robot actions via 3D scene information, giving both SOTA visual-affordance prediction on standard benchmarks and demonstrated zero-shot manipulation + affordance-guided robot learning on real hardware.

Key claims

Section titled “Key claims”
  • Object-centric actionable affordances — visual heatmaps + grasp poses + trajectories — can be extracted at scale from egocentric human videos by combining SfM 3D reconstruction with hand-mesh reconstruction, sidestepping the human-robot embodiment gap by targeting an embodiment-agnostic representation [§Abstract].
  • The EgoAffordance dataset comprises 204K episodes with 5.6M visual affordances and 11.6M grasp/trajectory affordances — the resulting corpus scale is claimed to be a load-bearing lever for downstream affordance prediction and zero-shot robot deployment [§Abstract].
  • A single unified VLM-based foundation model can jointly learn cross-modal correlations across visual, grasp, and trajectory affordance channels, rather than requiring separate models per channel [§Abstract].
  • VLAff reports SOTA performance on visual affordance prediction benchmarks [§Abstract, Results].
  • The same model’s outputs, converted via 3D scene information, drive real-robot zero-shot manipulation and affordance-guided robot learning [§Abstract, Results].

Method

Section titled “Method”

VLAff’s pipeline has two stages. First, a data-generation pipeline runs 3D SfM and hand-mesh reconstruction over egocentric human video corpora, extracting per-episode: (a) 2D visual affordance heatmaps localizing where on the object contact should happen, (b) 6-DoF grasp poses for how to grasp, and (c) 3D trajectory waypoints for how to move — producing EgoAffordance (204K episodes / 5.6M visual + 11.6M grasp+trajectory affordances). Second, VLAff is a large VLM-based foundation model that takes an RGB observation plus a natural-language instruction and jointly emits all three affordance channels, exploiting cross-modal correlations across them. At deployment, the emitted affordances are combined with 3D scene information (depth / point cloud) to yield executable robot commands, either as a zero-shot policy or as guidance signals for a downstream policy.

Results

Section titled “Results”
  • SOTA visual affordance prediction on standard benchmarks (specific numbers not in the abstract) [§Abstract].
  • Real-robot demonstrations of zero-shot manipulation and affordance-guided robot learning, driven by executing the model’s grasp poses + trajectories against 3D scene geometry [§Abstract].

Why it’s interesting

Section titled “Why it’s interesting”

VLAff stakes out a distinct interface position on the crowded human-video-to-robot axis tracked by Human-to-Robot Retargeting: instead of retargeting motion (Do as I Do: Dexterous Manipulation Data from Everyday Human Videos kinematic 4D reconstruction, Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots 3-DoF wrist translation), retargeting task intent (LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition object flow + palm pose), or retargeting task progress (Robots Acquire Manipulation Skills in Seconds from a Single Human Video (HOST) HOST), it retargets affordances — a triple of where-to-interact heatmap + grasp pose + trajectory — as the embodiment-agnostic bridge, then executes with 3D scene grounding. This complements LUCID’s short-horizon flow + palm-pose intent by keeping the grasp channel explicit (not delegated to a per-embodiment sensorimotor policy) and pushing the trajectory channel into the same unified model rather than a separate execution head, sharpening the “what is the right embodiment-invariant abstraction” open question on the Human-to-Robot Retargeting page. On the VLA Models recipe-lever board, it sits alongside Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models Embodied-R1.5’s pointing-centric VLM in preferring a spatial-primitive VLM output over a flow-matching action head, but broadens the output primitive from single points/bboxes to a heatmap + grasp + trajectory bundle — with the training substrate coming from auto-annotated human video rather than teleop.

See also

Section titled “See also”