Behavior Prompting Policy: Demonstrations as Prompts for Manipulation

ListenDemonstrations as Prompts: Teaching Robots On the Fly11:39

Show notes

A look at Behavior Prompting Policy (BPP) from Stanford's REAL Lab, which lets a robot learn a new task at inference time from a single human demonstration — no fine-tuning, no gradient updates. We dig into the architecture, the finding that task *diversity* beats per-task quantity, the controlled ablations on procedurally generated benchmarks, and where language conditioning still wins. Plus the obvious parallels for how we think about in-context conditioning in generative models. **Sources** - Behavior Prompting Policy: Demonstrations as Prompts for Manipulation (Austin Patel, Ben Pekarek, Joel Enrique Castro Hernandez, Shuran Song)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Behavior Prompting Policy (BPP) is an in-context visuomotor policy from Austin Patel, Ben Pekarek, Joel Enrique Castro Hernandez, and Shuran Song (REAL Lab, Stanford) that specifies tasks at inference time via a single behavior prompt — a full sequence of observations, proprioception, and actions from a human demonstration in the same sensorimotor space as the robot. Unlike language or goal-image conditioning, the prompt carries dense spatial-temporal cues; BPP encodes it once per rollout, cross-attends against the current observation, and passes the extracted information to a diffusion action decoder. The paper contributes an algorithm (BPP), a hardware interface (iPhUMI, an iPhone-based UMI extension with ARKit SLAM + wireless prompt streaming), and two benchmarks (DrawAnything and LIBERO-Gen, up to ~2000 procedurally-generated tasks) that isolate test-time adaptation to new low-level actions rather than just new objects. Key finding: task diversity, not per-task quantity, drives ICL emergence — few demos across many tasks beats many demos across few tasks at fixed budget.

Key claims

In-context task adaptation via a single sensorimotor demonstration, with no gradient updates and no meta-learning objective at training time; the prompt is provided as an additional model input and the model learns end-to-end how to leverage it [§1, §3.3].
Prompt encoder attention functions as a temporal lookup + spatial-difference-resolution operation: on DrawAnything, attention closely tracks task progression, giving step-by-step dense sub-goal guidance; on LIBERO-Gen, attention follows a more discrete pattern that identifies next key events / transitions between primitives [Fig. 5, §4.1].
On DrawAnything unseen drawings, BPP reduces error by 80.7% vs a Goal-Image baseline and 33.3% vs ICRT (prior behavior-prompt model) [§4.1]. The goal-image baseline is bottlenecked at reconstructing continuous drawing motion from only the final state; ICRT is diagnosed as OOD-brittle because it retains full rollout history in-context.
On LIBERO-Gen (up to ~2000 tasks), BPP without pretraining rivals a foundation VLA (π0.5-family, cited as [5]) fine-tuned on LIBERO-Gen on both LIBERO-Gen Combination (held-out pick-place location pairs) and LIBERO-Gen Chain (held-out two-step tasks composed of individually seen primitives) [§4.1, Fig. 4d,e]. In an ablation removing 2nd-step training tasks so the model has never seen a 2nd step after a 1st step, BPP achieves +20.8 pp over Language conditioning vs +10.7 pp when 2nd-step tasks are included [Fig. 4d,e].
Task diversity dominates per-task quantity: at fixed demonstration budget, many-tasks-few-demos-per-task outperforms few-tasks-many-demos-per-task; increasing the training task count improves unseen-task adaptation even at just 5 demos/task [Fig. 6d,e].
Complexity of training tasks matters: training only on 1–3-part procedural drawings yields poor adaptation to complex unseen drawings; training on 4–6-part drawings gives the best adaptation but at proportionally higher data cost [Fig. 6f].
Prompt-representation ablations: observations are necessary to anchor the temporal lookup, actions provide useful transitions between downsampled observation timesteps, proprioception (cursor position) is redundant when already visible in the image [Fig. 6a]; observation downsampling below 1 Hz degrades performance [Fig. 6b]; attention pooling per prompt chunk (merging observation + proprioception + actions into a single embedding per timestep) outperforms separate modality tokens [Fig. 6c].
Inference-time efficiency: prompt chunk embeddings are computed once per rollout and cached; the action decoder is decoupled from the prompt encoder so multiple diffusion denoising steps do not re-reference the full prompt [§3.3, Inference].
iPhUMI as a dual-purpose interface: extends the UMI gripper by replacing the GoPro with an iPhone 15 Pro, using ARKit for instant on-device SLAM (no environment mapping step) and wirelessly streaming behavior prompts to a workstation at test time — the same device collects training data and specifies deployment prompts [§3.4].
Explicit failure mode: low training task diversity weakens BPP’s task conditioning relative to language. In the three-task laundry-folding case study, BPP sometimes completes the wrong fold or hesitates on task identity; language conditioning does not exhibit these failures because the training-set details can be memorized directly and language is a more direct identifier [§4.2, §A].
BPP has not been shown to enable one-shot execution of entirely new action primitives — the LIBERO-Gen adaptation gains are on new compositions and instructions over primitives the training set already contains [§4.1, §4 preamble].

Method

BPP consists of a prompt encoder and a diffusion action decoder. The prompt is temporally downsampled (typically 1 Hz for observations and proprioception; actions kept at full rate) and split into per-timestep chunks, each aggregated by attention pooling into one chunk embedding — this both aligns modalities from the same timestep and shortens the prompt sequence. A transformer decoder then cross-attends the current observation (tokenized one token per history timestep) against the pooled prompt chunks with learned positional embeddings on both sides, producing an extracted-prompt representation. The action decoder concatenates the current observation, this extracted representation, and the diffusion timestep, then applies a CNN action-diffusion architecture with FiLM (from Chi et al. 2023) to iteratively denoise a chunk of future actions.

Training is standard flow/action-diffusion behavior cloning with one modification: each step samples a single demonstration from the training data as the prompt, then loads a batch of receding-horizon observations + future-action chunks from other demonstrations of the same task, so the policy must reason over spatial and temporal differences between the prompt and the current observation. No explicit correspondence between prompt and rollout is required, and no additional data beyond an existing multi-task imitation dataset. Task groupings define prompt granularity — two grasping strategies for the same object need two task labels for the prompt to select between them.

iPhUMI extends the UMI gripper [1] by swapping the GoPro for an iPhone 15 Pro to gain ARKit-based instant localization (skipping the UMI SLAM mapping step) and a wireless app that streams a captured behavior prompt to a workstation at deployment.

Results

DrawAnything-Sim (2000 train tasks × 5 demos, 50 unseen human-collected drawings): BPP achieves 80.7% error reduction vs Goal-Image and 33.3% reduction vs ICRT [23] on unseen drawings; a Goal-Image policy fails on new drawings while BPP recovers most of them [§4.1].
DrawAnything-Real (ARX arm + iPhone wrist camera, 1000 training tasks split human/scripted, 10 eval tasks 4 seen/6 unseen): BPP reconstructs unseen drawings from a single iPhUMI demo under full 6-DoF real-world action [§4 preamble, §4.1].
LIBERO-Gen Combination (10 held-out pick-place location pairs seen individually): BPP improves over Language and Goal-Image conditioning; matches finetuned π0.5-family VLA baseline despite no foundation pretraining [Fig. 4c].
LIBERO-Gen Chain (10 held-out two-step tasks, primitives include pick-place + open drawer + push plate + turn stove): BPP +10.7 pp over Language in the full-training regime, +20.8 pp in the ablation regime where 2nd-step tasks are removed from training [Fig. 4d,e].
Laundry folding (3 real bimanual tasks, iPhUMI prompts): BPP successfully selects and executes the correct fold from a single human demo across the three training tasks; however, exhibits weaker task discrimination than language conditioning in this low-diversity regime, with documented wrong-fold and hesitation failures [§4.2].
Data ablations (DrawAnything-Sim): many-tasks-few-demos > few-tasks-many-demos at fixed budget; task count scales unseen success at 5 demos/task; training on 4–6-part drawings only gives best unseen performance but at cost [Fig. 6d–f].
Prompt representation ablations: multi-modal + ≥1 Hz observations + attention pooling per chunk are each load-bearing; removing observations breaks prompt lookup, removing actions removes useful temporal transitions [Fig. 6a–c].

Why it’s interesting

BPP is the third filed one-shot / ICL-native manipulation entry alongside GEN-1.5: Embodied Foundation Models are One-Shot Learners and S1: In-Context Learning for Robotics, and stakes out a distinctly architectural position on the ICL-for-manipulation axis: rather than betting on emergent ICL from pretraining scale (GEN-1.5’s “engine-scale emergence” thesis, no ICL-specific architecture) or on data + curriculum at scale (S1’s video-in / action-out policy trained on demonstration-conditioned episodes), BPP identifies specific mechanistic ingredients — attention-pooled per-timestep chunk embeddings, cross-attention against the current observation, decoupled prompt-vs-diffusion loops — and validates them via controlled ablations on procedurally-generated benchmarks. Its diagnosis that ICRT’s full-rollout-history context causes OOD spurious correlations is the sharpest filed critique of the transformer-context-with-history recipe.

Complements S1: In-Context Learning for Robotics‘s two-axis framing (task horizon × OOD novelty): BPP explicitly disclaims one-shot execution of entirely new action primitives and confines its adaptation claim to compositional new tasks over seen primitives — the low-diversity laundry-folding failure mode is a direct concrete instance of the “task diversity threshold” S1 argues for from the opposite (scaling-law) direction. BPP’s controlled ablations on task-diversity-vs-quantity are the mechanistic complement to S1’s 1k→100k-hour scaling result. Sibling in the “ICL substitute for retargeting” position on Human-to-Robot Retargeting — the human demo drives the robot in a single forward pass, no explicit retargeting step, with the demonstration and rollout allowed to differ in environment configuration (though same environment). Also plugs into Synthetic Training Data via LIBERO-Gen, whose procedural task-generation framework produces ~2000 tasks + demonstrations to make the diversity ablations tractable — a benchmark-side answer to “how do we study ICL without industrial-scale data collection.”

Load-bearing caveat: the diversity threshold problem BPP names explicitly (laundry-folding hesitation) is exactly the regime where language conditioning wins, so the “unified task descriptor” future-work suggestion is doing real work — the current filed evidence supports BPP’s diversity-lever position rather than its fully-general replacement for language.