Anirudha Majumdar — pure zero-shot task inference is the stronger test of robot foundation models

TL;DR

Anirudha Majumdar (Princeton, replying to Pete Florence’s launch tweet for GEN-1.5: Embodied Foundation Models are One-Shot Learners GEN-1.5) proposes the sharper version of the in-context-learning demo: strip the physical prompt entirely — place objects in front of the robot and see whether it can infer the task zero-shot from scene composition alone. His prediction is that success rates would be “non-trivial,” which would separate the contribution of the in-context demonstration from the model’s pure zero-shot task-inference capability. He points to Russ Tedrake’s Stanford LBM talk as prior evidence that this ablation is worth running.

Key claims

The one-shot-in-context evaluation of embodied foundation models conflates two things: task-inference-from-scene (which is zero-shot) and behavior-shaping-from-demonstration (which is what the in-context example adds) [tweet body].
The proposed ablation — no in-context example, only objects on a table — would produce non-trivial success rates on current large robot policies [tweet body].
Russ Tedrake demonstrated something in this vein for Large Behavior Models (LBMs) in his 2024 Stanford talk (referenced timestamp) [tweet body].

Method

Reply-tweet in a public thread; no experiments run. The suggested experiment is: given a scene of objects placed in front of the robot with no demonstration and no natural-language instruction, sample the policy and score how often it produces a task-consistent behavior. Compare to (a) the same scene with a one-shot physical prompt and (b) the same scene with a language instruction. The delta identifies how much of GEN-1.5’s reported 59% one-shot success is actually one-shot vs. pure zero-shot capability inherited from the pretraining data distribution.

Results

None reported (Twitter reply). The referenced LBM talk (Tedrake, Stanford 2024) is cited but not summarized.

Why it’s interesting

This is the first filed evaluation-methodology critique of the GEN-1.5 headline result, and it sharpens what would otherwise be a difficult concept-page claim on Inference-Time Scaling — that “in-context demonstrations lift robot success rates by X” — into two separable claims: pure zero-shot inference from scene, and demonstration-conditioned lift over it. Related to VLA Models‘s open question of what recipe levers actually matter: if Majumdar’s prediction holds, the “engine-scale emergence of ICL” position of Generalist (see also Towards Machines with a Thousand Hands) may be measuring pretraining-data coverage rather than an ICL mechanism. Contrasts productively with Robots Acquire Manipulation Skills in Seconds from a Single Human Video (HOST) (HOST) which reports numbers on the ablated axes GEN-1.5 does not: cascade-explicit one-shot from human video with published no-parameter-update results. The wiki has no filed follow-up experiment that runs Majumdar’s proposed ablation on any of the recent generalist robot policies — this is an open question, not a resolved one.