GEN-1.5: Embodied Foundation Models are One-Shot Learners

TL;DR

Generalist AI’s GEN-1.5 is a multimodal robot foundation model (video + sensor + language + proprioceptive inputs → 100 Hz action trajectories, 30-second context) that exhibits emergent one-shot / few-shot / zero-shot learning of physical skills without any architectural or objective machinery designed for it — no meta-learning outer loop, no explicit in-context-learning training, no simulation data in pretraining. A single 3–12 s demonstration inserted into the context window yields 59% average success across 10 tasks with zero gradient steps (“physical prompting”), and 10 gradient steps on 5 min (~50 demos) lifts this to 83%. In the one-gradient-step regime on 1 min of data, held-out success is 66.5% while weights change by less than 0.15% — read as “reminding the model of something it nearly knows” rather than task-specific training. GEN-1.5 has trained continuously for 8+ months on Generalist’s real-interaction pretraining engine (the same one behind GEN-0 and Towards Machines with a Thousand Hands GEN-1), and every metric they track — data absorption, compute efficiency, per-task adaptation cost — kept improving through the run.

Key claims

A single 3–12 second sensorimotor demonstration placed in the 30 s context window (“physical prompting”) yields immediate task competence with zero gradient updates: 59% (±10%) average success across 10 diverse tasks including zippers, jar-opening, and wallet-money retrieval [§One-Shot Learning In-Context].
Physical prompts compose: two independently recorded demonstrations of different tasks placed together in context are chained into one continuous behavior, with the model generating intermediate motions (repositioning, regrasping, error recovery) that appear in neither prompt [§Compositional Generalization with Physical Prompt Engineering].
Zero-shot sim-to-real via in-context learning: a demonstration recorded entirely in a simulator serves as a physical prompt for a real-robot task, despite pretraining containing zero simulation data (no rendered video, no simulated dynamics) [§Zero-Shot Sim-to-Real Transfer with In-Context Learning].
Human-to-robot in-context transfer: a person demonstrating a task with their own hands in view of the robot’s cameras can serve as the physical prompt, and the robot reproduces the task on its own hands [§Human-to-Robot In-Context Learning].
Few-shot gradient adaptation drops from tens-of-thousands of steps (prior robot models) to 1–10 gradient steps on 1–5 minutes of data (~10–50 demonstrations), lifting one-shot 59% → 83% (±9%) with 10 steps on 5 min per task [§Few Gradient Step Adaptation, §Introducing GEN-1.5].
In the extreme regime, one gradient step on one minute of data reaches 66.5% success on held-out tasks while changing model weights by less than 0.15% — interpreted as slight reconfiguration of knowledge already present, not new representation building [§Few Gradient Step Adaptation].
In-context learning sometimes beats few-step fine-tuning on the same data: for some tasks the pretrained model prompted with a single demo outperforms 1–5 gradient steps trained on that same demonstration [§Introducing GEN-1.5].
None of these capabilities were explicitly trained for: no architectural changes for in-context learning, no meta-learning inner/outer loop, no auxiliary objectives encouraging improvisation. They emerge from pretraining scale on real-interaction data [§Introducing GEN-1.5].
Improvisation and novel tool use emerge from fine-tuning + pretraining prior: fine-tuned on 5 min of brush-based sweeping data, the model uses a banana as a makeshift brush, and given a dustpan (absent from fine-tuning and, to the team’s knowledge, from pretraining in this configuration) invents a lift-and-dump strategy — a fundamentally different contact sequence [§Physical Generalization, “Novel tool use improvisation”].
Improvisation strengthens as the number of fine-tuning gradient steps decreases, presumed to be because lightly-adapted models stay closer to their pretrained priors and can draw on a broader repertoire [§Physical Generalization].
The pretraining sampling procedure did not include bespoke infrastructure for packing multi-example prompts into context — physical prompts introduce discontinuous jumps in time that the model never saw in training, but in-context learning still emerges [§One-Shot Learning In-Context].

Method

GEN-1.5 is a large multimodal transformer taking video (30 s memory), other sensors, language, and proprioceptive inputs, and emitting 100 Hz action trajectories. Its 30 s context window holds rolling observations for closed-loop control; “physical prompting” reuses this window by placing a single sensorimotor demonstration (sensor stream + action trajectory) at the front, either recorded from another robot or from a person using handheld grippers. Once the prompt is in context the model performs the task immediately, no training. Pretraining runs on Generalist’s data engine capturing physical interaction in homes, warehouses, and factories, sampling randomly from continuous spans — no context-packing infrastructure. The run has been continuous for 8+ months (in parallel with GEN-1’s release), with periodic surgical architectural and algorithmic changes. Compositional prompts, sim-to-real prompts, and human-hand prompts are formed by concatenating a demonstration from a different source into the same context slot — no interface changes, no separate training. Few-shot gradient adaptation uses hyperparameters similar to pretraining, sampled sequences from 1–5 min of data, 1–10 steps. There is no meta-learning outer loop; the base is the same base used for in-context inference.

The team offers two hypotheses for why in-context learning emerges without being trained for: (1) an analogy to language, where “burstiness” and Zipfian structure in the token distribution have been linked to in-context learning in LMs, and (2) that physical work contains naturally repetitive cycles and the model may learn to detect and extend such patterns, as language models do with general sequences.

Results

One-shot in-context: 59% ± 10% avg success across 10 tasks [§Introducing GEN-1.5]; zero gradient updates.
Few-shot fine-tune (10 steps on 5 min ≈ 50 demos): 83% ± 9% avg [§Introducing GEN-1.5].
One-step regime (1 step on 1 min): 66.5% on held-out tasks, weight change <0.15% [§Few Gradient Step Adaptation].
Sim-to-real prompt: demonstration recorded in simulator → real-robot task success; prompted behavior further generalizes to different hands and new object positions/sizes [§Zero-Shot Sim-to-Real Transfer].
Composition: two independently recorded prompts (unzip pencil pouch + retrieve money) chain into one bimanual behavior with intermediate repositioning/regrasping/recovery that appears in neither prompt [§Compositional Generalization].
Novel-tool improvisation: brush → banana (used similarly); brush → dustpan (entirely new lift-and-dump strategy) [§Physical Generalization].
Emergent behaviors from single-block-into-single-bowl fine-tuning: obstacle removal (lifting paper off the bowl and sometimes replacing it), bimanual jar-lid rotation when trained only on one-handed, block-sorting by color/category when only trained on single-block placement, generalization to unseen cups/bottles for lid-twisting [§Physical Generalization].
No aggregate benchmark table, no head-to-head against π*0.6 / GR00T / MolmoAct / Spirit-v1.5 / Embodied-R1.5 on shared suites (LIBERO, RoboArena, ArmnetBench); tasks are described as “simple and short-horizon” by the authors themselves [§Introducing GEN-1.5].

Why it’s interesting

Sharpest filed answer to the Inference-Time Scaling concept’s open question of whether in-context learning transfers cleanly from language to embodied control — GEN-1.5 says yes at pretraining scale, with three concrete extensions language ICL does not have crisp analogs of: (a) cross-substrate ICL (sim demo → real prompt) that works even when pretraining contains no simulation data at all, (b) cross-embodiment ICL (human hand → robot hand) that works from raw human video observed through the robot’s cameras, and (c) physical prompt engineering — composing two independent demonstrations that were never co-observed and having the model bridge them. Language-domain analogues of (a) and (b) do not exist in the wiki’s filed literature. The 1-step-<0.15%-weight-change datapoint reframes the adaptation regime as closer to Long context as weights‘s SEAL / D2L / TTT-E2E axis than to conventional fine-tuning — but with the update parameterized as a nearly-null gradient step on the shared backbone rather than as a document-emitted LoRA or a self-generated implication corpus, and without meta-learning through the inner loop.

Sits alongside Towards Machines with a Thousand Hands as the second post from Generalist articulating what their pretraining engine buys beyond raw performance: the earlier post scaled the embodiment axis (~9,000 end effectors); this one scales the behavior repertoire under a fixed prompt — same core bet that pretraining data quantity + quality trumps recipe cleverness, cashed out on a different scaling axis. Complements the VLA Models recipe-lever board’s existing entries: unlike RECAP (π*0.6: a VLA That Learns From Experience (RECAP)), Embodied-R1.5, or Spirit-v1.5, GEN-1.5 does not name a lever at all — the position is that the engine is the lever and everything else emerges. The counter-point already on the wiki is Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor critique; GEN-1.5 does not report tactile/contact numbers and the tasks are open-loop-plausible without instrumented force feedback, so this filing does not resolve that debate. Also contrasts with Robots Acquire Manipulation Skills in Seconds from a Single Human Video (HOST) (HOST) which stakes out an explicit no-parameter-update cascade recipe (task-progress → predicted observations → action derivation) to achieve one-shot from human video with published head-to-heads; GEN-1.5 claims the same regime as an emergent property of a monolithic model.