S1: In-Context Learning for Robotics

TL;DR

Skild’s S1 is a robotic foundation model trained from scratch as an in-context learner: task specification is a video demonstration inserted into a video-in / action-out policy at inference time, with no fine-tuning, no post-training, and one set of weights for every example shown. Unlike concurrent one-shot manipulation work (GEN-1.5: Embodied Foundation Models are One-Shot Learners, RoboTTT), S1 targets both axes the team argues actually matter — task horizon (up to 10 minutes) and out-of-distribution novelty (pancake flipping, plant potting, pour-over coffee, kit assembly, none in the pretraining corpus). A controlled 1k–100k-hour ICL-vs-language-prompting scaling study reports language-conditioned VLAs winning at 1k hours (53% vs 43%) but ICL winning at scale (66% vs 9% on unseen tasks at 100k hours). A single physical prompt is worth ~380 post-training episodes on unseen tasks by their interpolation, though a 2k-episode fine-tuned VLA still crosses at 86%.

Key claims

In-context learning for manipulation from a single video prompt, zero gradient updates, one set of weights across everything shown: task specification is a video demonstration (short or long, seen or unseen); the policy translates it directly to robot actions [§S1: An In-Context Learner].
Two-axis evaluation framing for ICL — task-in-distribution vs OOD, and short-horizon vs long-horizon — with the explicit claim that concurrent robotics ICL work (Generalist AI GEN-1.5, RoboTTT) covers short-horizon in-distribution tasks and S1 is the first foundation model shown to do long-horizon (up to 10 minutes) OOD tasks from a single video prompt [§Why In-Context Learning for Robotics?, §S1 In Action].
Controlled ICL-vs-language-prompting scaling study on filtered pre-training data from 1k to 100k hours, matched architectures/data/compute: on seen tasks the language-conditioned VLA leads at 1k hours (53% vs 43% ICL) but ICL wins at 100k hours; on unseen tasks language plateaus at 9% while ICL reaches 66% at 100k hours [§ICL scaling laws, ICL Scaling section].
Data-quality budget claim: “for every dollar we spend on collecting data, we spend three on quality control” — every data point screened for low-level precision, task coherence, and annotation fidelity before entering pre-training [§ICL scaling laws].
A single video prompt is worth ~380 post-training examples on unseen tasks by interpolation between measured post-training points; the crossover happens near 380 demonstrations and post-training passes single-shot ICL only at 2,000 demonstrations (86%) [§ICL demonstration efficiency].
Distribution-shift robustness quantified along two axes: distance from pre-training conditions and distance from the demonstration in context. Under L5 (opposite-arm re-execution) the language VLA degrades up to 3× more than the ICL policy; ICL only degrades significantly on prompt-shift L5 [§Quantifying ICL robustness to distribution shifts].
Emergent common-sense behaviors observed under ICL, unprompted by design: mistake recovery on OOD tasks (skateboard-wheel assembly), object substitution when the prompt uses an unavailable object (watering-can prompt executed with a cup), demonstration correction (controlled egg-drop where the demonstrator was sloppy), and robustness to mid-execution perturbations (sliding objects away, swapping objects, changing lighting — none in prompt) [§Emergent Properties].
The training recipe is described as “conceptually simple”: pre-train on episodic data where the task is specified only through an in-context demonstration, potentially from a different scene / viewpoint / embodiment, so the policy must learn intent + functional correspondences + task progress to predict actions [§S1: An In-Context Learner]. No meta-learning outer loop, no ICL-specific architecture, no context-packing infrastructure disclosed.
Task diversity + scale are what drive ICL emergence — in the high-diversity regime, scene ambiguity forces the model to attend to context, so pre-training becomes an outer loop that teaches the model how to learn from demonstrations [§S1: An In-Context Learner].
Multi-source data engine argument: no single robotics data source wins on all three of hardware proximity / diversity / scalability, so Skild scales all of them in-house — teleop (proximal, low-scale), egocentric human video (scalable, high domain gap), everything in between [§The Data Engine].
Plant-potting task deployment timeline: 11 minutes from demonstration to autonomous execution, most of it spent moving furniture and setting up the scene, not adapting the model [§Long-horizon unseen tasks].

Method

S1 is trained on episodic data where the task specification is only an in-context demonstration paired with the actual episode. Because the demonstration may come from a different scene, viewpoint, or embodiment than the episode itself, the policy is forced to implicitly extract the demonstrator’s intent, functional correspondences, and task progress in order to predict actions — training as an outer loop that teaches the policy how to learn from context. At inference the demonstration drives an inner loop without any weight changes. S1 is built on NVIDIA AI infrastructure. No architectural details, no context-window length, no parameter count, no training-compute number, and no data-mix breakdown are disclosed in this post; the team promises follow-up posts on training. Same weights are used for every task shown across the blog.

Two evaluation-methodology moves are worth naming. First, tasks are graded on average per-step success rate with human intervention to recover from failures during rollouts — ensuring every step is graded rather than truncating rollouts at first failure. Second, distribution-shift robustness is measured on two independent axes (distance from pre-training conditions vs distance from the in-context demonstration), rather than treating “task identity” as a discrete in/out flag.

Results

ICL scaling on seen tasks (per-step success, average across held-in tasks): language-conditioned baseline 53% at 1k hours vs ICL 43%; at 100k hours ICL passes language [§ICL scaling laws, Seen tasks paragraph]. Attributed to language admitting many valid executions while a demonstration specifies a mode.
ICL scaling on unseen tasks: language 9% at 100k hours vs ICL 66% at 100k hours — a 7× gap at the same dataset size, growing with data [§ICL scaling laws, Unseen tasks paragraph].
Demonstration efficiency (unseen tasks): language-conditioned VLA post-trained on 1–2000 teleop episodes; interpolated crossover with 1-shot ICL at ~380 post-training examples; single ICL demo hits 66% with zero training, VLA needs 2000 demos to reach 86% [§ICL demonstration efficiency].
Distribution-shift robustness (task-side): at L2/L3 object-pose perturbations both approaches survive; at L4 (matched-affordance object substitution) language VLA degrades; at L5 (opposite-arm re-execution) language VLA degrades up to 3× more than ICL [§Quantifying ICL robustness].
Distribution-shift robustness (prompt-side): ICL is robust to object-position mismatches and object substitutions between the demonstration and deployment scene (L4); only significant degradation at L5, where the demonstration implies a fundamentally different execution plan [§Quantifying ICL robustness].
Emergent behavior evidence: mid-execution object-slide, object-swap, and lighting-change perturbations (none in prompt) do not break execution; on unseen skateboard-wheel assembly the policy retries after failed insertions; on a watering-a-plant prompt with only a cup available the model uses the cup; on a fill-a-glass prompt where the glass is nearly full the model tops it off; on an egg-drop prompt where the demonstrator was sloppy S1 executes the same step with a controlled motion [§Emergent Properties, Robustness / Mistake recovery / Common-sense behavior / Demonstration correction].
Long-horizon OOD demos: plant potting (including digging into soil to make room for the plant), pancake cooking (including flipping — verified absent from pretraining corpus per the launch tweet), pour-over coffee (including pressing the filter into the funnel), kit assembly — up to 10 minutes, dozens of manipulation steps each, one video prompt per task, no post-training [§Long-horizon unseen tasks].
Deployment timeline: 11 minutes from demonstration to autonomous execution on plant potting, dominated by scene setup rather than model adaptation [§Long-horizon unseen tasks].

No shared-benchmark head-to-head numbers against GEN-1.5, RoboTTT, π*0.6, Spirit-v1.5, GR00T, or LIBERO/RoboArena/ArmnetBench are reported; task suites are internal to Skild for both the seen and unseen ICL scaling studies. The 100k-hour data-scale claim is the largest reported for an ICL-native manipulation policy on file.

Why it’s interesting

Sharpest concurrent counter-position to GEN-1.5: Embodied Foundation Models are One-Shot Learners on the wiki. Both papers argue in-context learning is the missing “GPT moment” for robotics, both train a base policy on episodic data with in-context demonstrations, both report one-shot task competence from a single video prompt with no gradient updates, and both explicitly name each other in their concurrent-work sections. Two structural differences separate them: (i) Skild frames the axis of interest as task horizon × OOD novelty and claims S1 is the first to hit both simultaneously (10-minute unseen tasks); GEN-1.5 does not report long-horizon aggregate numbers and the team itself calls their tasks “simple and short-horizon”. (ii) Skild ships a controlled scaling law for ICL vs language prompting from 1k to 100k hours at matched compute; GEN-1.5 reports point estimates from a continuously-running 8-month training curve without matched-recipe head-to-heads. The 7× gap at 100k hours between ICL and language prompting on unseen tasks — with language plateauing at 9% — is the sharpest filed evidence that language conditioning is a wrong-turn interface for the OOD regime, complementing Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor counter-position with a data-side rather than modality-side critique. Sits alongside Robots Acquire Manipulation Skills in Seconds from a Single Human Video (HOST) (HOST) as the two explicit-recipe filings that show one-shot from human video with published numbers — HOST via an explicit cascade (task-progress → predicted observations → action derivation), S1 via an emergent property of a video-in / action-out policy trained end-to-end on demonstration-conditioned episodes.

Also sits on the Human-to-Robot Retargeting cluster’s “ICL-substitute-for-retargeting” position (bridge the human/robot embodiment gap in a single forward pass rather than via an explicit retargeting step), joining GEN-1.5 as the second entry on that specific position — but with substantially harder demonstrated tasks. And extends Inference-Time Scaling‘s emerging “physical prompts as an axis of test-time compute” thread with the first filed scaling law on this axis: unlike RLM’s token-budget curve or MiroThinker’s tool-call curve, S1 reports the demonstration-budget vs task-diversity Pareto directly.

Load-bearing caveats worth naming for future readers: no architecture / parameter count / context length / pretraining-mix details, no baseline architecture disclosed for the “conventional VLA” scaling comparison, and the “1 demo = 380 post-training examples” number depends on interpolation between measured post-training points rather than direct measurement at the crossover. The team promises follow-up posts on training recipe.