In-Context World Modeling for Robotic Control

TL;DR

ICWM (Fudan NLP) treats system identification as in-context adaptation for VLAs. Rather than conditioning the policy only on the current observation and the language instruction — and thereby implicitly assuming the training-time setup — ICWM gives the policy a short prefix of self-generated, task-agnostic interactions and asks it to infer the latent system variables (camera viewpoint, robot morphology) implicitly from that history. No parameter updates, no demonstration prompts of the actual task; the context is used to learn how the system works, not what to do. Reported gains over standard VLA baselines on novel camera viewpoints in both simulation and real-robot deployments.

Key claims

Conditioning VLAs only on the current observation + language instruction implicitly fixes the execution context to whatever was seen in training, which is why standard VLAs need data-intensive fine-tuning for any new camera viewpoint or morphology [Abstract].
ICWM differs from standard In-Context Learning by using the context window for system identification (how the system operates) rather than task specification (what task to perform) [Abstract].
The system-identification context consists of a short history of self-generated, task-agnostic interactions executed before task execution [Abstract].
Implicitly capturing world dynamics from the interaction prefix is enough to adapt to novel configurations without parameter updates [Abstract].
Reported to significantly outperform standard VLA baselines on novel camera viewpoints in both simulation and on real-world robot platforms [Abstract].

Method

ICWM frames the system configuration (camera pose, morphology, etc.) as a latent variable that the policy must infer from a short, self-generated interaction history pre-pended to the policy’s context window. The interactions are task-agnostic — the policy is not shown demonstrations of the downstream task — so the prefix functions as system identification rather than ICL-style task prompting. At test time, the policy executes some short rollout (the “self-generated interactions”), the resulting (observation, action) history is fed back into the model as context, and then task execution proceeds conditioned on both the language instruction and the interaction prefix. No fine-tuning is required to adapt to a new viewpoint or morphology. Method-level details (architecture, training objective, the exact data-generation procedure for the interactions, the schedule for how the prefix is collected at deploy time) are not in the abstract; the body of the paper is paywalled in the fetched HTML stub.

Results

The abstract reports only directional results: “significantly outperforms standard VLA baselines on novel camera viewpoints” in both simulation and on real-world robot platforms. No table numbers, no morphology-generalization numbers, and no baseline list are extractable at filing time. Headline-quant refresh pending a body fetch.

Why it’s interesting

This is the actual paper behind HuggingPapers tweet 2070794187063099893 — In-Context World Modeling for Robotic Control (ICWM) — filed last week as a content-not-retrievable stub because the HuggingPapers tweet attached only an image preview with no link to the arxiv ID. With the arxiv ID resolved, ICWM can now be evaluated against the four existing positions on the VLA Models page:

It is structurally distinct from action-pretraining at scale (π*0.6: a VLA That Learns From Experience (RECAP)), clean-teleop data (Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models), unified VLM with strong pointing/grounding (Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models), and the frozen-WFM-plus-action-expert factorization (μ₀: A Scalable 3D Interaction-Trace World Model). ICWM puts the generalization lever in the test-time context window, not in pretraining scale, data quality, or representation choice.
It is a candidate answer to one of the open chicken-and-egg sub-problems Sitzmann named on the World Foundation Models page: how to get action-conditional world-modeling capability without large paired action-observation pretraining data. ICWM’s framing — the robot generates its own short interaction prefix on the fly, and the model uses that to infer dynamics — is structurally analogous to DAWN’s coupled world-action inference on autonomous driving (The DAWN of World-Action Interactive Models), but pushed to the in-context regime instead of being baked into the architecture.
Closest filed sibling on “new morphologies” is UMA: Unified Motion-Action Modeling for Heterogeneous Robot Learning (UMA), which builds an embodiment-invariant 3D-motion interface; ICWM tackles the same problem from the opposite direction by keeping the interface fixed and using context to absorb the embodiment delta.

The headline claim sits in a part of the design space — test-time adaptation as the primary lever — that the wiki has not previously filed a credible entry for. If the body of the paper substantiates the “without parameter updates” claim with quantitative camera- and morphology-generalization numbers, this is a structurally new VLA recipe; if the body reveals it requires per-deployment context-collection that itself amounts to fine-tuning by another name, the framing reduces to a context-engineering variant of standard VLA.