Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness

TL;DR

Wenli Xiao demonstrates OpenAI’s GPT-6 Astra doing physical in-context learning out of the box: drop a recording of a human doing a novel task into the codex app, prompt Astra to “drive a robot arm the same way,” and it works on the first pass. Under the hood Astra emits target end-effector poses that the researcher’s motion-planning tool harness converts to joint targets via IK. A first-pass no-fine-tune ICL data point on the recipe-lever board that pushes the thijs — Astra paints the Golden Gate Bridge with a real robot arm paint-brush and Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK YAM-arm threads one step further — the demonstration is a human video (not a text prompt or a physical demo in the robot’s scene), and the retargeting from human motion to robot motion is handled entirely in the LLM’s forward pass on top of a thin IK tool.

Key claims

Astra (GPT-6) can retarget a human-video demonstration to a robot arm on the first attempt, with no fine-tuning and no VLA in the loop [Post 1/thread].
The interface between Astra and the robot is tool calls in a motion-planning harness — Astra outputs target end-effector poses and the harness solves IK to joint targets [reply from author to @_k_sridhar].
The demonstration channel is a human-body video dropped into the codex app as context, not a text description, a physical demo in the robot’s own scene, or a teleop trace [Post 1/thread].
The pattern is characterized by the author as “physical ICL out of the box” — no per-task training loop, no specialized robotics prompt scaffolding beyond the harness [Post 1/thread].

Method

Tweet thread with a short (~6 s) video demonstration (Post 1) plus a reply clarifying the harness architecture: Astra outputs target end-effector poses; the researcher-provided motion-planning tool harness runs IK to produce joint targets. The robot is a small tabletop arm (visible in the video). The demonstration is a recording of a human performing the task (the “novel task” is not named in the thread; the video shows the demo being handed to Astra via the codex app and Astra then driving the arm). No paper, no code, no numbers. No comparison against Claude / Gemini / prior GPT models on the same rig; no report of the harness’s toolset beyond “motion planning + IK,” no per-attempt latency, no token cost, no success rate across attempts, no baseline task without the human demo. Companion thread from @thermalpastor references a related use case (Astra writing and debugging a G1 humanoid cycling controller in MuJoCo).

Results

Qualitative only. Author reports the task “just works on the first pass” from a single human-video demo dropped into the codex app. No metric captured — no similarity score against the demonstration, no per-attempt success/failure judgment, no independent evaluator, no wall-clock or token budget. No cross-task or cross-object generalization measurement. The reply thread confirms the harness setup but does not disclose which motion-planning library or IK solver is used, whether Astra had prior context about the arm’s kinematics, or whether the human-video was pre-processed before being handed to Astra.

Why it’s interesting

Sits at the intersection of three concepts this wiki actively tracks: the LLM-as-controller row of VLA Models‘s recipe-lever board, the physical-tool-use cell of Tool-Use Agents, and the no-explicit-retargeting-step position on Human-to-Robot Retargeting that GEN-1.5: Embodied Foundation Models are One-Shot Learners GEN-1.5, OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence DM0.5, and Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization Zero-WAM already stake out — but staged one abstraction level higher: no VLA at all, and the human demonstration is a pre-recorded video rather than a live in-scene demo. Two moves are worth flagging for the internal recipe-lever conversation: (i) the harness architecture (Astra emits EE targets, tool harness runs IK) is the tool-use-agents answer to what metal-arm-harness — Arm-agnostic coding-agent harness for MakerMods Metal (Isaac Sin) MakerMods and Inspect Robots — An Open-Source Evaluation Framework for Physical AI (Robocurve) Inspect Robots build as reusable substrate — and it is what makes Astra’s physical ICL practically deployable without any policy training; (ii) the human-video demonstration as the interface between task specification and robot execution is the same collapse Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots‘s 3-DoF wrist reduction, LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition LUCID’s object-flow-as-intent, and Robots Acquire Manipulation Skills in Seconds from a Single Human Video (HOST) HOST’s task-progress manifold all target with a designed representation — Astra achieves it here without any such representation, purely from in-context understanding of the video. The productive-tension counterpoint is Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level‘s sensorimotor position (contact-rich manipulation is not solved at the VLM/harness level) and Comparing Fable 5.1 to Fable 5 on robotic manipulation‘s specific failure mode for LLM-controllers-via-IK (Fable 5 stalls at first contact 13/20 times on bowl tasks) — this thread’s first-pass success does not report the contact-establishment regime where that mode fires. Kyle’s Slack framing (“How hard would this be to try?”) is well-posed: the harness side is a weekend project (motion planner + IK on a small arm + codex app integration), the load-bearing question is whether the pattern holds beyond the specific task in the video.