Skip to content
Bud's Wiki

HomeBody: A Humanoid That Explores, Remembers, and Acts on Its Own

TL;DR

Section titled “TL;DR”

HomeBody (Stanford Movement Lab + Caltech) wires GPT Astra directly to a Unitree G1 humanoid via a fixed library of composable motor skills, with no learned VLA policy or environment-specific training in between. The system first explores an unseen kitchen, builds an Isaac Sim digital twin from its own camera + SLAM + joint-pose + waypoint data as persistent spatial memory, then executes long-horizon tasks — tidying the space and retrieving a remembered object from an underspecified request. Every turn, the frontier VLM selects a skill (navigate, pick, place, open drawer, pick-from-drawer) and its typed spatial target from the current ego view, map context, gripper state, and recalled observations, and dispatches through a structured tool call whose implementation the VLM does not need to know. Reported limits are Real2Sim setup time and API cost, VLM latency between skills, and finger-servo overheating.

Key claims

Section titled “Key claims”
  • The VLM selects a skill and its target from the current ego view, map context, gripper state, recalled observations, and the previous result, then passes the selection through a structured tool call — the VLM does not need to know the skill’s low-level implementation [§Body].
  • Pick calls are typed as an image point normalized to 0–1000 plus a hand selector; the point prompts segmentation and Fast-FoundationStereo estimates depth from D435i stereo images, then calibration projects the masked geometry into 3D and an analytic grasp is predicted [§Body].
  • The arm planner builds a spline reference with minimum-jerk timing, solves inverse kinematics along the path, and checks the swept motion for collision clearance before execution [§Body].
  • Navigation calls take a 2D goal and facing point in map coordinates, measured in meters; a placing call specifies which hand to use, a 3D release target in the torso frame, and a release distance [§Body].
  • Drawer opening combines handle alignment, a hooking posture, and backward walking into one skill, coordinating the transition from reaching to pulling as a single tool [§Body].
  • The Real2Sim step grounds the VLM’s spatial reasoning: Astra itself acts as the Real2Sim agent, building the Isaac Sim digital twin from the humanoid’s own collected exploration data, and the twin lets the VLM reason about locations beyond the ego view [§Body].
  • Reported limitations: Real2Sim setup time and API cost, VLM latency between skill invocations, and finger-servo overheating during extended operation [§Body].

Method

Section titled “Method”

A frontier VLM (GPT Astra) sits in a closed loop over a real Unitree G1 humanoid. In the exploration phase, the robot moves through the unseen space under VLM direction to collect ego video, SLAM, joint pose, and waypoint data; Astra converts this into a persistent Isaac Sim digital twin that thereafter serves as external spatial memory. In the execution phase, each turn the VLM ingests the current ego view + map/twin context + gripper state + recalled observations + previous-tool result, chooses a skill from a fixed library (navigate, pick, place, open_drawer, pick_from_drawer), and emits a structured tool call with a typed spatial target. The skill layer handles low-level physics: segmentation from a point prompt (SAM-style), Fast-FoundationStereo depth from D435i stereo, camera calibration into 3D, analytic grasp synthesis, minimum-jerk spline trajectory, IK along the path, and swept-volume collision check. The whole loop runs on a pretrained AMO whole-body controller as System 0, with no learned VLA policy or environment-specific fine-tuning. The project page notes the code repository (Stanford-TML/homebody) currently holds only a README + figures with “code coming soon.”

Results

Section titled “Results”

The project page is qualitative — the two demonstrated tasks are (a) tidying up an unseen Stanford kitchen and (b) retrieving a remembered object from an underspecified request after prior exploration, both on a real Unitree G1. No success-rate tables, per-attempt latency numbers, token/cost accounting, or head-to-head against a trained VLA are reported at filing time. The Real2Sim comparison viewer shows LiDAR vs simulation reconstructions of the same kitchen but does not quantify reconstruction accuracy. Named failure modes: Real2Sim setup time and API cost dominate wall-clock; VLM latency shows up as pauses between skill calls; and the G1’s finger servos overheat during extended sessions.

Why it’s interesting

Section titled “Why it’s interesting”

HomeBody is the sharpest whole-body-humanoid instance yet of the “robot-use agents” paradigm Robot-Use Agents Phillip Isola named this month — the LLM is the sole high-level policy, no VLA sits between it and the composable skill library, and the load-bearing new component is an Isaac Sim digital twin built by the same model functioning as external persistent spatial memory across the exploration → tidy → retrieve arc. That fills a specific gap in the LLM-as-controller thread this wiki tracks on Tool-Use Agents and VLA Models: prior real-hardware demos (Jay Chooi (Robocurve) — Astra vs Fable 5.1 on real-robot manipulation via IK Robocurve YAM arms, Comparing Fable 5.1 to Fable 5 on robotic manipulation Fable head-to-heads, thijs — Astra paints the Golden Gate Bridge with a real robot arm Astra-paints, Wenli Xiao — GPT-6 Astra does physical in-context learning from a single human video via a motion-planning tool harness Astra-from-human-video) all operate in a single scene the LLM can see through the current camera; HomeBody is the first filed instance where the target object may be out of the ego view and the agent has to rely on structured memory the LLM itself authored to plan. Structurally it is closest to Agent as Policy for Robotic Manipulation AGP (typed EE-pose/joints/gripper tool interface with an experience-library-shortens-repeated-trials story) and to Show-Harness: Just a VLM Agent Can Play Robots Show-Harness (semantic-action-unit vocabulary over frozen VLM), but pushed to whole-body humanoid + long-horizon multi-room execution rather than tabletop. Sits in productive tension with Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level Malik’s sensorimotor counter-position (drawer opening + finger-servo overheating point at exactly the contact-rich regime that counter-position predicts LLM-controlled setups will struggle in) and with the Previewing the Model Hardware Standard MHS agenda for standardized device-driver interfaces (HomeBody’s typed skill calls are the hand-rolled precursor to what MHS would formalize). Also a useful data point for the Human-to-Robot Retargeting discussion: the robot’s own exploration data — not human video — becomes the retargeting substrate, and the retargeting happens through analytic IK + planning rather than a learned policy.

See also

Section titled “See also”