Skip to content
Bud's Wiki

HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

TL;DR

Section titled “TL;DR”

HABIT is a bimanual robot manipulation dataset (referenced arXiv preprint 2606.31682) collected with a human sharing the workspace in every episode — a deliberate departure from existing manipulation datasets where policies learn tasks in isolation. Two Franka Research 3 arms are teleoperated across 60 tasks under three canonical Human-Robot Interaction roles from the HRI literature: Collaborator (joint goal, direct physical interaction), Coworker (separate tasks, shared workspace, must avoid collision), and Supervisor (human directs the robot via gestures/demonstrations, robot infers intent from vision alone). The collection protocol enforces that every recorded action is grounded in a cue visible to the cameras — no pre-execution from memory — and mixes in deliberate variations (random gesture delay, varied human speed, decisive robot retraction on collision risk) to instill yielding and reactivity as defaults. Fine-tuning π0.5 and a second VLA on HABIT beats human-absent teleop baselines on the four tasks where a robot-only baseline is even defined, and mid-training on HABIT before task-specific fine-tuning transfers as a reusable prior at every data budget tested.

Key claims

Section titled “Key claims”
  • HABIT is a 60-task bimanual dataset (two Franka Research 3 arms, five synchronized RGB cameras) in which every episode includes a human sharing the workspace, addressing the gap that “existing robot manipulation datasets are collected in human-absent settings, so policies trained on them can perform a task in isolation but fail to act appropriately when a person is present” [§Overview].
  • Tasks are organized under three Human-Robot Interaction roles — Collaborator (joint goal, physical interaction), Coworker (parallel work, shared workspace, collision-avoidance), Supervisor (human directs via gestures, robot infers from vision) — each defined by how the human and robot relate within a subtask [§HRI Roles].
  • Each task follows a task workflow: a directed graph of interleaved human (Hi) and robot (Ri) subtasks that fixes how the two agents divide and sequence work, with representative workflows shown per role (Table Service / Shelf Cleaning / Cup on Coaster) [§Task Workflows].
  • The core collection rule is camera-visibility grounding: “Each operator acts only after directly observing the partner’s behavior. We prohibit any coordination signal that is not captured in the recorded observations, so every demonstrated action is grounded in cues that are also available to the policy” [§Reactive Recording]. If the operator pre-executes from workflow memory, the triggering cue falls outside the camera input and no policy can recover the behavior.
  • Three deliberate design choices on top of reactivity: decisive retraction on imminent collision (yielding becomes the default recorded response), deliberately varied human speed across episodes (data captures pace adaptation), and randomized delay before gestures in Supervisor tasks (forces the recorded behavior to depend on actually reading the gesture rather than shortcutting to the known target) [§Design Choices].
  • Fine-tuning π0.5 and a second VLA on HABIT outperforms a Robot-only baseline on the four tasks where a robot-only baseline is applicable — Collaborator tasks need a human partner so Robot-only is not defined there [§Evaluation, evaluation figure].
  • HABIT transfers as a reusable prior: mid-training π0.5 on HABIT and then fine-tuning on a new task beats fine-tuning from scratch at every data budget tested [§Transfer results].

Method

Section titled “Method”

HABIT is a data-collection recipe rather than a model. Two teleoperated Franka Research 3 arms and five synchronized RGB cameras record 60 tasks, each specified as a directed subtask graph over interleaved human (Hi) and robot (Ri) actions and instantiated under one of the three HRI roles. Two human operators are present: one drives the robot via teleop, one plays the human collaborator/coworker/supervisor in the scene. The protocol forbids either operator from acting on private workflow knowledge — the robot operator must wait until the partner behavior is visible in the recorded camera stream before executing the next subtask, and the human operator inserts randomized delays before issuing supervisory gestures. Collision-avoidance is baked into the recording rule: whenever the robot risks contacting the human or a human-held object, the robot operator decisively retracts, so the dataset contains yielding as the modal response rather than as an exception. Human motion speed is varied episode-to-episode so the policy is trained on a distribution of paces. Downstream evaluation fine-tunes two open-source VLAs — π0.5 and a second unspecified VLA — on HABIT, and separately mid-trains π0.5 on HABIT before task-specific fine-tuning to test transfer.

Results

Section titled “Results”

The dataset spans 60 tasks across the three HRI roles. On the four evaluation tasks where a Robot-only (human-absent) baseline is defined, both π0.5 and the second VLA fine-tuned on HABIT beat the Robot-only baseline on success rate (higher is better); Collaborator tasks require a human partner so Robot-only is N/A there [§Evaluation]. Each role has a characteristic failure mode — collision in Coworker, missing the handover window in Collaborator, ignoring the gesture in Supervisor — that HABIT-trained policies avoid where Robot-only ones do not [§Failure-mode video comparison]. On the transfer axis, mid-training π0.5 on HABIT then fine-tuning on a held-out task beats direct fine-tuning at every data budget tested [§Transfer plot], positioning HABIT as a reusable prior for human-robot interaction rather than a task-specific fine-tune substrate. Absolute numbers are not quoted on the project page for the transfer / failure-mode results; the underlying arxiv preprint 2606.31682 is referenced but was not resolvable as an arxiv abstract page at filing time.

Why it’s interesting

Section titled “Why it’s interesting”

HABIT stakes out a data axis none of the currently-filed VLA-recipe papers occupy: what a manipulation dataset should contain so a policy behaves correctly when a person is physically in the workspace, not just when a task is solvable in isolation. It cleanly complements the recipe-comparison thread on VLA Models — π*0.6: a VLA That Learns From Experience (RECAP) (action-pretraining + RL), Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models (clean teleop as the lever), Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (unified-VLM pointing), ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation (bimanual behavior-cloning at scale) — by adding a what-should-be-in-the-teleop-data argument that is orthogonal to model architecture and RL post-training. It is a direct empirical counterpoint to human-absent bimanual corpora like ABC-130K, which HABIT’s motivating claim (“policies trained on them can perform a task in isolation but fail to act appropriately when a person is present”) targets by name. The camera-visibility-grounding rule also sharpens the Human-to-Robot Retargeting discussion: HABIT is not retargeting human motion but retargeting the coordination interface — every trigger cue must exist in the recorded observation — a design principle that transfers to how egocentric-video pretraining substrates (HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data) should be curated when the downstream policy is expected to interact with a person.

See also

Section titled “See also”