The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

TL;DR

The Imitator Game (Zhou et al., HKU + collaborators) argues that current robot policies learn trajectory-level observation-to-action mappings and thus imitate what a demonstrator does rather than what they intend, and proposes a four-level benchmark (L0–L3) that progressively widens the gap between the human demonstration and the robot’s own scene to isolate where trajectory replay stops working. The benchmark ships with IG-10K (20,000+ paired human-robot episodes, 50+ tasks, 6 domains, real + simulated, all four levels) and Imitator Arena (blind A/B human evaluation platform). Across nine state-of-the-art models performance is stable from L0 to L2 but collapses at L3, identifying functional substitution — achieving the same intent through a different object affordance — as the decisive barrier to intent-level imitation. Human-video-conditioned models beat caption-conditioned ones but every model falls below 13% zero-shot on unseen tasks; IG-10K-pretrained fine-tuning on only 10 paired human-robot demos yields large gains that grow with pretraining scale.

Key claims

The imitation gap is a level variable, not a binary: policy success on IG-10K is stable across L0–L2 (matching / near-matching robot scenes) but collapses at L3, with functional substitution (same-intent, different-affordance object) as the identified decisive failure mode [§Abstract, §Experimental findings].
IG-10K is presented as the largest environment-aligned paired human-robot dataset instantiated across all four L0–L3 levels in both real and simulated settings — 20,000+ paired episodes, 50+ tasks, 6 domains [§Paired human-robot data].
Under matched protocol, human-video-conditioned policies outperform caption-conditioned ones — evidence that language alone under-specifies the intent-level imitation problem the benchmark targets [§Experimental findings, Which imitation interface is strongest].
On the 5 held-out unseen tasks all nine SOTA models fall below 13% zero-shot success, and fine-tuning IG-10K-pretrained checkpoints on only 10 paired human-robot demonstrations produces large gains that grow with pretraining scale [§Unseen, §Experimental findings — Does a bigger corpus help].
The evaluation is triangulated across three axes — simulation, real-world deployment, and blind A/B human Arena judgments — over 10 tasks (5 seen, 5 unseen) × 4 levels, for 40 task-level pairs and 9 models / 15 trained variants [§Experimental findings].

Method

The benchmark defines four levels of imitation difficulty that share the same target outcome but progressively widen the gap between the human demonstration and the robot’s own scene: L0 matches the demonstration setting closely; each subsequent level perturbs object placement, object identity, and affordance until at L3 the robot must achieve the demonstrator’s intent through a different object affordance (functional substitution). IG-10K is the environment-aligned paired human–robot dataset instantiated across all four levels in both real and simulated deployments, unified in a single format with multi-view video, per-frame 3D MANO hand pose, segmentation masks, and multi-abstraction language annotations. Imitator Arena is a blind A/B human evaluation platform: a reference clip and two anonymous rollouts are shown side by side and judged. The paper evaluates 9 SOTA models (15 trained variants) across simulation, real-world hardware, and Arena judgments on ten tasks (5 seen, 5 unseen); as a full-suite reference DP and ACT are additionally trained on all 50 tasks / 200 task-level variants of IG-10K-sim with mild placement randomization.

Results

Headline findings from the project page — the paper itself is only released with abstract-level detail at filing:

Nine SOTA models scored across L0–L3 show stable performance L0→L2 then a sharp drop at L3, with functional substitution named as the decisive barrier [§Experimental findings].
Zero-shot on unseen tasks stays below 13% for every model evaluated [§Abstract].
Fine-tuning IG-10K-pretrained models with only 10 paired human-robot demonstrations yields large gains, and the gain grows with pretraining scale [§Abstract, §Unseen].
Human-video-conditioned models beat caption-conditioned ones under matched protocol [§Which imitation interface is strongest].
Full-suite DP and ACT reference numbers on all 50 tasks / 200 task-level variants are released as a starting-point leaderboard [§full-suite leaderboard].

Per-task, per-model numbers are hosted on the project’s rolling leaderboard rather than in the abstract; the abstract does not provide a single headline number.

Why it’s interesting

The Imitator Game supplies a diagnostic axis that the VLA-generalization discussion on the wiki has been missing. VLA Models currently tracks recipe levers (action-pretraining, clean teleop, unified-VLM pointing, world-model+action-expert, ICL) that all move a single aggregate success number; L0–L3 disaggregates the human-to-robot imitation gap into levels and localizes where each recipe collapses — the paper’s own L2→L3 cliff at “functional substitution” is the sharpest filed candidate for the operational definition of “intent-level” imitation. This complements LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition LUCID, which stakes out task intent as an explicit interface between a human-video module and a sensorimotor policy — the Imitator Game is the evaluation harness that would let LUCID’s “intent-level” claim be measured directly. It also sharpens GEN-1.5: Embodied Foundation Models are One-Shot Learners GEN-1.5’s emergent-ICL claim by giving it a level-graded held-out substrate: GEN-1.5’s aggregate 59% one-shot number does not distinguish L0–L2 (trajectory replay is sufficient) from L3 (functional substitution required). Finally, the “human-video beats caption” finding is a direct data point in the ongoing debate on Human-to-Robot Retargeting over whether language is a sufficient interface between demonstrator and robot, or whether the raw sensorimotor stream is load-bearing — the paper argues the latter.