DexHoldem: An Agentic Robotics Benchmark for Dexterous Manipulation in Texas Hold'em

TL;DR

DexHoldem is a real-world benchmark for dexterous hand embodied systems built around Texas Hold’em tabletop interaction, bundling three deliberately coupled evaluation surfaces: a policy bench of 1,470 teleoperated demonstrations spanning 14 card and chip action primitives, a 36-problem agentic perception bench that tests parsing of turn gates, cards, chips, bets, outcomes, and robot activity from scene observations, and three full-system case studies that close the perception-to-action loop on a physical table. On the current benchmark, the best policy reaches 61.2% task completion and 47.5% scene-preserving success, the best perceiver reaches 66.8% field-wise accuracy (GPT-5.5 under Codex), and strict problem-level exact-match tops out at 34.3% (Claude Opus 4.7 under Claude Code). The system is implemented as a skill for coding-agent harnesses (Claude Code, Gemini CLI, Codex), each binding its natively supported vision-language model.

Key claims

1,470 teleoperated demonstrations across 14 card and chip action primitives form the policy-training and policy-evaluation substrate; policies are scored by a four-level rubric over 80 real-world primitive rollouts per model [Benchmark Scope, Results].
Pretrained flow-matching VLAs lead the policy table: π0.5 reaches 38 SP, π0 reaches 38 SP, RDT 24 SP, DP (DINO) 21 SP; from-scratch baselines (DP-Transformer, RDT-small, ACT, BAKU, DP-UNet) trail by double digits in Scene-Preserving success [Results, Policy table].
The agentic perception bench scores a structured game state over 8 fields (loop stage, turn ownership, blind position, community cards, chip inventories, current bets, robot activity, outcome); the perceiver sees the current capture plus all predecessor states as context [Agentic Perception Bench].
Best field-wise accuracy comes from Codex + GPT 5.5 at 66.8 average; best strict problem-level exact-match comes from Claude Code + Opus 4.7 at 34.3 — the two frontier systems lead on different axes rather than one dominating [Results, Perception table].
Harness/perceiver pairing matters as much as model tier: Codex + GPT 5.4 scores 31.5 overall vs 25.9 for Codex + GPT 5.4 mini and 25.0 for Claude Code + Sonnet 4.6; Gemini CLI + Gemini 3 Flash scores 20.4 overall vs 10.2 for Gemini 3.1 Flash L [Results, Perception table].
The embodied system follows a perceive-route-execute loop: the agent parses the current table image into structured game state, a deterministic router loads persistent game-state memory (verified hole cards, multi-step action progress), validates parsed fields, enforces safety limits, selects an action class (wait / view card / push/pull chips / show hand / request human help), and translates it into instruction codes that route the dexterous policy for physical execution [Embodied System].
Three full-system case studies close the loop on a physical table and surface specific failure classes: recovery, human-help requests, and repeated primitive dispatches compound during real play [Abstract, Results].

Method

The system is implemented as a skill for coding-agent harnesses — Claude Code, Gemini CLI, and Codex — each binding its natively supported vision-language model as the perceiver. The agent runs a perceive-route-execute loop: parse the current table image into a structured 8-field game state, consult a deterministic router that holds persistent game-state memory (including verified hole cards and multi-step action progress), validate parsed fields against safety limits, and select one of five action classes (wait, view card, push/pull chips, show hand, request human help). The selected action is translated into instruction codes that dispatch the dexterous policy for physical execution on the robot hand.

Policy evaluation is factored by action primitive (14 total, covering cards and chips) and scored under a four-level rubric (SP, DC, TF, DF) with Scene-Preserving Success Rate (SPSR) and Task Completion Rate (TCR) as the headline metrics across 80 real-world primitive rollouts per model. The perception bench is a snapshot-based evaluation: each of 36 problems is a frame drawn from a real system-level deployment trajectory, and the perceiver must emit the 8-field structured state with all predecessor states supplied as context. Scoring is per-field accuracy (only on the subset of problems where that field applies — e.g. “showdown outcome” only on showdown problems) plus a strict problem-level exact-match across all applicable fields.

Results

Policy bench (80 primitive rollouts per model, four-level rubric SP / DC / TF / DF):

Pretrained family: π0.5 38 SP / 11 DC / 31 TF / 0 DF, π0 38 SP / 8 DC / 33 TF / 1 DF, RDT 24 SP / 13 DC / 40 TF / 3 DF.
From-scratch family: DP (DINO) 21 SP, DP-Transformer 11 SP, RDT-small 11 SP, ACT 8 SP, BAKU 5 SP, DP-UNet 1 SP — DP-Transformer carries the largest DF column (18), the rest cluster at 0–7.

Perception bench (36 problems, strict problem-level EM in bold, field-wise average right-most):

Claude Code + Opus 4.7: 34.3 EM / 49.1 avg
Codex + GPT 5.5: 31.5 EM / 66.8 avg (best average)
Codex + GPT 5.4: 31.5 EM / 54.5 avg
Codex + GPT 5.4 mini: 25.9 EM / 49.2 avg
Claude Code + Sonnet 4.6: 25.0 EM / 41.8 avg
Gemini CLI + Gemini 3 Flash: 20.4 EM / 51.5 avg
Claude Code + Haiku 4.5: 13.9 EM / 37.4 avg
Gemini CLI + Gemini 3.1 Flash L: 10.2 EM / 34.2 avg

Field-wise, strongest axes across all perceivers are blind-position identification (BI 91.7–100.0) and turn ownership (TO 68.5–94.4); weakest are community cards (CC 23.1–61.5), current bets (CB 10.4–45.8), and showdown outcome (SO 0.0–76.2). GPT 5.4 mini and Claude Haiku 4.5 both land SO at 0.0 — concrete evidence that terminal-event parsing is a distinct capability from mid-hand state recovery.

Three full-system case studies on a physical table are reported qualitatively, demonstrating how wait branches, recovery dispatches, human-help requests, and repeated primitive dispatches compound during real dexterous play — the paper does not report headline numbers for whole-hand play.

Why it’s interesting

DexHoldem lands two things the current recipe-lever board for VLA Models does not yet have a shared substrate for: a strictly discrete-state ground truth (poker game state is enumerable and verifiable per frame, unlike kitchen or industrial manipulation where “task state” is implicit) paired with fine-motor execution on cards and chips (clearance and friction regimes the sensorimotor counter-position of Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level specifically predicts to fail). The policy table’s clean pretrained-vs-from-scratch gap on SP (24–38 vs 1–21) and large DF gap on DP-Transformer (18) matches the broader “pretrained flow-matching VLAs dominate from-scratch BC under limited demonstration data” pattern seen on Reality Check — Pokeandwiggle's Real-World Robot Manipulation Leaderboard on an FR3 Duo Farm Reality Check and in ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm ArmnetBench, now with a cards-and-chips primitive set those benchmarks do not cover.

On the perception side, the frontier-LLM-as-game-state-parser interface complements the Tool-Use Agents / coding-agent-as-robot-policy cluster (OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies OpenRUA, RobotUse: Allocating Computation, Context, and Decisions RobotUse, Introducing Waddle: Agents that Control Robots Waddle) with a benchmark whose verifier is a structured discrete state rather than task-level success — directly measurable field-wise, and the per-field diagnostic breakdown is the kind of attribution signal those harness papers currently lack.

The verifier shape (strict problem-level exact-match + field-wise diagnostics) also makes it a candidate VLM-as-Evaluator substrate for agentic perception specifically — the perceiver is a frontier VLM-under-harness rather than a judge over generations, and the evaluation measures the harness contract (state format, predecessor context) alongside the model.