ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

TL;DR

ARC-AGI-3 is the third installment of the ARC Prize Foundation’s intelligence benchmark, and the first that targets agentic rather than static-grid fluid intelligence. It consists of 135 turn-based interactive environments (25 public, 55 semi-private, 55 fully private), each handcrafted by an in-house game studio, with no instructions, no stated goals, and a deliberately tiny action space (5 keys + Undo + a cell-click action on a 64×64 grid of 16 colors). Scoring uses RHAE (Relative Human Action Efficiency) — action counts on first contact normalized against the second-best human’s action count — meaning intelligence is operationalized as learning efficiency in actions, not as final task success. As of March 2026 humans solve 100%; frontier AI scores below 1%.

Key claims

ARC-AGI-3 introduces an interactive, turn-based, agentic format that replaces the static grid-puzzle format used in ARC-AGI-1 and 2 — the first major format change in the series since 2019 [§1, §2.1].
Intelligence is operationalized as action efficiency, not task completion: RHAE compares the agent’s per-level action count against the second-best human’s action count, squares the ratio, and aggregates across levels and environments [§4.1].
Beating ARC-AGI-3 requires matching or exceeding human-level action efficiency on environments seen for the first time, averaged across all private environments — not just solving them [§2.2].
Agents must learn four functional components on the fly with no instructions: exploration (actively obtaining information), modeling (turning observations into a world model), goal-setting (inferring what to target), and planning/execution (mapping action paths and course-correcting) [§2.1].
The benchmark is split into a 25-environment public demo set, a 55-environment semi-private set (for testing API-served frontier models), and a 55-environment fully private set (reserved for the official Kaggle competition) [§3.6, Table 1].
ARC-AGI-3 inverts the public/private ratio used by ARC-AGI-2 (10:1) — the public set is now a demonstration interface (~18% of environments), not a training resource, deliberately under-representing the private set’s mechanic distribution [§3.6].
Environments are built in a custom Python game engine targeting 1,000 fps after Unity was tried and rejected as too heavy for the iteration rate required [§3.3].
Acceptance criterion for an environment: a random policy must not solve a non-tutorial level more often than 1 in 10,000 attempts, verified by 1M-step random-policy fuzzing and explicit graph-based state-space construction with reachability bounds [§3.5.1, §3.5.2].
The paper argues that frontier LRMs have begun to defeat ARC-AGI-1 and 2 not via genuine generalization but via dense task-space coverage from LRM-generated synthetic ARC tasks plus reasoning-trace training; cited evidence is Gemini 3 Deep Think spontaneously using ARC’s integer-to-color mapping in its reasoning chain on a prompt that never mentions ARC or that mapping [§1.3.3].
Stated limit of current LRM “fluid intelligence”: reasoning capability remains bound to LRM knowledge coverage and the existence of verifiable feedback — modern reasoning models excel only in verifiable domains where the base model already has sufficient domain knowledge [§1.3.2].
The action space is intentionally small (5 keys + Undo + 1 cell-select action on 64×64) so that benchmark difficulty lives in environment logic, not control complexity; internal reasoning steps and retries are not counted as actions [§2.3.2].
Difficulty arises from composition of mechanics across levels (each environment has ≥6 levels with later levels integrating concepts learned earlier), not from increasing per-step complexity; environments use only Core Knowledge priors (objectness, geometry, basic physics, agentness) and forbid language, numbers, letters, and cultural symbols [§3.4].

Method

ARC-AGI-3 is built by an in-house game studio with four production stages (Specification → Internal → External human testing → Done), three to four environments per developer in parallel, and a custom Python engine. Each environment is a series of ≥6 turn-based levels; at each turn the agent receives a 64×64 grid frame (or animation sequence) and submits one action from a per-environment subset of {five key actions, Undo, cell-select with (x, y) coordinates}. A level ends at a terminal “win” frame. Validation has two layers: deterministic qualification (1M-step random-policy fuzzing must not beat non-tutorial levels >1/10,000) and exploratory graph-based state-space construction with hash-merged equivalent states, which estimates reachability and stochastic solvability bounds even when the full graph cannot be enumerated. Scoring (RHAE): for each level, the AI’s action count is compared to the second-best human’s action count via the square of the ratio; per-level scores are averaged within an environment (weighted across levels), and environment scores are averaged into a final 0–100% score. Internal operations (tool calls, reasoning chains, retries) are explicitly excluded from the action count — only environment-affecting commands are counted.

Results

As of March 2026, humans solve 100% of environments while frontier AI systems score below 1% [Abstract, §2.5].
Reported leaderboard from secondary coverage (not the arxiv paper itself): Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%; a 30-day developer preview reported the best non-frontier system at 12.58% [external sources, not paper].
ARC-AGI-1 status as of writing: defeated to ~53.5% in 2024 by test-time-training-based deep-learning solutions (winner of ARC Prize 2024); the 85% grand prize threshold remains unclaimed [§1.2].
ARC-AGI-2 status: NVIDIA NVARC took ARC Prize 2025 first place at 24% with synthetic data generation + test-time training on a 4B-parameter model; 85% threshold also unclaimed [§1.2].
The benchmark explicitly forecasts a measurement gap: the ARC team accepts that ARC-AGI-1 and 2 have been substantially memorization-shortcut-attackable via dense LRM task synthesis, and ARC-AGI-3 is designed so the private set is OOD from any publicly available demonstration data [§1.3.3].

Why it’s interesting

Connects directly to The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity, which argues that LRMs collapse beyond a critical compositional complexity and that reasoning effort decreases approaching that threshold — ARC-AGI-3’s “humans 100%, frontier AI <1%” data point is the empirical counterpart from a different methodology (interactive exploration vs. controllable puzzles), and the paper’s framing of LRMs as knowledge-bound rather than truly fluid is the same critique. Complements RL Environment Platforms as a purpose-built evaluation counterpart to SETA / Toolathlon-GYM / OpenReward — those platforms are RL-training environments with verifiable rewards; ARC-AGI-3 is also a turn-based interactive environment with a verifier-cheap, action-counted scoring function, but it is deliberately not available as training data (private set is OOD from public). Worth tracking for two reasons specific to Luma: (i) the action-efficiency metric is a transferable evaluation idea for any internal agentic system where step count matters (cost, latency, embodied risk), echoing the Critical-Steps metric in Reasoning RL §Key claims; (ii) the paper’s argument that ARC-AGI-1/2 was attacked by dense synthetic-task coverage rather than improved generalization is a methodological warning for any internal benchmark that ships its full distribution publicly.