VISTA: A Visual Harness for Reasoning in an Interactive World

TL;DR

VISTA is a minimalist agentic harness for ARC-AGI-3 built around three claims: perceive the world through raw high-dimensional visual input (a 512×512 PNG upscale of the 64×64 grid, never told the grid dimensions), reason in free-form natural language, and maintain a lossless visual memory the agent can revisit at frame / region / pixel granularity via inspect and read_pixels tools. On the 25-game public set, Claude Opus 5.0 driving VISTA scores a perfect 100.00 mean RHAE (all 25 games perfect, 183/183 levels solved) using 56% fewer actions than first-time human players; GPT-5.6 Sol reaches 98.27 (22/25 perfect). The same setup transfers to a 3D rendering and a 1D text serialization of the same world, arguing the harness is representation-agnostic. Self-reported, public-set only — no semi-private/private measurement.

Key claims

The agent is never told the world is a 64×64 grid — it receives only 512×512 PNGs (an 8× nearest-neighbor upscale with one-pixel cell gridlines) and available actions, and must form visual concepts entirely from its pretrained visual priors [§Visual perception].
Reasoning is free-form natural language with no imposed schema: the model decides what to reason about, when to reason, and can take notes in GUIDE.md (durable cross-level abstractions) and WORKING.md (per-level scratchpad) [§Language reasoning, §Reason then act].
Visual memory is lossless: every returned frame is stored at full resolution indexed by turn and frame; the agent invokes inspect (revisit an earlier frame or enlarge a region) or read_pixels (exact color samples from a region) on its own decision — an “explicit attention mechanism” at frame/region/pixel granularity that the KV cache cannot provide [§Lossless visual memory].
ARC-AGI-3’s RHAE metric counts only environment actions; internal reasoning and read-only visual inspection are free — the harness explicitly exploits this by moving reasoning and re-examination out of the action budget [§Method, §VISTA Results].
The harness contrasts explicitly with program-based ARC-AGI-3 approaches (Schema, Tycho, Retrodict): where Schema forces the model to write and validate an executable step() program (~4,000 lines of Python for one game), VISTA lets a page of free-form notes stand in — “fuzzy but flexible” reasoning suffices [§Comparing different reasoning processes, Table §Program vs Free-form].
The same recipe generalizes across three sensory representations of the same world (2D image, 3D rendering, 1D text grid) — all theoretically information-equivalent but not computationally equivalent; text-grid works but is token-inefficient, and 3D rendering works and is closer to how humans perceive the physical world [§Comparing different observation representations].
On the 25-game public set with Opus 5.0 (Effort xhigh), VISTA reports 100.00 mean RHAE, all 25 games at 100.00, 183/183 levels completed, 7,542 total actions vs 17,135 human (56% fewer) [§VISTA Results].
With GPT-5.6 Sol (Effort max), VISTA reports 98.27 mean RHAE, 22/25 perfect games, 10,063 total actions vs 17,135 human; residual 1.73 points concentrated in a few levels where the model spent extra actions discovering a mechanism or recovering from an incorrect game model [§GPT-5.6 Sol].
Context-length is handled by written continuation state: when approaching the context limit the model writes a concise summary and resumes from the current visual state in a fresh context, keeping notes, visual memory, and action history available [§The Complete Pipeline].
The pipeline is prompt-minimal: a single ~50-word system prompt asks the agent to maintain a “compact, revisable model of the game and its current state” and to state expected visual results before each play [§The agent prompt].
Results are self-reported on the 25 Public games only; no ARC Prize semi-private or private measurement is claimed [§VISTA Results].

Method

The harness constrains the agent to a Perceive → Reason → Act loop with three shared resources: raw PNG frames from the environment, a lossless visual memory indexed by (turn, frame index), and two mutable text notes (GUIDE.md cross-level, WORKING.md per-level). On each turn the environment returns a sequence of animation frames (all recorded in memory) plus the current status and available actions. The agent can call inspect(turn, frame, region) to view any past frame or enlarged sub-region and read_pixels(region) to sample exact color values; both are read-only and don’t count against RHAE. Before invoking play(action), the prompt asks the agent to state its expected visual result. On a prediction mismatch, the agent updates its notes rather than falling into a fixed error-recovery routine. The harness is instantiated with two off-the-shelf backends: Opus 5.0 through the Claude Code CLI, GPT-5.6 Sol through the Codex CLI — same prompt, same tools, same reasoning setting across games within each backend. No fine-tuning, no RL, no scaffolding-agent tree.

Results

Opus 5.0 (Effort xhigh): 100.00 mean RHAE, 25/25 games perfect, 183/183 levels, 7,542 agent actions vs 17,135 human (~0.44× human) [Public 25-game table].
GPT-5.6 Sol (Effort max): 98.27 mean RHAE, 22/25 perfect, 25/25 games completed, 10,063 agent actions [Public 25-game table].
Human tester estimate on the same games: ~48% RHAE (from blog-openai-how-enabling-two-settings-tripled-2026-07); VISTA + Opus 5.0 is >2× the estimated human baseline in the same units.
Cross-representation generalization: agent still performs on 3D rendered and 1D text-grid versions of the same games; text-grid is less token-efficient but functional [§Comparing different observation representations, no per-representation table published].
6 of the 8 replays for each backend are linked from the project page for the full action trace, visual inspections, and agent notes.

Why it’s interesting

VISTA sits at the opposite extreme of the same design axis as Schema — Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public on ARC-AGI-3 — Schema won by making the model write and verify a total-history executable simulator; VISTA wins by making the model observe and revisit a total-history lossless image archive. Both isolate the harness as the load-bearing lever over the frontier model, and both reach ≥95 RHAE on the public set with the same class of base models where the ARC-Prize official harness reports Sol Max at ~7.8% (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence). Together with How enabling two settings tripled our scores on the ARC-AGI-3 benchmark‘s retained-reasoning + compaction result, this is the third filed instance in a converging story: on interactive-agentic benchmarks, the harness contract dominates model capability, and the way to win is to not discard the model’s evidence — Schema keeps every recorded transition, OpenAI keeps every CoT token, VISTA keeps every pixel. Distinguishes itself from Schema on the Thinking with Modalities axis: Schema is program-space reasoning (substrate near “external-tool Think-Act-Observe” but code-native); VISTA is free-form natural-language reasoning grounded on a queryable lossless image archive — closer to substrate (a) tagged-latent + substrate (c) tool-based zoom, but with the model’s own pretrained visual priors doing all the perception. Directly relevant to any Luma agent harness that operates on video or 3D observations: the “lossless memory + inspect/read_pixels tools” pattern is a cheap alternative to KV-cache compression for keeping long-horizon visual evidence available, and the RHAE-style “read-only inspection is free” scoring is worth adopting internally.