How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

TL;DR

OpenAI reports that GPT-5.6 Sol’s 7.8–13.3% score on the ARC-AGI-3 public set was gated not by model capability but by two API settings the official harness leaves off: retained reasoning (private CoT preserved across turns instead of discarded after each action) and compaction (summarize-and-continue when the context window fills, instead of a rolling window that truncates the oldest actions). Enabling both settings via the Responses API — the same defaults ChatGPT and Codex ship with — moves the public-set score from 13.3% → 38.3% RHAE at ~6× fewer output tokens. The post is a harness autopsy, not an ARC Prize leaderboard row: the semi-private / private sets are not measured, and this remains below Opus 5’s ~30.2% under ARC’s own protocol.

Key claims

The official ARC-AGI-3 harness deliberately uses a “generic” contract with no tools or special features; ARC’s stated rationale is that a simple harness makes model shortcomings more visible and comparisons more fair [§Introduction].
Under that harness, GPT-5.6 Sol scores 7.8% and GPT-5.5 scores 0.4% on the public 25-game set — a headline result that the post argues measures the harness contract at least as much as model capability [§Introduction].
Two API settings — retained reasoning (private CoT preserved across turns) and compaction (summarize-and-continue when context fills) — are the defaults for OpenAI’s own consumer products (ChatGPT, Codex) via the Responses API [§Two settings].
Enabling both settings moves GPT-5.6 Sol from 13.3% (official-harness re-run) to 38.3% RHAE on the public set — a ~3× score lift [§Results].
The same settings cut output tokens by 6× on the public task set — the compaction path is not just a capability enabler but a token-cost reduction, similar in shape to the O(L²) → O(L) argument of validated compaction in Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems [§Results].
Estimated average human tester scores 48% RHAE based on official gameplay logs; the harness gap is the difference between “far below the human baseline” and “in the same range” [§Introduction].
Models are not told the scoring metric and cannot see their score during play; only text representations of frames and the current level are observed [§Introduction].
Post is explicit that the 38.3% is a harness/setting comparison on the public set, not an ARC Prize verified row on semi-private / private; ARC’s own July snapshot has Sol Max at ~7.8% under the official protocol [§Results, external context].

Method

The post describes two API-level knobs and argues both are load-bearing:

Retained reasoning. Under the official harness each turn is a fresh chat completion, so the model’s private chain-of-thought is discarded after emitting an action. The Responses API instead preserves the reasoning across turns, letting the model build on hypotheses it formed in earlier moves rather than re-deriving them from scratch each time. This is the “1. keep a compressed summary of all the reasoning till that point in context” observation from the shared Slack note.
Compaction, not truncation. When the context window fills, the official harness’s rolling-window strategy drops the oldest actions. The Responses API instead runs OpenAI’s canonical compaction — a summarize-in-place step that preserves what the model has learned about the game’s mechanics rather than deleting the evidence for those beliefs. This is the “2. don’t use a rolling window of context (better to run compaction once context limit is reached)” observation from the Slack note, and matches the ACM framing that only validated compaction achieves linear cost with preserved fidelity (Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems).

The two settings compose: retained reasoning fills the context faster (private CoT is a lot of tokens), which makes the choice of what to do at the window limit more consequential. Both settings are trivially available via the OpenAI Responses API and, per the post, are already the defaults in ChatGPT and Codex.

Results

GPT-5.6 Sol, official harness: 13.3% RHAE on the public set [§Results, headline].
GPT-5.6 Sol, retained reasoning + compaction: 38.3% RHAE on the public set — ~2.9× score, at ~6× fewer output tokens [§Results, headline].
Human tester average (OpenAI estimate): 48% RHAE based on official gameplay logs [§Introduction].
GPT-5.6 Sol, ARC Prize official protocol (Sol Max snapshot, Jul 24): ~7.8% [ARC Prize board, external context].
Opus 5 High under ARC’s protocol: ~~30.2% (~~“three times the next best model” framing from Anthropic’s launch week) [ARC Prize board, external context].
Not measured: semi-private and private ARC-AGI-3 sets, which is where ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence designed the benchmark to actually score frontier claims.

Why it’s interesting

Direct methodological complement to Schema — Frontier Models with Our Harness Achieve ~99% on ARC-AGI-3 Public: both papers demonstrate that on ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence the harness contract — not the underlying model — is the dominant lever. Schema’s controlled 42.83% → 98.98% swing under the same Opus 4.8/Fable 5 pair sits at one extreme (program-space world model + total-history verifier); this post sits at the other extreme (two API defaults + canonical compaction, no scaffolding at all) but reaches a similar conclusion about the direction of the effect. The 6× output-token reduction is a live datapoint for the O(L²) → O(L) validated-compaction cost argument in Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems — retained-reasoning-plus-compaction is one instance of what that paper calls Agentic Context Management, deployed at the API layer rather than as a system on top. Directly relevant to Luma’s own harness work per Sid’s note: if the same harness change pattern (private-CoT retention + summarize-on-fill instead of truncate-oldest) transfers, expected gains are on the same order — better task scores at lower token spend, not the usual efficiency/quality tradeoff.