ARC-AGI-1 public eval examples leaked into ARC-AGI-2 training data

TL;DR

Alexia Jolicoeur-Martineau (author of TRM, Less is More: Recursive Reasoning with Tiny Networks) reports that some ARC-AGI-1 public-evaluation examples appear in the ARC-AGI-2 training set. Quote-replying to @JFPuget — who hit >75% pass@2 on the ARC-AGI-1 public eval with the same TRM checkpoint and recipe used for ARC-AGI-2 — she confirms she investigated and found the overlap, so training on both ARC-AGI-1 and ARC-AGI-2 training data inflates ARC-AGI-1 numbers via direct memorization rather than generalization.

Key claims

• Some ARC-AGI-1 public-evaluation examples are present in ARC-AGI-2 training examples [post body].
• Training on both ARC-AGI-1 and ARC-AGI-2 training splits therefore leaks the ARC-AGI-1 public eval, producing “crazy good” public-eval accuracy that is not a measure of generalization [post body].
• JFPuget’s quoted result — >75% pass@2 on the ARC-AGI-1 public eval from a TRM checkpoint trained on the combined data — is consistent with this leak hypothesis [quoted tweet].

Method

Single-tweet primary report; no methodology beyond “I had investigated this.” The quoted JFPuget tweet provides the triggering empirical anomaly (TRM hits a much higher ARC-AGI-1 public-eval pass@2 under the combined-training recipe than its paper-reported 44.6%).

Results

• TRM paper reports 44.6% (two-attempts) on ARC-AGI-1 public eval (Less is More: Recursive Reasoning with Tiny Networks §5, Table 4). JFPuget reports >75% pass@2 with the same checkpoint and recipe once trained on the combined ARC-AGI-1 + ARC-AGI-2 data — a >30-point jump attributed to the example overlap.

Why it’s interesting

This is a primary-source disclosure of a contamination problem from the TRM author herself, and lands directly on the methodological warning that ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence makes its central design constraint: ARC-AGI-3’s private set is deliberately OOD from public demonstration data precisely because the ARC team accepts that ARC-AGI-1/2 became attackable via dense synthetic / overlapping training coverage rather than improved generalization. The new piece of evidence here is that the overlap is not just behavioral (LRMs picking up ARC-style integer-to-color mappings from synthesis pipelines) but a literal example-level leak between the two official datasets — so a fair ARC-AGI-1 number requires either holding out ARC-AGI-2 training data, or moving evaluation to ARC-AGI-3’s private set. For the wiki’s Reasoning RL cluster the takeaway is concrete: any internal recipe that reports ARC-AGI-1 numbers trained with ARC-AGI-2 data should treat them as upper bounds, not as evidence of compositional generalization.

See also

• Less is More: Recursive Reasoning with Tiny Networks — Alexia’s TRM paper, the model at the centre of the JFPuget anomaly; reported 44.6% on ARC-AGI-1 in the paper
• ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence — ARC-AGI-3’s private/public split is the explicit fix for this class of contamination; the framing argument that ARC-AGI-1/2 were attacked by training-distribution overlap rather than generalization is now backed by a primary source
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity — independent argument that LRM “reasoning” is brittle outside training-distribution coverage; this leak is a concrete example of how a benchmark number can look like reasoning and be memorization