History May Repeat Itself: RSI Seen from a Previous AI Era

TL;DR

Slide deck from Yuandong Tian’s keynote at the ICLR 2026 Recursive Self-Improvement (RSI) Workshop, drawing analogies between the AlphaZero/AlphaGo self-improvement loop circa 2016–2018 and what an RSI system built on top of an LLM might look like in 2026. The talk frames RSI as model + harness (network + MCTS in the AlphaZero analogy) and walks through four ingredients that current LLM-RSI attempts have to solve: (1) self-generated data and environments, (2) memory and long-context continuous learning, (3) better search strategies over learned action representations (LaMCTS), and (4) the cost of training and evaluation. Not a research paper — a position/synthesis talk citing the speaker’s own prior NeurIPS/ICML/TPAMI work plus current LLM-RSI threads (Language Self-Play, AlphaEvolve, Training-Free GRPO, Claude Code memory).

Key claims

The AlphaZero recipe (self-play → training data → model update, no human knowledge) is the historical reference design for RSI, and the analogue today is “model + harness” where the model has general knowledge and the harness is the MCTS-like outer loop [Slide: AlphaZero, Slide: Model + Harness = Self-Improvement].
Tian’s ELF OpenGo (ICML 2019) is offered as the empirical anchor: trained on 2k GPUs for 2 weeks with no human knowledge, then evaluated on a single V100 GPU, beating top professional Go players 5-0 across multiple opponents at 80k rollouts / 50s thinking time [Slide: OpenGo].
The key 2026 difference from 2018 RSI is that the model now arrives with abundant world knowledge, diverse thinking patterns, and concept understanding — replacing the “human knowledge / heuristics” and “machine-learned models” boxes of the AlphaZero era with a single LLM box [Slide: 2026: What’s the difference?].
Self-generated environments are a candidate solution to the data bottleneck: a Challenger generates hard environments on the fly while a Solver learns via RL on the current environment distribution, citing Kuba’s Language Self-Play For Data-Free Training [Slide: Generating Synthetic Environment/Data Distribution on the Fly].
Zero-sum self-play (LSP-Zero) suffers from reward hacking and needs general-sum regularization (LSP) with quality metrics added to the objective to remain productive [Slide: LSP-Zero vs LSP].
Memory architecture matters for continuous-learning RSI: Claude Code’s design splits memory into User / Feedback / Project / Reference, all stored as transparent auditable markdown files [Slide: Claude Code Memory Architecture].
Hybrid vector+BM25 retrieval (OpenClaw: 0.7 vec + 0.3 text, with MMR diversity, temporal decay, top-K) is presented as a concrete recipe for query → ranked-results in a single call against a SQLite database [Slide: OpenClaw Hybrid Search from Memory].
Long-context KV-cache design has a cost/latency rule of thumb borrowed from Manus: keep prompt prefix stable and make context append-only, because changing the prefix invalidates the entire KV cache downstream [Slide: Cost and Latency in Context Design].
Known long-context failure modes (“Lost in the Middle”, GSM-Infinite) decompose into three memory operations the agent stack has to support separately: Retrieval, Evolution, and Condensation (with State-Updating, State-Abstraction, Causal-Inference as sub-operations) [Slide: Long-lasting Issues in Long Context, citing AMA-Bench].
Test-time fine-tuning toward maximal-reward (TTT-Discover, Yuksekgonul et al.) trades diversity for task-specific performance — the model becomes specific to the task at the cost of losing diversity [Slide: Continuous Learning].
Even strong models may still need an agent harness: a 23M-parameter transformer trained to play “grandmaster-level” chess without search performs worse than the same model with search at blitz time, suggesting harness is not made obsolete by a more knowledgeable base [Slide: Back to Games, citing Ruoss et al.].
The representation of the action space dominates search efficiency: in NAS, a Global action space (set-depth, set-all-K, set-all-C) beats a Sequential action space (add-layer, set-K, set-C) given the same 1364-network budget [Slide: How to represent the “Action Space”].
LaMCTS (Latent Space Monte Carlo Tree Search) is offered as the speaker’s prior recipe: learn the action-space partition first, then search using the learned representation within a fixed rollout budget — applied to NAS-Bench-101, molecule design (QED, DRD2, HIV, SARS), and multi-objective settings [Slide: LaMCTS, citing NeurIPS 2020 / TPAMI 2021 / NeurIPS 2021 / ICLR 2022].
RL is not asserted to be the best RSI strategy — the talk explicitly flags Training-Free GRPO (Cai et al.) and AlphaEvolve (Novikov et al., coding agent for scientific/algorithmic discovery) as alternatives worth considering [Slide: Is Reinforcement Learning (RL) the best strategy?].
The cost of training and evaluation is the fourth open problem; proposed mitigations are scaling ladders, surrogate models, and interpretability (“open the black box”) [Slide: Cost of Training and Evaluation, Summary].

Method

This is a position talk, not a paper — there is no method beyond the synthesis itself. The structure is: (a) recap the AlphaZero/OpenGo self-improvement loop and identify it as the closest historical RSI template; (b) enumerate four ingredients that any LLM-era RSI system has to solve (data, memory, search, cost); (c) for each ingredient, point at one or two recent works that attempt it (Language Self-Play, Claude Code memory + OpenClaw retrieval, LaMCTS / Training-Free GRPO / AlphaEvolve, scaling-ladders/surrogates/interpretability); (d) close with the meta-question of whether the talk’s own research ideas could themselves be discovered automatically by AI.

Results

No quantitative results from the talk itself. Numbers referenced from prior work that the deck embeds:

ELF OpenGo: trained on 2k GPUs for 2 weeks, then beats Kim Ji-seok (ELO 3590, #3), Shin Jin-seo (#5), Park Yeonghun (#23), and Choi Cheolhan (#30) each 5-0 from a single V100 at 80k rollouts / 50s, and 27-0 against Taiwanese pros at 2k rollouts on a single GPU [Slide: OpenGo, citing Tian et al., ICML 2019].
Grandmaster-level chess without search (Ruoss et al.): ~23M parameters reach grandmaster-level blitz performance, but with-search > without-search at the same model — used as the punchline that harness still matters [Slide: Back to Games].
LaMCTS NAS budget: 1364 networks on NASBench-101, with Global action-space representation strictly better than Sequential at that budget [Slide: How to represent the “Action Space”].
LaMCTS molecule design: QED, DRD2, HIV, SARS single-objective tasks plus multi-objective combinations (GSK3β+JNK3, QED+SA+SARS, GSK3β+JNK3+QED+SA), latent representation learned from a 1.8M-molecule unlabeled corpus [Slide: LaMCTS, Molecule Design].

Why it’s interesting

This talk is the highest-altitude framing for several threads the wiki has been quietly accumulating. The “data” leg maps directly onto Synthetic Training Data — Language Self-Play (Challenger/Solver with general-sum regularization to avoid reward hacking) is structurally the same recipe as InfTool’s closed multi-agent loop in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing and the in-loop synthesis in NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: a generator + a verification/regularization gate. The “memory and long context” leg sits on top of Parametric memory (geometric/embedding-table storage) and Inference-Time Scaling (RLM/MiroThinker/TTT-E2E as scaffold-, depth-, and weight-update axes for extra inference compute) — the talk’s six memory operations (Retrieval, Evolution, Condensation × State-Updating, State-Abstraction, Causal-Inference) are the missing taxonomy that those concept pages have been gesturing at. The “search” leg connects to Reasoning RL and to the Training-Free GRPO entry already filed as Training-Free Group Relative Policy Optimization, and the LaMCTS thread (learn the action-space partition before searching it) reframes recent latent-action work like Path-Constrained Mixture-of-Experts from a search-efficiency angle rather than just a routing one. Most usefully for the team: the talk explicitly does not assert RL is the right answer and names AlphaEvolve + Training-Free GRPO as the relevant counter-examples — a cleaner statement of the open question than any individual filed paper makes.