Dream-RSI: Recursive Self-Improvement through Evolving Worlds

TL;DR

Dream-RSI is a Google / UMD / DeepMind / UVA framework for recursively self-improving the exploration policy of a coding agent by reusing the agent’s own discovery history as a zero-cost replay simulator. The key move: a completed discovery tree — every branching decision plus the realized code-execution outcome at each node — is treated as an exact off-policy simulator over the search space that was actually realized, so a candidate exploration policy can be evaluated by traversing the recorded tree in a different order rather than by running new expensive rollouts. Each outer lap deploys the current policy online, appends its discovery tree to a growing history pool, then rewrites the policy code multiple times and scores every revision by replay — the winner is redeployed and, by construction, is never worse than the incumbent. Across algorithm engineering (Lasso solvers), mathematical optimization (Sum Diff, Auto Correlation, Circle Packing), and GPU kernel engineering (VGG16, ConvDiv) Dream-RSI matches or beats hand-tuned exploration under substantially reduced discovery cost, with headline reductions of 2.43× fewer generations on VGG16, 2.09× higher ConvDiv score at comparable budget, and 162× fewer discovery-agent calls than SimpleTES on Lasso.

Key claims

Meta-level (exploration-policy) feedback is the bottleneck in long-horizon recursive self-improvement: judging a new exploration strategy normally requires running a full multi-thousand-cycle discovery loop end-to-end, and the meta-policy space is vast, so online meta-optimization pays a full rollout for every candidate [§Meta-level feedback is the bottleneck].
A completed discovery run is already a simulator over the realized search space: it records a tree of exploration decisions with the code-execution outcome at each node, so replaying an alternative policy over it is just traversing the same tree in a different order — every outcome the alternative policy would query is already on disk, so replay costs zero executions [§History is already a simulator].
The replay simulator is exact, not approximate: unlike a learned world model it predicts nothing, because it is the search space that was realized. The trade-off is a sharp support boundary — a policy can only be dreamt where history actually went — which forces the method to be a loop rather than a one-off [§History is already a simulator].
The improvement is monotone by construction: the offline phase writes M policy revisions (π⁰ = π_t, π¹, …, π^{M-1}) and scores each by replay over the whole history; since the currently deployed policy π⁰ is a candidate, the winner π_{t+1} = π^{m*} is guaranteed to be no worse than π_t [§Why the policy can never get worse].
Every online deployment enlarges the simulator pool; a policy dreamt against multiple worlds beats one tuned to the luck of a single run, and the improved policy in turn reaches regions no earlier policy would, so the worlds evolve alongside the agent [§01 History is the world to dream in, §Recursive self-improvement on the meta-exploration layer].
The learned policy is adaptive, not monotonically greedier or broader: as scores improve it first conserves compute (evaluated attempts drop from 110 to 50); when progress plateaus it re-widens, and those widenings coincide with the next jumps in round-best score [§What the learned policy actually does].
On algorithm engineering (Lasso), Dream-RSI with Gemini-3.7-Flash reaches average runtime 2350.6 ms across six held-out datasets at 1,879 discovery-agent calls, vs SimpleTES (gpt-oss-120b) 3804.8 ms at 51,200 calls — a ~1.6× quality gain at ~27× less compute; the reproduction SimpleTES† at 51,200 calls averages 8318.4 ms [§Algorithm engineering, Table].
On mathematical optimization Dream-RSI (Gemini-3.1-Pro) matches or edges every prior AlphaEvolve-family / SimpleTES / ThetaEvolve system on Sum Diff (1.145427), Circle Packing (2.635983), and stays within 0.003 of the SimpleTES record on Auto Correlation (1.456375 vs 1.453675) at <1,000 generations vs SimpleTES’s 51,200 [§Mathematical optimization, Table].
Injecting semantic directional guidance (abstracted high-level insights from prior trajectories, prompt-conditioned) underperforms its unguided counterpart under equal budgets in both paradigms: strong semantic priors over-constrain diverse exploration in long-horizon multi-thread discovery, so replay-based off-policy feedback strictly dominates prompt-injected reflection at this scope [§Semantic guidance is worse than replay].

Method

Dream-RSI sits as a lightweight orchestration layer around an unchanged coding agent (Gemini-3.1-Pro or Gemini-3.7-Flash in the reported experiments) and a fixed evaluator that scores discovered candidates. Exploration is factored out as executable code — the policy decides which attempt to continue next, how many attempts to run in parallel, and when to stop — which makes it a first-class object the outer loop can rewrite.

Each outer iteration t of the loop looks like this. Online phase: the current exploration policy π_t drives the coding agent through a discovery run, producing a tree 𝒯_t whose nodes carry every branching decision the policy made together with the code-execution outcome that decision produced. 𝒯_t is appended to the history pool ℋ_t = (𝒯_1, …, 𝒯_t). Offline phase: a policy-development agent writes M successive revisions π⁰ = π_t, π¹, …, π^{M-1} of the exploration policy’s code. Each revision is scored by replaying it over the whole history pool ℋ_t — walking the same recorded trees in a different order, with different parallel groupings and different stopping decisions. Because every outcome an alternative policy would query at any tree node has already been executed and cached, replay returns immediately at zero new executions. The winner π_{t+1} = π^{m*} is redeployed online for iteration t+1; the incumbent’s presence in the candidate slate guarantees π_{t+1} is never worse than π_t.

Eight discovery tasks are evaluated across three domains: algorithm engineering (Lasso regularization path on Gisette / RCV1 / DNA / Leukemia / Colon / Duke Breast), mathematical optimization (Sum Diff, Auto Correlation, Circle Packing), and GPU kernel engineering (VGG16, ConvDiv). The controlled ablation baseline is Recursive Fixed Exploration — identical agent, evaluator, initialization, and per-round budget, but the exploration policy is never rewritten; both methods start from the same hand-written parallel-refine policy so round 1 is identical by construction. External baselines include AlphaEvolve, AlphaEvolveV2, OpenEvolve, CodeEvolve, ShinkaEvolve, TTS-Discovery, ThetaEvolve, EvoX, and SimpleTES for math optimization; sklearn, glmnet, and SimpleTES for Lasso.

Results

Algorithm engineering — Lasso. Dream-RSI (Gemini-3.7-Flash, 1,879 discovery-agent calls) reaches average runtime 2350.6 ms across six held-out downstream datasets, vs Recursive Fixed Exploration (Gemini-3.7-Flash, 3,200 calls) 2516.7 ms, SimpleTES (gpt-oss-120b, 51,200 calls) 3804.8 ms, and the paper’s own SimpleTES† reproduction at 51,200 calls 8318.4 ms. Dream-RSI with the stronger Gemini-3.1-Pro at only 317 calls averages 2931.0 ms — 162× fewer discovery-agent calls than SimpleTES on Lasso at comparable or better runtime [§Algorithm engineering, Table 1].

Mathematical optimization. Dream-RSI (Gemini-3.1-Pro) attains Sum Diff 1.145427 (best in table), Circle Packing 2.635983 (ties with SimpleTES / ThetaEvolve / CodeEvolve / TTS-Discovery), Auto Correlation 1.456375 (best in table when the 51,200-generation SimpleTES record 1.453675 is excluded; within 0.003 of it). Recursive Fixed Exploration at the same budget lands at 1.144047 / 1.456001 / 2.635983 — Dream-RSI’s advantage is largest on Sum Diff [§Mathematical optimization, Table 2].

GPU kernel engineering. Reported headline gains: 2.43× fewer generations on VGG16 at comparable performance and 2.09× higher score on ConvDiv at a comparable budget [§Three discovery domains headline callouts].

Semantic-guidance ablation. Applied to both Dream-RSI and Recursive Fixed Exploration, injecting abstracted high-level directional insights into the prompt consistently underperforms the unguided counterpart at equal budget, which the paper interprets as strong semantic priors over-constraining diverse exploration [§Semantic guidance is worse than replay].

Learned-policy behaviour. The learned exploration policy is adaptive rather than monotonically greedier / broader — evaluated attempts drop from 110 to 50 as scores improve, then re-widen at plateaus, with those widenings aligning with the next jumps in round-best score [§What the learned policy actually does].

Why it’s interesting

Dream-RSI sits squarely in AI-for-AI Research but stakes out a design point that none of the prior filings on that page occupy. AIDE²: First Evidence of Recursive Self-Improvement (AIDE²) also runs a meta-optimizer over an inner discovery agent and reports the first evidence of RSI, but its outer loop invokes real rollouts of candidate inner agents — expensive, and the reason Weco explicitly declined Level 2 RSI when the ignition test showed no asymptotic gain. Dream-RSI’s contribution is that the meta-evaluation is off-policy replay over already-executed discovery trees, so screening thousands of candidate exploration policies costs zero executions and only the winner pays. This is a plausible reason the reported cost ratios (162× fewer calls than SimpleTES on Lasso) sit an order of magnitude beyond AIDE²’s budget savings.

The framing also complements The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement‘s five-stage RSI autonomy roadmap: Dream-RSI is a concrete instantiation of stage-3 “experience autonomy” (agent chooses what experience to accumulate and how to use it) that leaves the coding agent itself unchanged — the improvement lives entirely in the outer exploration policy, not in weights or in the inner agent’s prompt. That is a different point on the loop-locus axis from Ornith-1.5: From Self-Scaffolding to Self-Improvement (loop moves into pretraining), AREX: Towards a Recursively Self-Improving Agent for Deep Research (loop lives inside a single task via a learned improvement-state buffer), and The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation (loop lives inside inference-time multi-turn planning).

Practically for Luma: the “replay discovery trees as an exact off-policy simulator” primitive is directly usable in any research infrastructure that already logs branching decisions with per-node outcomes (RL-training sweeps, kernel-tuning sweeps, prompt-optimization runs), and the ablation showing prompt-injected semantic reflection hurts diverse exploration is a useful counterweight to the “give the agent a summary of what it tried” default that pervades agent scaffolding.