AREX: Towards a Recursively Self-Improving Agent for Deep Research

ListenAREX: The Agent That Checks Its Own Homework10:14

Show notes

AREX is a family of "Recursively Self-Improving" deep research agents that splits the ReAct loop into an inner search loop and an outer self-improvement loop that audits a provisional answer constraint-by-constraint, then launches targeted follow-up research. To keep this viable over long horizons, the agent learns its own context-update tool — folding history compression into the policy instead of a separate summarizer — and is trained with synthetic-task mid-training plus long-horizon RL that rewards decisive-evidence and directional-correction steps. We unpack the two-loop design, the training recipe, and what the "verify-then-refine" idea might mean for image and video generation work. **Sources** - AREX: Towards a Recursively Self-Improving Agent for Deep Research

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

AREX is a family of “Recursively Self-Improving” (RSI) deep research agents that separates the standard ReAct loop into an inner research loop (search + provisional answer) and an outer self-improvement loop that audits the provisional answer constraint-by-constraint, identifies unresolved claims, and launches targeted follow-up research. To keep the outer loop viable over long horizons the agent learns an autonomous context-update tool that compresses the growing interaction history into a compact improvement state — preserving verified evidence and unresolved constraints without a separate compression model. The recipe is a synthetic-task mid-training stage plus long-horizon RL with dense rewards on decisive-evidence steps. A 4B dense and a 122B-A10B MoE variant are trained; both are reported to substantially outperform comparable-scale baselines and stay competitive with much larger models on BrowseComp, WideSearch, DeepSearchQA, HLE, and other reasoning + tool-use benchmarks [Abstract].

Key claims

Deep research has a discovery–verification asymmetry: finding an answer that jointly satisfies multiple constraints is costly, but verifying a candidate can be decomposed into tractable constraint-wise checks — motivating a two-loop architecture where verification of a partial answer guides subsequent search [Abstract].
The outer self-improvement loop audits an inner-loop candidate constraint-wise, flags unresolved claims, and dispatches targeted follow-up research rounds — as opposed to simply searching longer or scaling context [Abstract].
To sustain RSI over long horizons AREX learns an autonomous context-update tool that compresses interaction history into a compact improvement state containing verified evidence and unresolved constraints — and does so without relying on an external compression model [Abstract].
The training recipe is agentic mid-training on verified synthetic tasks and high-quality trajectories, followed by long-horizon RL with reward shaping that emphasizes decisive-evidence steps and directional-correction steps to mitigate sparse final-answer rewards [Abstract].
Two model instantiations are released: a 4B dense model and a 122B-A10B Mixture-of-Experts model [Abstract].
Across BrowseComp, WideSearch, DeepSearchQA, HLE, and other reasoning + tool-use benchmarks, AREX substantially outperforms comparable-scale baselines and remains competitive with models using substantially more activated parameters [Abstract].

Method

The high-level architecture is a nested two-loop agent. The inner research loop is a standard ReAct-style trajectory that gathers web evidence with tool calls and produces a provisional answer for the current improvement state. The outer self-improvement loop takes that provisional answer plus the interaction history, runs a constraint-wise audit, and produces an updated improvement state — a compact summary containing the verified evidence supporting each satisfied constraint and the outstanding claims that still need to be checked. Follow-up inner-loop rounds are launched conditioned on the updated state, targeting the unresolved constraints rather than restarting search from scratch. The load-bearing engineering piece is the autonomous context-update tool that produces the improvement state end-to-end from within the agent, avoiding the two-model pattern where a separate summarizer manages the running context. Training has two stages: (1) agentic mid-training on verified synthetic tasks and curated high-quality trajectories, giving the model priors for verification and context-update; (2) long-horizon RL where sparse final-answer rewards are supplemented by dense credit on the decisive-evidence steps (steps that acquire load-bearing evidence) and the directional-correction steps (steps that back out of erroneous research directions). Two backbones are trained: a 4B dense and a 122B-total / 10B-active MoE.

Results

Headline claims from the abstract (specific numbers not disclosed in the abstract itself): AREX substantially outperforms comparable-scale baselines on BrowseComp, WideSearch, DeepSearchQA, and HLE, and remains competitive with substantially larger active-parameter models. The 4B is positioned as the small-model datapoint and the 122B-A10B MoE as the larger one; direct per-benchmark deltas will need to be pulled from the full paper (§Experiments) once the PDF is read in a follow-up refresh. The reward-shaping section is claimed as necessary to close the sparse-reward gap on long-horizon rollouts, but ablation numbers are not stated in the abstract [Abstract].

Why it’s interesting

AREX is the fifth filed deep-research agent in the wiki alongside MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling (MiroThinker), Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design (Marco DeepResearch), Tongyi DeepResearch: A New Era of Open-Source AI Researchers (Tongyi DeepResearch), and Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (SMTL), and it stakes out a distinct design choice on the same benchmark surface (BrowseComp / HLE / GAIA-family). Where MiroThinker’s central claim is that interaction depth is a third scaling axis (600 tool calls, recency-based retention), Marco-DeepResearch’s is verification as the load-bearing gate at every stage (adversarial QA synthesis, verifier sub-agents, verifier-guided TTS with “Discard All”), and SMTL’s is parallel branching over ReAct paths, AREX’s is a two-loop within-task iterative refinement architecture with a learned context-update tool — closer to Marco-DeepResearch’s “Discard All” primitive but generalized from a single context reset into a persistent, compact improvement-state buffer that survives across outer-loop iterations. The learned autonomous context-update tool is also a data point for the broader question of whether long-horizon agents should use a separate summarizer model (the common pattern) or fold the compression into the policy itself; the “RSI” branding here is a within-task refinement loop and does not fit the AI-for-AI Research cluster (which is about outer loops that improve the agent’s own weights/harness across runs) — worth flagging because the terminology overlap invites confusion with AIDE²: First Evidence of Recursive Self-Improvement and History May Repeat Itself: RSI Seen from a Previous AI Era, which use “RSI” in the harness-improves-harness sense.