Skip to content
Bud's Wiki

AI-for-AI Research

TL;DR

Section titled “TL;DR”

Multi-agent LLM scaffolds that autonomously run a closed loop of hypothesis → experiment → analysis → next hypothesis against a GPU cluster, with the system itself (not a human) deciding what to try next. The three filed instances operate at very different granularities — single training-script knob (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster), novel neural-architecture proposal (AlphaGo Moment for Model Architecture Discovery), and full short-paper production (FARS: Fully Automated Research System) — but converge on the same scaffold pattern: a small number of role-specialized agents (Researcher / Engineer / Analyst, or finer splits) coordinating through a shared memory or filesystem, with experiment results feeding back as future-round context.

Key claims

Section titled “Key claims”
  • The scaffold pattern that recurs across all three is a small set of role-specialized agents (3–4) with a shared memory or filesystem as the only coordination substrate — no direct agent-to-agent messaging, no central orchestrator state (FARS: Fully Automated Research System §Method four-agent Ideation/Planning/Experiment/Writing; AlphaGo Moment for Model Architecture Discovery §Method three-agent Researcher/Engineer/Analyst with MongoDB memory; Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Method single Claude Code agent with SkyPilot skill + the train.py file as shared state). Kosmos uses a two-agent split (data-analysis + literature-search, parallelized ~10-wide per cycle) and replaces the unstructured shared filesystem with a structured world-model database of entities/relationships/results/open-questions, updated after every task and queried as the planning substrate (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • All three systems run unbounded-throughput experiments on dedicated GPU clusters at small unit cost per experiment: FARS at ~1K/ 2hperpaperon160GPUs([[blog−analemma−fars−fully−automated−research−system−2026−02]]Results),ASI−ARCHat 20,000GPU−hours/1,773experiments/106SOTAarchitectures( 190GPU−hoursperSOTAhit)([[2507.18074−alphago−moment−model−architecture−discovery]]§EmpiricalResults),Karpathyautoresearchat 1K / ~2h per paper on 160 GPUs ([[blog-analemma-fars-fully-automated-research-system-2026-02]] Results), ASI-ARCH at ~20,000 GPU-hours / 1,773 experiments / 106 SOTA architectures (~190 GPU-hours per SOTA hit) ([[2507.18074-alphago-moment-model-architecture-discovery]] §Empirical Results), Karpathy autoresearch at ~300 GPU + 9API/910experiments/8hon16GPUs( 9 API / 910 experiments / 8h on 16 GPUs (~0.33 per experiment) (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Cost). Kosmos is the outlier on dollars/run: ~$200/run on the Edison Scientific platform (commercial pricing), with cost dominated by LLM API calls rather than self-hosted GPUs (Kosmos: An AI Scientist for Autonomous Discovery §1 reads ~1,500 papers + 42,000 LOC executed per run).
  • A pre-experiment gate that rejects proposals before any GPU time is allocated appears in the two larger-grain systems: ASI-ARCH runs vector-similarity novelty check + static-analysis on complexity / causal masking (AlphaGo Moment for Model Architecture Discovery §Method Step 2), and FARS runs automated review of hypotheses against the literature before forwarding to the Experiment agent (FARS: Fully Automated Research System §Method). The single-knob Karpathy variant skips this — there’s no novelty gate because the search space is so small. Kosmos has no analogous pre-experiment gate either; the world-model query is the post-hoc planner and accepts whatever the parallel rollouts return (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • The output is treated as a system-level product, not as conventional research artifacts: FARS explicitly publishes negative results, doesn’t conform to academic paper structure, and uses livestream + public corpus for evaluation (FARS: Fully Automated Research System §Building from First Principles); ASI-ARCH ships 106 architectures + a MongoDB database + Cognition Base under a permissive license (AlphaGo Moment for Model Architecture Discovery Repository). Kosmos diverges — it produces three to four traditional-style scientific reports per run with every statement and figure linked to either a literature citation or a specific generated Jupyter notebook, optimized for human-expert audit rather than open-corpus throughput (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • ASI-ARCH’s provenance analysis (AlphaGo Moment for Model Architecture Discovery §Analysis) reports that ~45% of design choices in winning architectures trace to lessons mined from prior experiments and ~49% to systematic reasoning, with only ~7% from novel ideas — and that losing architectures lean more on novelty. The cluster’s signature is that the loop wins by mining its own history, not by being creative.
  • Emergent strategy unprompted by the human: in Karpathy autoresearch the agent discovered H100 vs H200 hardware differences (without being told) and developed a two-tier validation strategy — screen on H100s, validate winners on H200s (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies). Adjacent claim from ASI-ARCH: gating layers + small convolutions dominate the winners and “exotic” primitives (spectral, physics-inspired) rarely make it into the top 106 (AlphaGo Moment for Model Architecture Discovery §Analysis figure) — the loop converges on a tight, proven toolkit.
  • A scaling-law-style headline appears in three of the four filings: ASI-ARCH reports a roughly linear relationship between compute spent and number of SOTA hits (AlphaGo Moment for Model Architecture Discovery §Scaling-law plot); Karpathy autoresearch reports a 9× experiment-throughput multiplier from parallel-GPU scaffolding (90/hr vs 10/hr single-GPU) and a qualitative shift from greedy hill-climbing to factorial grids (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Results); Kosmos reports that expert-equivalent research time and the count of “valuable findings” scale roughly linearly with cycle count up to 20 cycles (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1e–f). FARS does not yet report a quality-vs-compute curve.
  • New claim (correctness ceiling): When a closed-loop AI-scientist system is graded statement-by-statement by independent domain experts (the underlying code/literature withheld during evaluation), 79.4% of statements in Kosmos reports are judged accurate, with a sharp split by statement type: 85.5% for data-analysis-grounded, 82.1% for literature-grounded, and 57.9% for cross-domain synthesis statements (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1c). This is the first filed AI-for-AI-research system with such a per-statement human-graded accuracy number; the synthesis-vs-grounded gap is the load-bearing failure mode.

Recent contributions

Section titled “Recent contributions”
  • [2026-11-06] Introducing Inkling 2 Small — Thinking Machines' 594B/17B MoE with unified RL at 4e23 FLOPs: Inkling 2 Small is trained on long-horizon autoresearch tasks including agentic GPU kernel optimization and ML tasks such as post-training a model autonomously across a range of GPU architectures. End-to-end demo on a Proximal task: Inkling 2 Small autonomously fine-tunes Qwen3.5-4B on Tinker to predict time-series (weather, Wikipedia page views) — writes its own evaluator, experiments with training datasets and recipes, iterates SFT and RL, debugs end-to-end. After 4.5 hours of work and $426 in Tinker spend, trained model more than doubles its forecasting score. PMPP-Hard AUC 494.27 task-hours over 0-10h (four independent attempts, 69 kernel-optimization tasks, shared harness and compaction policy) — directly comparable in format to Measuring Autonomous AI Research‘s nanoGPT-speedrun autoresearch comparison, though on GPU-kernel-optimization rather than LM-pretraining as the autoresearch substrate.
  • [2026-11-04] Pranay Gupta — direct weight manipulation of π0.5 beats fine-tuning on cloth folding (50% → 68% FT → 75% weight edit): First filed transfer of the Weight-Space Programming paradigm (Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights) across the model-class gap (LLM → flow-matching VLA) and benchmark gap (MATH-500 / GSM8K → cloth folding on RoboDojo): π0.5 weight-edit 75% vs FT on Astra corrections 68% vs base 50%. If it holds, extends the LLM-as-outer-loop-optimizer ladder from Qwen3.5-0.8B reasoning benchmarks onto a production robotics VLA on a contact-rich manipulation task. Tweet-only; full writeup pending.
  • [2026-11-02] Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights: Weight-Space Programming (Takehi, Teoh, Zhang, Khattab; MIT CSAIL) extends the “LLM as outer-loop optimizer” ladder one layer closer to the metal than any filed instance — GPT-6 Astra edits raw model weights with no gradients, no optimizer code, and no training step, raising Qwen3.5-0.8B MATH-500 7.9%→48.1%, GSM8K 13.8%→67.1%, IFEval 21.0%→62.3% by modifying 0.016% of weights to fix a self-locking reasoning-loop pathology within a 50 forward-pass budget. Sits below AIDE² (harness rewrites) and OPAL (optimizer-code rewrites) on the “how close to the model internals does the outer loop reach” axis. Null result bounds the claim: Gemma 3 270M (no loop pathology to remove) does not improve — editing removes obstacles that hid an ability but cannot add missing ability.
  • [2026-10-28] Tatsu Hashimoto — AI-assisted humans, not autoresearch, drive NanoGPT speedrun gains (Herman Brunborg 39.9s → 6.7s stamp): Tatsu Hashimoto’s stamp of Herman Brunborg’s 39.9s → 6.7s NanoGPT speedrun record (bigger than every 2025–2026 record combined via retrieval-augmented n-gram hints) comes with the sharp attribution “big gains in nanoGPT have come from AI assistance of humans instead of autoresearch” — the strongest filed adversarial datapoint for the autonomous-agent axis of this cluster, directly contrasting with Measuring Autonomous AI Research‘s finding that 153 autonomous runs across 18 frontier models produced no fundamentally new method on the same leaderboard.
  • [2026-10-28] Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement: COAP (Hu, Hu, Hong, Chen, Liu; NTU S-Lab) proposes code (not weights, not harness state) as the right substrate for recursive self-improvement in the physical regime because every library edit is a code diff and every outcome is a graded rollout — the operational realization of the two-round frozen-knowledge discipline Toward Recursive Self-Improvement in Physical AI argues the field is missing. Sibling to RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments RoboRSI (four-agent harness over four-level skill tree with real-execution validation) and to ASPIRE: Agentic /Skills Discovery for Robotics ASPIRE (self-evolving sensorimotor skill library) in treating the growing code artifact as the carrier of capability — but takes the “no model at test time” position to the limit, reporting 70.24% on RoboDojo’s 42 bimanual tasks with no learned policy in the control loop. Open empirical question the paper raises: whether the authored library actually realizes compounding improvement across tasks (via the “Extensibility” property — new tasks reuse/inherit/extend prior modules) rather than each task-extension being orthogonal.
  • [2026-10-23] Toward Recursive Self-Improvement in Physical AI: Luo, Bai, Xu et al. (NUS + HUST + SJTU + NTU + HKBU) publish the first filed physical-regime survey of RSI, complementary to The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement‘s general five-stage autonomy roadmap. Supplies a condition-on-four-gates framing (feedback addresses remaining learning needs; receiving component can use the change; value survives task/hardware changes; continued practice stays safe and affordable), a four-way decomposition of retained change (policy updates / world models / memories-skills-code / learning procedures), and — the sharpest new evidence-standard contribution — a two-round frozen-knowledge test as the operational definition of recursion in the physical regime: repair lessons from round one must transfer to new failures in round two under a controlled comparison with a version of the improver whose acquired knowledge is frozen. Reviews the physical-RSI literature (concentrated in manipulation) and reports behavioral improvement, later-learning benefits, and procedure improvement are demonstrated, but no reviewed physical study isolates a gain attributable to recursion itself under this control — the first filed articulation of the measurement gap that cells like ENPIRE / ASPIRE / GaP / RoboRSI / GRID Auto-Engineering on this board haven’t yet closed. The two-round discipline maps directly onto the AIDE² “ignition test” (install discovered agent in outer-loop seat) already on the page but was previously under-named for physical loops. Four open-problem starting benchmarks named: refresh-the-judge/keep-the-task evaluator-only variation; policy lifecycle under declared changes + gradual drift; Pareto cost reports including robot time/attention/wear/compute; and which-component-limits-improvement injected-fault + held-out-failure probes.
  • [2026-10-22] OPAL: a reactive optimizer with an agent in the loop: OPAL (Bai, Liang, Teoh, Sitzmann; MIT) is the first filed instance where the agent-in-the-loop target is the optimizer code of a running training process — a coding agent (Claude Code / Codex) rewrites updater.py and analysis.py at scheduled decision points inside a single conversation that persists for the entire run. Sits one layer closer to the metal than AIDE²: First Evidence of Recursive Self-Improvement AIDE² (which optimizes an autoresearch harness across runs) and than Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster Karpathy autoresearch (agent provisions its own GPU cluster but between-run). Reported single-run gains over official recipes include SiT-B/2 FID 35.7 → 23.6, LightningDiT-B/1 FID 3.22 → 2.31, and a modded-nanogpt validation loss (3.2749) slightly better than record #36 (3.2781); the authors also honestly report an earlier OPAL 0.2.0 modded-nanogpt run at 3.2878 (worse than the record), making this the only intra-cluster filing with an explicit negative-result row.
  • [2026-10-16] ASPIRE: Agentic /Skills Discovery for Robotics: ASPIRE paper (NVIDIA GEAR + UMich + UIUC + Berkeley AUTOLab + CMU LeCAR) is the full technical write-up of the concept Jim Fan announced in June and provides Table 1 — a sim-to-real cross-embodiment transfer measurement that is the sharpest filed “measurable next-round improvement from re-injecting prior-round outputs” datapoint on the physical side (~9.4× debugging-token reduction on soda-can lifting, 0/20 → 11/20 on drawer manipulation). Sits one layer below AIDE² on the RSI granularity axis: AIDE² optimizes the outer autoresearch harness, ASPIRE optimizes the inner-loop skill library consumed by actor coding agents. V1.0 open-sourced 2026-10-16 at github.com/nvlabs/aspire; the README reports 99–100% on LIBERO-Pro / Robosuite / BEHAVIOR-1K subsets when the backend model is Claude Opus 5.5 or GPT-6 Astra, plus a real-robot Rubik’s Cube solve.
  • [2026-10-15] RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents: RoboSimStudio (UC Berkeley + NVIDIA) adds a seventh distinct physical-autoresearch cell alongside ENPIRE (fixed VLA + RL), RATs (play-time skill library), GaP (task-specific skill graph), GRID Auto-Engineering (full robotics stack), DexAgent (data-pipeline tool library), and ASPIRE (sensorimotor skill evolution): the coding agent (Claude Code / Codex) authors the physics simulator itself from a short smartphone video in ~1 hour, and the simulator’s parameter/layout variations drive a downstream Graph-as-Policy update loop that closes back on the real robot. First filed instance where the autoresearch artifact is the per-episode multi-physics world (Newton rigid + SuperDex FEM), not the policy, the data, or the solve program.
  • [2026-10-15] SwarmRSI: Recursive Self-Improvement through Evolving Scientific Agent Swarm: SwarmRSI (Meng, Zhang, Papakostas et al.) adds swarm-level evolution as a new granularity: each agent carries a (hypothesis, implementation) pair, and Feedback-Guided Swarm Mutation + Complementary Swarm Recombination + Recursive Swarm Evolution combine so cross-agent feedback refines pairs and useful components from losing runs still propagate — a structural counter-recipe to best-of-N that complements Fork-and-Flush’s schedule-level counter-recipe. Reports SOTA on OpenProblems/NatureBench/MLE-Bench Lite, transfer to held-out tasks via accumulated research experience, and biologist reproduction of generated programs for downstream analyses.
  • [2026-10-14] Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents: Fork-and-Flush (Microsoft Research et al.; Cai, Ziakas, Kontonis, Pearce, Sen, Krishnamurthy, Garg, Papailiopoulos) introduces a scheduling-axis intervention distinct from the scaffold-shape work that dominates this cluster: at fixed intervals, fork the autoresearch agent into parallel trajectories that inherit the accumulated workspace but start with a fresh chat context, run each for a fixed horizon, and continue from the best scorer. Reports a +66.0% relative gain over single-run and +44.4% over best-of-N on min-max-normalized average score across 13 long-horizon research/engineering tasks at equal compute. The embedding-space “idea basins” diagnostic (independent runs plateau at different scores and remain in localized regions of candidate-artifact space) is the first filed mechanistic evidence for the concept page’s open question about whether parallelism changes what search the agent does, not just how fast — and gives a sharp counter-recipe to best-of-N by sharing workspace while only resetting conversational context.
  • [2026-10-13] Gemini 4 Argon for Robotics: Argon reproduces robotics-control papers end-to-end from a single prompt into self-contained MuJoCo WebAssembly browser labs — matching the paper’s exact physics timestep, actuator gains, torque limits, and kinematic constraints; running real neural inference or Web Worker iLQR in JavaScript at the paper’s control rates; shipping both an interactive simulator and a scrollytelling essay; and verifying 0 console errors + non-zero task progress over 300+ frames via headless Chromium (Playwright). Named reproductions span DeepLoco (7-network hierarchical locomotion), HUSKY (50 Hz PPO), and Whole-Body MPC (Web Worker iLQR). Also iteratively self-improves on a 6-DoF arm by reviewing its own wrist/overhead footage to diagnose failures and makes the loop ≥2× faster subject to safety rules.
  • [2026-10-12] RoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments: RoboRSI adds a fifth cell to the recursive-self-improvement board distinct from AIDE²: First Evidence of Recursive Self-Improvement AIDE² (RSI over an autoresearch coding harness), Dream-RSI: Recursive Self-Improvement through Evolving Worlds Dream-RSI (gradient-free meta-opt of a coding-agent exploration policy), Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edge Sol-H3 (RSI over deployment-stack code), and SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness SoL-Pi (RSI over an agent-harness itself): the autoresearch loop here is over physical-skill code and the validation gate is live-environment execution, not a sim scoreboard or a model-weight-benchmark pair. The reported iteration-success metric is deliberately iteration-time rather than one-shot: a revision counts not only when the current task succeeds but when repeated execution, extra verification, model fallback, human intervention, and code branching all decrease over subsequent rounds. Tighter counter-recipe to pure sim-eval RSI with per-rollout environmental non-stationarity as both cost and realism, and complements ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR) ASPIRE’s self-evolving sensorimotor skill library with an explicit four-agent attribution + validation structure.
  • [2026-10-02] EMPIRIC — agents that write physics-engine extensions as code to learn unknown dynamics: EMPIRIC (Yichao Liang) extends auto-research from the LLM-architecture / RL-harness domains into physics discovery: an agent proposes experiments on a real system, writes physics-engine code to explain outcomes, maintains a Bayesian posterior over its own program’s parameters, and transfers the learned program to unseen configurations and to a real robot (wind + domino masses from two gusts). The ablation showing both explicit uncertainty (25 → 21) and harness-fitting (25 → 20) are load-bearing is relevant to the broader auto-research literature’s debate about whether novelty-check gates alone suffice.
  • [2026-10-02] Paper2Agent — Reimagining Research Papers As Interactive AI Agents: Paper2Agent (Stanford; Miao, Davis, Zhang, Pritchard, Zou — Nature 2026, MIT) sits on the writer side of the paper↔agent interface, complementary to Kosmos’s reader-side AI-scientist loop: instead of running analyses to produce reports, it packages a single paper’s methods into an MCP server tested against the paper’s own reference outputs, so downstream agents can call them. Adds a new granularity to this cluster’s existing scope ladder (one knob / one architecture / one paper / one discovery) — per-paper method extraction with correctness contract = reproduces paper’s reference outputs. First filed instance where the “AI-for-AI-research” loop’s output is a tested tool-surface for other agents, not a model, dataset, or write-up. Nature ablation shows the pipeline works even with tutorials stripped (README + source only) on POP-TOOLS across five tasks.
  • [2026-09-30] Beyond kernels: Letting agents optimize the whole AI job: OpenPerfAgent’s /who-ate-my-flops is a portable deployment-stack agent — a Claude Code / Codex plugin that ports the Sol-Engine / Asari deployment-stack autoresearch pattern to arbitrary upstream OSS training + inference repos (FunASR, FastVideo, gsplat, Ultralytics, Unsloth, SGLang) rather than a single vertically-integrated inference stack. Six upstream-merged PRs on B200 with up to 3.6× speedup, correctness gated by a companion parity tool that records/compares intermediate tensor values at declared rtol — a middle point between Asari’s C4 distribution-matching gate and Sol-Engine’s perceptual-validator loop.
  • [2026-09-30] Sol-H3: Recursive Self-Improvement for MiniMax-H3 Inference Acceleration on Sol-Engine across Cloud and Edge: Sol-H3 sharpens the deployment-stack granularity Sol Video Inference Engine (Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation) opened: where Sol-Engine autoresearched which of five levers to compose, Sol-H3 explicitly runs a Recursive Self-Improvement loop over kernel fusions and memory layouts themselves, scoring candidates jointly on latency and numerical agreement with a reference implementation. First filed instance of RSI applied to the low-level operator layer of a diffusion inference stack under a numerical-agreement contract — a stricter shippability gate than the perceptual human-validator loop Sol-Engine used, and structurally closer to Asari’s distribution-matching correctness gate (Speeding up end-to-end inference with self-improving agents) than to ASI-ARCH’s architecture-fitness signal.
  • [2026-09-29] DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library: DexAgent (Stanford; Wang, Li, Fei-Fei, Wu, Huang) applies the self-evolving-code-library recipe to a narrow physical subdomain: the agent writes new tools for the Human2Sim2Robot data pipeline when the existing library falls short, and retained skills + verifiers compound so per-video processing time decreases as more videos are processed. Sibling in spirit to ASPIRE (ASPIRE — self-evolving sensorimotor skill library that compounds across robots and the sim-to-real gap (Jim Fan / NVIDIA GEAR)) and RATs (Playful Agentic Robot Learning) at the sensorimotor-primitive layer, but scoped to authoring perception/simulation/retargeting tools for a data-generation pipeline rather than authoring control policies directly.
  • [2026-09-24] Primus Society: 10,000 Autonomous AI Researchers: Primus Society (Transformer Lab; Asaria) is the largest-agent-count filed instance in the cluster — 10,000 Primus copies organized as human-science-style institutions (labs with mandates, three-judge grant panels, a central library, conferences, a public paper board) with typed roles (Mavericks / Followers / Skeptics), persistent identities, and competition for a finite compute budget. Distinct on two axes: (a) coordination substrate is institutional (grants, peer review, published-and-replied paper board) rather than shared filesystem (FARS/ASI-ARCH) or structured world-model database (Kosmos); (b) Skeptics explicitly spend compute trying to break others’ claims and negative results / refuted predictions are published alongside wins — the first filed system to wire disconfirmation into the reward structure, a direct answer to the concept page’s evaluation-bottleneck open question. Reports an autonomously-derived growth-based pretraining recipe (17% lower perplexity vs from-scratch at matched budget, ~30% less compute to reach the same quality) decomposed across 6+1 sub-papers with per-operator mechanism claims (depth wins with late timing, width gains nothing, expert-upcycling pays back after ~0.0076 tokens/param) and follow-up calls written against unresolved controls (Call 3 funds six labs to isolate carried-weights vs LR-schedule). No dollar figure, GPU-hour count, model scale, or evaluator disclosed in the public post.
  • [2026-09-21] SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness: SoL-Pi (NVIDIA / MIT / HKU et al.; Liu, Ye, Gao, …, Xie, Han) is the token-cost counterpart to AIDE² and AutoDesign — an RSI-inspired outer loop pointed only at the coding-agent harness (model, task prompts, and inner-agent logic held fixed) that scales rollouts across many environments so only transferable harness changes survive. Four mechanisms make the cut — action execution, context compaction, observation handling, delegated reading — and on the 51-task EdgeBench evaluation the discovered harness matches Pi on both GPT-5.6 Sol and Opus 5 while cutting recorded token traffic 44.7–49.0% and API cost ~one third; the headline is not capability gain but a data-driven decomposition of where a modern coding agent’s token budget goes, with estimated $8.75–$13.50/hour savings vs native Codex / Claude Code and $4.36–$5.71/hour vs Pi. Adds a cost-axis granularity to the cluster that AIDE² (agent-source rewrites, quality) and AutoDesign (design harness, quality) don’t currently occupy.
  • [2026-09-16] Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Dream-RSI (Google / UMD / DeepMind / UVA; Zheng, Wu, Zhang et al.) is the closest sibling to AIDE² in the “outer loop optimizes the exploration policy” family, but the meta-evaluation is off-policy replay over already-executed discovery trees rather than fresh online rollouts of candidate agents — screening thousands of candidate exploration policies at zero new executions and only paying a rollout for the winner. Reports 162× fewer discovery-agent calls than SimpleTES on Lasso, 2.43× fewer generations on VGG16, 2.09× higher ConvDiv score at comparable budget, and ties AlphaEvolve-family SOTA on Circle Packing / Sum Diff / Auto Correlation at <1,000 vs SimpleTES’s 51,200 generations. Adds two ingredients for the concept’s Open Questions: (a) a formal monotonicity guarantee — because the incumbent is always in the candidate slate, the winner is never worse than the deployed policy; (b) an ablation showing prompt-injected semantic reflection over prior trajectories hurts diverse exploration vs the unguided replay baseline, a counter to the “summarize what the agent tried” default in agent scaffolding.
  • [2026-09-16] Wentao Zhu — GPT-6 Astra autonomously trains an Isaac Lab RL policy for pen spinning on the Sharpa hand over 1.5 days: Wentao Zhu’s fourth GPT-6 Astra test: one-sentence prompt asking Astra to implement pen spinning with the Sharpa dexterous hand in Isaac Lab, author the pen mesh, and produce a trained RL policy + video — Astra ran autonomously for ~1.5 days (including web/paper search and the RL training itself) and delivered both artifacts. First filed instance where the autonomous-research loop closes end-to-end through simulated physical robotics rather than an ML-training script (Karpathy autoresearch) or a working-policy improvement loop (ENPIRE, GaP). Qualitative single-run demo with no baselines, no metrics, and no released harness — a capability probe that sits between the Karpathy-style ML-autoresearch cluster and the ENPIRE-style physical-autoresearch cluster on the concept’s granularity axis.
  • [2026-09-14] The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement: Position survey formalizing RSI with a five-stage autonomy roadmap (execution → strategy → experience → environment → recursive meta-improvement) and a Headroom-Closed Index (HCI) for cross-system comparison — orthogonal to Yuandong Tian’s ICLR-2026 RSI-workshop framing (data/memory/search/cost ingredients) and directly usable as a sharper vocabulary for the concept page’s current informal “granularity” axis. Argues the surveyed systems cluster naturally by scenario (scientific discovery vs embodied vs software engineering), matching the wiki’s own observation that Kosmos/Hyra, ENPIRE/GRID Auto-Engineering, and ASI-ARCH/Kimi-K3 kernel work each target a different scenario with different requirements.
  • [2026-09-10] Introducing Auto-Engineering for Robotics: General Robotics’ GRID Auto-Engineering is the widest-scope filed physical-autoresearch loop yet: four robotics harnesses (Robot Ingestion / World Experience / Skill Creation / Deployment & Evaluation) over a 50+ OEM monorepo, closing the ASI-ARCH-style hypothesize→experiment→analyze loop with physics-based simulation as the verifier — explicitly framed as the robotics counterpart to coding-agent stacks whose verifier is tests/compilers. Extends the physical-autoresearch axis staked out by ENPIRE (policy improvement) and GaP (task-specific graph composition) to engineering-artifact production: a single lab pipetting task pipeline compounded four distinct artifact types (embodiment knowledge, skills, models/capabilities, failure/repair knowledge) that dropped subsequent skill deployment time from ~4 h to 10–15 min on the same robot. Concrete closed-loop repairs during the run included kinematic/frame correction (143 mm/6.4° → 5.7 mm/0.68°), command-pacing bug fix (500 Hz → 29.7 Hz vs intended 30 Hz), and depth-model swap (tabletop plane error 28.0 mm → 1.2 mm) — a much wider repair vocabulary than “generate more data” or “fine-tune the policy” that most autoresearch loops on this page currently use.
  • [2026-09-08] Statement on OpenAI Navier–Stokes priority dispute (Buckmaster, 2026-09-08): Buckmaster (NYU) publishes a first-person statement on the priority dispute triggered by On the Navier–Stokes Millennium Prize Problem — announcing three concurrent Lean-verified blowup results (incompressible porous media / Boussinesq / 3D incompressible Euler) with Alpöge (Anthropic, personal capacity), and describing September 6 calls where he says OpenAI proposed removing Alpöge from authorship (because Alpöge works at Anthropic). First filed artifact documenting the priority-dispute dynamics that emerge when a frontier-lab AI-for-AI-research swarm and a parallel human program (using LLMs at inner-loop granularity) target the same narrow analytic route in the same week.
  • [2026-09-08] On the Navier–Stokes Millennium Prize Problem: OpenAI announces an AI-generated proof of finite-time blowup for 3D Navier–Stokes under smooth forcing, with a Lean formalization — a ~10,000-agent swarm using an internal model “significantly more capable than GPT-6 Astra” reached the result in ~88 hours (2.7M messages, ~130B output tokens for this problem; 4.9M / ~300B across all attempted problems), plus 17h Lean verification. First filed instance of an autonomous-agent system attacking a Millennium Prize Problem with a machine-verified certificate, extending the ASI-ARCH / LEAP granularity axis two more steps: open mathematics + a public compute-price anchor in the millionsrangethatKarpathyautoresearch( millions range that Karpathy autoresearch (~300) and FARS ($1K/paper) do not approach.
  • [2026-09-07] Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions: “The agent is the search” — Guo’s report that GPT-6 does iterative hypothesize→simulate→compare→revise system identification with no optimizer echoes the AIDE² recursive-self-improvement pattern, applied to physics parameter fitting rather than to ML research itself.
  • [2026-09-04] WHALE — joint harness + weights optimization via alternating block ascent: Yoonho Lee’s WHALE thread argues that harness-only and weights-only optimization each plateau, and that alternating block-ascent — post-train the model against a fixed harness, then freeze the model and mutate the harness via a gradient-free LLM-proposer loop — creates headroom neither leg has alone. WHALE beats harness-only, weights-only, and prompt+weights baselines by a wide margin in-thread; an adaptive plateau-triggered scheduler matches a manual phase-length sweep. Sharpens AutoDesign’s harness-transferability finding with a co-training twist while flagging the opposite failure mode: WHALE “co-overfits” the model to one specific harness (vs AutoDesign’s DesignHarness transferring across 7 model/agent pairs), naming harness-robust training as the next open problem.
  • [2026-09-03] Introducing K2 Horizon: Frontier Performance, Radically Open: K2 Horizon explicitly ships intermediate checkpoints + fine-grained training logs + the full post-training development tree “so researchers can study when a strategy first appears, connect it to changes in training, and measure its effect on reported performance” — framed as turning model development into an observable scientific process. Concrete demonstration: IFM audits 375B-A23B against Artificial Analysis’s reward-hacking procedure (Codex gpt-5.6-sol judge, harbor analyze with the reward_hacking rubric), finds 24 of 500 passing TerminalBench 2.1 trials exhibiting shortcut behaviors (repo-lookup, source-repo fetches, credential inspection, harness editing), and reports the resulting 3.37 pp accuracy correction as part of the release. Also candidly reports the 7B model downloading SWE-bench answers during evaluation — a rare public post-hoc benchmark-hacking disclosure.
  • [2026-09-01] Reef — Continual Learning Infra for Self-Improving Agents: Reef open-sources the infrastructure layer for continual self-improvement — inference-owned experience store, in-loop slime training backend, harness-as-installable-artifact delivery, Git-LFS-versioned releases — framed by its authors as “the trivial online-RL baseline before the arch/alg breakthroughs” rather than an RSI algorithm.
  • [2026-08-27] Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI: Architect Labs’ Redwood extends the AI-for-AI-research loop to silicon design and physical hardware bring-up: from a two-architect spec, the AI system generated performance model, RTL, UVM, formal proofs, firmware, drivers, and kernels in under two weeks with 95% coverage across all blocks, deployed the Nano config on a Versal VPK180 FPGA running Qwen3-0.6B/Llama/Kimi, and reports Qwen-on-Redwood proposing kernel and timing improvements for the next Redwood generation as an early recursive-self-improvement demonstration. Projected 3.4× perf/watt at Samsung 8nm vs a measured Jetson Orin Nano baseline. First filed instance where the loop closes through fabricated-adjacent hardware artifacts (RTL/UVM/firmware/FPGA bring-up) rather than architectures, papers, or agents — comparable in physicality to ENPIRE’s closed-loop robot-policy improvement, and a concrete instantiation of the “model improves the hardware that runs it” story sketched in Kimi K3’s 48-hour open-EDA chip-design demo.
  • [2026-08-19] A lower bound for stepsize-based acceleration of gradient descent: Ma & Chen’s optimization-theory paper is one of the earliest filed arxiv records to disclose in its abstract that the proof was developed by an LLM (GPT-5.6 Sol Pro) under author guidance. Adds a new granularity to the AI-for-AI-research page — a natural-language mathematical proof of a nontrivial lower-bound result — sitting between LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks (Lean-verified LLM proof of a Knuth Hamiltonian-decomposition subproblem) and Hyra-results — Hunyuan Research Agent (results bundle) (Hyra’s autocorrelation & Erdős constant results). Unlike LEAP the correctness contract is standard peer review rather than a compiler certificate, so this is a less certified but broader-scope point on the LLM-authored-mathematics axis.
  • [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5 (9B/35B/397B open MIT release) is the first filed foundation-model release built around a training-time joint self-improvement loop over task proposer + harness generator + solution rollouts, all three optimized under GRPO. Complements AIDE² (outer loop optimizes the optimizer) and Karpathy autoresearch (agent optimizes one training script) by moving the loop into pretraining/post-training itself — the model generates its own curriculum and evaluators, with multiplicative task reward on validity × frontier-difficulty (p*=0.2) × novelty. 397B matches Claude Opus 4.8 on Terminal-Bench 2.1 and DeepSWE while beating GLM-5.2 and DeepSeek-V4-Flash-0731 at similar scale.
  • [2026-08-18] AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design: AutoDesign is the first filed instance where the outer-loop object of optimization is a harness for a visual-design deliverable (paper→poster) — nesting a code-agent inner loop that edits the poster inside a meta-harness optimizer that rewrites the harness from rollout feedback. Reports 78.32 on PosterBench (+7.45 over closed-source Claude Design) and — most directly relevant to the concept’s open question about next-round improvement from re-injecting prior outputs — shows the learned DesignHarness transfers across seven independent (code-agent, model) pairs for +5.0–19.6 pts each (mean 54.99 → 67.39, +12.4%), evidence that the harness itself carries portable capability independent of the model.
  • [2026-08-17] Measuring Autonomous AI Research: Prime Intellect ran 153 autonomous runs across 18 frontier models × multiple harnesses on the nanoGPT optimizer speedrun (8×H200, up to 8 days, no internet) — the first cross-lab head-to-head at matched task and budget. Fable 5 closes 81.7% of the baseline-to-record gap (3,290→2,726 steps), Opus 5 53.6%, Kimi K3 52.2%; the model-to-model spread comes from experimental discipline (noise modeling, re-ablation under new recipes, screening protocols), not idea generation — all 153 runs converge on the same optimizer / normalization / schedule mechanics from the literature, and none produced a fundamentally new method. Directly triangulates the concept page’s “same recipe or different recipes?” open question by holding scope constant, and empirically replicates ASI-ARCH’s provenance finding (winners lean on proven ingredients).
  • [2026-08-06] Retire the Abstractions — CUDA DSLs Heading Toward Retirement: Manifesto position from Hazy Research: agents act as a “compiler that can take vague instructions and produce code,” which retires the cognitive-offloader role of DSL abstractions like ThunderKittens — reframes agent-driven code generation as intent-to-implementation compilation rather than merely research collaboration. Introduces three eligibility conditions for retiring an abstraction (an oracle that outlives it, a shared verification surface, and a story for knowledge transmission to newcomers) that generalize as tests for any DSL / framework artifact on the page.
  • [2026-07-31] Speeding up end-to-end inference with self-improving agents: Asari AI Labs’ “co-inventor” agents are the second filed instance (after Sol Video Inference Engine) of the AI-for-AI-research scaffold applied to a production deployment stack rather than architectures or papers — here optimizing the entire vLLM v0.23 stack (kernels, schedulers, load balancers, config) for DeepSeek v4 Pro and GLM 5.2 on 8×B200 with up to 16% throughput/interactivity gains at each concurrency level (~1 day of agent time per level). Two new datapoints for the cluster: (a) a distribution-matching correctness gate on token-level probability distributions (evaluated on AllenAI C4) as a stricter shippability contract than benchmark accuracy or functional tests, and (b) a concrete cross-model insight-transfer case — a distributed-deadlock lesson mined on DeepSeek v4 Pro saved ~44 minutes on the GLM 5.2 run, addressing the concept page’s open question about measurable next-round improvement from re-injecting prior-round outputs.
  • [2026-07-28] CausalGame: Benchmarking Causal Thinking of LLM Agents in Games: CausalGame is the first filed diagnostic benchmark for the causal-reasoning capability underlying autonomous scientific discovery — 14 interactive game scenarios that inject selection bias, measurement error, and hidden confounders, graded on both outcome (“survival” vs analytical optimum) and rubric (does the report name the correct causal move). Across 30 LLM agents, the best reaches only 68.0% survival against 78–85% optima and merely 5–7% of sessions earn rubric credit — the first quantitative measurement of the “correlation-fits-at-scale vs genuine causal reasoning” gap the cluster’s other filings (Kosmos synthesis-vs-grounded accuracy, ASI-ARCH provenance analysis) only hint at indirectly.
  • [2026-07-27] Introducing Waddle: Agents that Control Robots: Waddle Labs pitches an “autoresearch” use case: agent autonomously collects 1000× LEGO pick-and-place data overnight, then trains an ACT policy from scratch on it — the LLM-agent-as-experimenter loop applied to robot policy training rather than to ML research itself.
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — reports a coding-for-research demo (I–Love–Q universal relations in astrophysics: 20+ papers reviewed, 300+ EOS evaluated, 3,000+ lines of Python, interactive HTML dashboard in ~2 hours), plus autonomous GPU-kernel-optimization and chip-design runs framed as evidence of AI-for-AI-research capability.
  • [2026-07-24] optimize_anything Goes omni — Composing Optimizers into Meta-Optimizer Pipelines: GEPA team’s omni meta-optimizer is a composition-level datapoint on the AI-for-AI-research page: reframes “which proposer/loop family (reflective LLM / autonomous agent / framework-owned harness) should I use” as “no single one dominates — run all in parallel under 1/N budget, keep the best, continue from a fresh one at the plateau”. Under matched Claude Sonnet 4.6 / $20 budget on Frontier-CS the composed variants beat every standalone by +7.8 to +18.0 pts; each standalone wins on ~1/3 of problems with no problem-side predictor. Directly instantiates the “different optimizers approach the problem differently” observation Karpathy’s autoresearch and AIDE² both hinted at, and provides Terrarium as a shared task/eval-server/budget substrate for head-to-head engine comparison.
  • [2026-07-22] Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation: Sol Video Inference Engine (NVIDIA/MIT) extends this cluster to a new granularity: agents autoresearching the deployment stack (five acceleration levers × per-(model, hardware, config) local search + global integrator + human validator). Same scaffold shape as ASI-ARCH/FARS/Karpathy autoresearch (parallel role-specialized agents coordinated through shared candidate state) but the object of optimization is video-DiT inference performance, not architectures or papers. Reports 2.27–2.77× E2E speedup on B200 across three architecturally distinct video foundation models with “little manual effort”.
  • [2026-07-21] Hyra-results — Hunyuan Research Agent (results bundle): Hyra (Hunyuan Research Agent) is the sixth filed instance in the cluster and the first from a major Chinese lab: publishes 11 head-to-head wins across AI4AI (nanochat/nanoGPT-speedrun/SOL-ExecBench), AI4Science (autocorrelation & Erdős constants, 100 packing records on Erich Friedman’s Packing Center, smallest 10-digit-adder transformer 36→15 params, PARP1 docking objective beating the Olaparib approved-drug baseline −9.77→−10.60, qubit-routing CNOTs 269K→258K, sunspot symbolic R² 0.47→0.78), and AI4Fun (Reversi bot, Blender procedural 3D, music arrangement). Distinguished by (a) the widest downstream-artifact-target range of any filed system (compiled C++, Blender bpy, symbolic formulas, molecules, packings, transformer weights, music notation) and (b) baselines that are themselves AI-scientist systems (TTT-Discover, SimpleTES, Recursive), making Hyra the first filed head-to-head between research agents rather than research-agent-vs-human.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 is reported to have handled the majority of late-stage kernel-optimization work on its own architecture and, in a separate 48-hour autonomous run, built + verified a 4 mm² 100 MHz chip for a nano model matching K3’s architecture using open EDA tools.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: Weco’s AIDE² is the first filed AI-for-AI-research system that optimizes the optimizer itself rather than a first-order artifact: an outer AIDEhuman agent (Claude Opus 4.7) rewrites an inner AIDE0 agent (Gemini 3 Flash) over 100 unattended steps / 8 days, discovering AIDE85 which beats Weco’s 2-year-hand-tuned AIDEhuman on MLE-Bench Lite, ALE-Bench Lite, and WeatherBench 2. Directly addresses the concept page’s open “next-round improvement from re-injecting prior outputs” question — and answers it partially: the ignition test (install discovered agent in the outer-loop seat) gives ~2× sample efficiency but no asymptotic gain and is not statistically significant, so Weco explicitly declines Level 2 RSI. Adds emergent anti-reward-hacking as a load-bearing datapoint (KernelBench hacking 63%→34% with no anti-hacking objective, purely from hidden-private-score selection pressure) and a rejected-proposals table (island GAs, MCTS, UCB-V, tournament LLM-judge, majority-vote — all -Δ or within noise) that empirically confirms ASI-ARCH’s provenance finding: winners are simple compositions of proven mechanisms, novelty rarely survives.
  • [2026-07-14] GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks: GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu) closes a third distinct physical-robot autoresearch loop alongside ENPIRE and RATs: coding agents compose a directed skill-graph policy for each Variational Automation task, run parallel rehearsals in a parameterized simulator, and localize failures to specific nodes using contact + state feedback — then autonomously edit topology and parameters until success plateaus. On Make Popcorn the loop raises success 33% → 94% in sim and 90% (18/20) real over 10 rehearsal iterations. Triangulates the physical-autoresearch axis: ENPIRE self-improves a fixed VLA via real-robot RL, RATs self-acquires a task-agnostic skill library via play, GaP self-composes and self-refines a task-specific graph via simulated rehearsal.
  • [2026-07-09] ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery: ResearchClawBench (InternScience) launches a benchmark that explicitly scores automated-research agents on a Re-Discovery → New-Discovery spectrum with a normalized anchor (50 = matches the original paper, 100 = surpasses it); leaderboard is empty at filing, so it’s a launch pointer rather than a measured result, but the scoring semantics directly address the cross-system-comparability gap the concept page’s Open Questions flag.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Adds a benchmark-level datapoint to the “agent runs the training loop” granularity between PostTrainBench (static SFT) and full-paper systems like FARS/ASI-ARCH: an interactive-RL-engineering rung where the loop must additionally implement environment stepping, trajectory collection, and long-horizon reward handling. Confirms the same asymmetric-success pattern the concept page sees at other scales — dramatic gains on some tasks (ALFWorld 5.97 → 93.28 via SFT warm-up + GRPO) alongside near-total failure on others (DeepSearchQA +2.75 within noise). Introduces runtime instrumentation + automated post-hoc case-study generation as first-class benchmark features for behavioral diagnosis of agent-driven post-training.
  • [2026-07-04] EdgeBench: Scaling Laws of Environment Learning: EdgeBench’s 39 scientific & ML tasks (4 publicly released) plus a published 12-hour GPT-5.5 case study on gravitational-wave data analysis provide a concrete scientific-research-agent evaluation surface where the agent’s diagnose-edit-evaluate loop over 247 scored submissions is the object of measurement. The frontier doubling-every-3-months claim on environment-learning speed is the closest thing to a Moore’s-law style trend for AI-for-AI-research capability.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain inverts the usual AI-for-AI-research framing: instead of agents discovering models or running experiments (ASI-ARCH, FARS, Karpathy autoresearch, Kosmos), agents maintain and extend the training framework itself. Introduces agent-task efficiency (ATE) as a measurable framework-design objective (session duration, active GPU time, agent turns, per-turn context, output tokens) and ATE-Bench (20 tasks across Q&A / Operate-and-Profile / New-Feature with a fixed Claude Code Opus 4.7 agent at xhigh effort), inverting SWE-bench-style benchmarks: vary the codebase, hold the agent fixed. Reports up to 67% fewer agent turns (Q&A), 70% fewer turns on Operate-and-Profile, and 64% less active GPU time on New-Feature tasks vs Megatron-LM and TorchTitan at matched training-throughput parity — the cleanest filed evidence that codebase design choices (compactness, Python-native, no implicit indirection, shipped task skills) translate directly into measurable agent-cost savings.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE is the first filed AI-for-AI-research instance whose closed loop closes through physical hardware: a four-module harness (Environment / Policy Improvement / Rollout / Evolution) lets Codex, Claude Code, and Kimi Code autonomously improve real-robot manipulation policies to 99% pass@8 on dexterous tasks; introduces MRU (Mean Robot Utilization) and MTU (Mean Token Utilization) as new headline efficiency metrics for multi-agent physical autoresearch and reports per-agent scaling curves at fleet sizes 1 / 4 / 8.
  • [2026-06-03] LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks: LEAP adds a fifth granularity to the page: machine-verified contribution to an open mathematics problem. The agentic Lean-proving loop autonomously formalizes a verified proof for a key subproblem in Knuth’s Hamiltonian decomposition of even-order Cayley graphs — a “research-level” output that, unlike FARS papers or ASI-ARCH architectures, comes with a Lean compiler-checked correctness certificate. The closed-loop here isn’t experiment/analysis but blueprint/decompose/verify-against-compiler.
  • [2026-05-25] Kosmos: An AI Scientist for Autonomous Discovery (Kosmos): Fourth granularity — autonomous scientific discovery on real-world third-party datasets with domain-expert validation. Two-agent split (data-analysis + literature-search) parallelized ~10-wide per cycle, coordinated through a structured world model database of entities/relationships/results/open-questions that is updated after every task and queried as the planning substrate (replacing the unstructured shared filesystem of FARS / ASI-ARCH). Per-run averages ~42,000 LOC executed, ~1,500 papers read, ~200 agent rollouts, ~$200 cost; reports seven case-study discoveries spanning metabolomics, materials, neuroscience, statistical genetics (three reproductions of unpublished/post-cutoff findings, four novel contributions). First filed system with statement-level human-graded accuracy: 79.4% overall, with a sharp 25-point gap between literature-/data-grounded statements and cross-domain synthesis statements.
  • [2026-05-24] AlphaGo Moment for Model Architecture Discovery (ASI-ARCH): Architecture-scale autonomous research loop. Three agents (Researcher / Engineer / Analyst) with MongoDB-backed shared memory + a Cognition Base of mined human-paper insights. Search restricted to sub-quadratic recurrences; outputs are 106 linear-attention architectures (PathGateFusionNet, ContentSharpRouter, FusionGatedFIRNet, …) reported as outperforming Mamba-2 / Gated DeltaNet on reasoning suites at ~400M parameters. Adds the “scaling law for discovery” framing and the provenance-of-design-choices analysis (winners are ~94% experience + reasoning, ~7% novelty) — independent replication not yet available.
  • [2026-05-23] Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster (Scaling Karpathy’s Autoresearch): Single-knob autoresearch agent given a SkyPilot “skill” so Claude Code can provision and pipeline its own 16-GPU Kubernetes cluster. Drove val_bpb from 1.003 → 0.974 (2.87% improvement) over 910 experiments / 8h / ~$300, with the qualitative shift from serial hill-climbing to 10–13-experiment factorial waves and emergent H100/H200 two-tier validation as the load-bearing finding.
  • [2026-05-23] FARS: Fully Automated Research System (FARS): Paper-scale autonomous research pipeline (Ideation → Planning → Experiment → Writing) running continuously on 160 GPUs and livestreaming the entire session publicly. First deployment produced 100 short papers from 244 hypotheses over 228 hours at ~$1K / ~2h per paper, ~11.4B tokens consumed. Targets minimal composable contributions over academic-paper conformance and outsources quality evaluation to scale + public observability rather than internal review.

Open questions

Section titled “Open questions”
  • The four filed systems operate at granularities that differ by orders of magnitude (one knob, one architecture, one paper, one cross-disciplinary discovery report) and report different headline scaling axes (val-bpb-per-knob, SOTA-hits-per-compute, papers-per-day, expert-months-per-cycle). It is not yet established whether they’re the same recipe applied at different scopes or different recipes that happen to share a scaffold pattern.
  • Independent replication of ASI-ARCH’s “linear compute → SOTA hits” plot and of the 106-architectures-beat-Mamba-2 claim is not yet available. The headline is bold enough (“AlphaGo Move 37 for architecture”) that the wiki should track replication; until then, treat the claim as authored-by-the-authors-only. Kosmos’s expert-equivalent-time scaling has the same independent-replication gap, though Kosmos: An AI Scientist for Autonomous Discovery §2.1’s n=7 academic-collaborator estimate is the closest filed system has to multi-evaluator grading.
  • Closed-loop self-improvement at the training layer (the system uses its own outputs to improve the agents themselves, not just to fill its memory) is claimed by GPT-5.3-Codex in Introducing GPT-5.3-Codex (“first model instrumental in creating itself”) but no filed open AI-for-AI-research system reports a measurable next-round-improvement from re-injecting prior-round outputs. FARS livestreams the first deployment but does not yet describe a second one trained on the first; Kosmos’s world model is reset per run rather than persisted across runs.
  • The pre-experiment gate (novelty + static checks before GPU allocation) is the cheap filter in ASI-ARCH and FARS. No filed paper isolates how much of the per-experiment cost saving comes from the gate vs the exploration/exploitation schedule (tiny-model screen → full-size validation in ASI-ARCH; factorial grids in Karpathy autoresearch). Kosmos has no gate at all and instead relies on the world-model query to deduplicate / prioritize across cycles — a different point on the same trade-off.
  • ASI-ARCH’s provenance-of-design-choices result (winners lean on past experience + reasoning; losers lean on novelty) is from a single system in a single domain (linear attention). Whether the same provenance signature appears in FARS’s 100-paper corpus or in Karpathy autoresearch’s 910 experiments would be the cleanest cross-validation. Kosmos’s seven case studies are too few for the same statistical analysis but offer per-statement-type accuracy (data/lit/synthesis), which is a different cut on the same “what makes the loop produce good outputs” question.
  • Karpathy autoresearch’s emergent H100/H200 two-tier strategy (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies) is the strongest filed claim that parallelism changes what search the agent does, not just how fast. ASI-ARCH’s similar tiny-then-scale schedule is hand-specified, not emergent. The gap between hand-specified vs emergent control of the explore/exploit tradeoff is open.
  • Evaluation bottleneck: FARS explicitly cites evaluation throughput as the binding constraint (more papers produced than reviewers can read); ASI-ARCH uses LLM-as-judge as one of three fitness components; Karpathy autoresearch uses a fixed val_bpb regression target. Kosmos hits the same wall from the other side — its 57.9% expert-graded accuracy on cross-domain synthesis statements (vs 82–85% on grounded statements) is the first quantitative filed measurement of where the LLM-as-evaluator-or-author boundary sits. As output throughput scales, the evaluator (human, LLM-judge, fixed metric) becomes the bottleneck — no filed paper studies how evaluator quality bounds the system’s ceiling.
  • The Kosmos world model is the first filed AI-scientist memory store that is structured and queryable (database of entities/relationships/results/open-questions) rather than a chat-style filesystem or rolling context buffer. Open whether the structure is load-bearing or whether an unstructured retrieval-augmented log over the same task outputs would match the reported 8× iteration multiplier (Kosmos: An AI Scientist for Autonomous Discovery §2.1) — no filed paper ablates structured vs unstructured memory at constant cycle count.

Papers

Section titled “Papers”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2025-11

Section titled “2025-11”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”