Skip to content
Bud's Wiki

AI-for-AI Research

TL;DR

Section titled “TL;DR”

Multi-agent LLM scaffolds that autonomously run a closed loop of hypothesis → experiment → analysis → next hypothesis against a GPU cluster, with the system itself (not a human) deciding what to try next. The three filed instances operate at very different granularities — single training-script knob (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster), novel neural-architecture proposal (AlphaGo Moment for Model Architecture Discovery), and full short-paper production (FARS: Fully Automated Research System) — but converge on the same scaffold pattern: a small number of role-specialized agents (Researcher / Engineer / Analyst, or finer splits) coordinating through a shared memory or filesystem, with experiment results feeding back as future-round context.

Key claims

Section titled “Key claims”
  • The scaffold pattern that recurs across all three is a small set of role-specialized agents (3–4) with a shared memory or filesystem as the only coordination substrate — no direct agent-to-agent messaging, no central orchestrator state (FARS: Fully Automated Research System §Method four-agent Ideation/Planning/Experiment/Writing; AlphaGo Moment for Model Architecture Discovery §Method three-agent Researcher/Engineer/Analyst with MongoDB memory; Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Method single Claude Code agent with SkyPilot skill + the train.py file as shared state). Kosmos uses a two-agent split (data-analysis + literature-search, parallelized ~10-wide per cycle) and replaces the unstructured shared filesystem with a structured world-model database of entities/relationships/results/open-questions, updated after every task and queried as the planning substrate (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • All three systems run unbounded-throughput experiments on dedicated GPU clusters at small unit cost per experiment: FARS at ~1K/ 2hperpaperon160GPUs([[blog−analemma−fars−fully−automated−research−system−2026−02]]Results),ASI−ARCHat 20,000GPU−hours/1,773experiments/106SOTAarchitectures( 190GPU−hoursperSOTAhit)([[2507.18074−alphago−moment−model−architecture−discovery]]§EmpiricalResults),Karpathyautoresearchat 1K / ~2h per paper on 160 GPUs ([[blog-analemma-fars-fully-automated-research-system-2026-02]] Results), ASI-ARCH at ~20,000 GPU-hours / 1,773 experiments / 106 SOTA architectures (~190 GPU-hours per SOTA hit) ([[2507.18074-alphago-moment-model-architecture-discovery]] §Empirical Results), Karpathy autoresearch at ~300 GPU + 9API/910experiments/8hon16GPUs( 9 API / 910 experiments / 8h on 16 GPUs (~0.33 per experiment) (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Cost). Kosmos is the outlier on dollars/run: ~$200/run on the Edison Scientific platform (commercial pricing), with cost dominated by LLM API calls rather than self-hosted GPUs (Kosmos: An AI Scientist for Autonomous Discovery §1 reads ~1,500 papers + 42,000 LOC executed per run).
  • A pre-experiment gate that rejects proposals before any GPU time is allocated appears in the two larger-grain systems: ASI-ARCH runs vector-similarity novelty check + static-analysis on complexity / causal masking (AlphaGo Moment for Model Architecture Discovery §Method Step 2), and FARS runs automated review of hypotheses against the literature before forwarding to the Experiment agent (FARS: Fully Automated Research System §Method). The single-knob Karpathy variant skips this — there’s no novelty gate because the search space is so small. Kosmos has no analogous pre-experiment gate either; the world-model query is the post-hoc planner and accepts whatever the parallel rollouts return (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • The output is treated as a system-level product, not as conventional research artifacts: FARS explicitly publishes negative results, doesn’t conform to academic paper structure, and uses livestream + public corpus for evaluation (FARS: Fully Automated Research System §Building from First Principles); ASI-ARCH ships 106 architectures + a MongoDB database + Cognition Base under a permissive license (AlphaGo Moment for Model Architecture Discovery Repository). Kosmos diverges — it produces three to four traditional-style scientific reports per run with every statement and figure linked to either a literature citation or a specific generated Jupyter notebook, optimized for human-expert audit rather than open-corpus throughput (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • ASI-ARCH’s provenance analysis (AlphaGo Moment for Model Architecture Discovery §Analysis) reports that ~45% of design choices in winning architectures trace to lessons mined from prior experiments and ~49% to systematic reasoning, with only ~7% from novel ideas — and that losing architectures lean more on novelty. The cluster’s signature is that the loop wins by mining its own history, not by being creative.
  • Emergent strategy unprompted by the human: in Karpathy autoresearch the agent discovered H100 vs H200 hardware differences (without being told) and developed a two-tier validation strategy — screen on H100s, validate winners on H200s (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies). Adjacent claim from ASI-ARCH: gating layers + small convolutions dominate the winners and “exotic” primitives (spectral, physics-inspired) rarely make it into the top 106 (AlphaGo Moment for Model Architecture Discovery §Analysis figure) — the loop converges on a tight, proven toolkit.
  • A scaling-law-style headline appears in three of the four filings: ASI-ARCH reports a roughly linear relationship between compute spent and number of SOTA hits (AlphaGo Moment for Model Architecture Discovery §Scaling-law plot); Karpathy autoresearch reports a 9× experiment-throughput multiplier from parallel-GPU scaffolding (90/hr vs 10/hr single-GPU) and a qualitative shift from greedy hill-climbing to factorial grids (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Results); Kosmos reports that expert-equivalent research time and the count of “valuable findings” scale roughly linearly with cycle count up to 20 cycles (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1e–f). FARS does not yet report a quality-vs-compute curve.
  • New claim (correctness ceiling): When a closed-loop AI-scientist system is graded statement-by-statement by independent domain experts (the underlying code/literature withheld during evaluation), 79.4% of statements in Kosmos reports are judged accurate, with a sharp split by statement type: 85.5% for data-analysis-grounded, 82.1% for literature-grounded, and 57.9% for cross-domain synthesis statements (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1c). This is the first filed AI-for-AI-research system with such a per-statement human-graded accuracy number; the synthesis-vs-grounded gap is the load-bearing failure mode.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 is reported to have handled the majority of late-stage kernel-optimization work on its own architecture and, in a separate 48-hour autonomous run, built + verified a 4 mm² 100 MHz chip for a nano model matching K3’s architecture using open EDA tools.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: Weco’s AIDE² is the first filed AI-for-AI-research system that optimizes the optimizer itself rather than a first-order artifact: an outer AIDEhuman agent (Claude Opus 4.7) rewrites an inner AIDE0 agent (Gemini 3 Flash) over 100 unattended steps / 8 days, discovering AIDE85 which beats Weco’s 2-year-hand-tuned AIDEhuman on MLE-Bench Lite, ALE-Bench Lite, and WeatherBench 2. Directly addresses the concept page’s open “next-round improvement from re-injecting prior outputs” question — and answers it partially: the ignition test (install discovered agent in the outer-loop seat) gives ~2× sample efficiency but no asymptotic gain and is not statistically significant, so Weco explicitly declines Level 2 RSI. Adds emergent anti-reward-hacking as a load-bearing datapoint (KernelBench hacking 63%→34% with no anti-hacking objective, purely from hidden-private-score selection pressure) and a rejected-proposals table (island GAs, MCTS, UCB-V, tournament LLM-judge, majority-vote — all -Δ or within noise) that empirically confirms ASI-ARCH’s provenance finding: winners are simple compositions of proven mechanisms, novelty rarely survives.
  • [2026-07-14] GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks: GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu) closes a third distinct physical-robot autoresearch loop alongside ENPIRE and RATs: coding agents compose a directed skill-graph policy for each Variational Automation task, run parallel rehearsals in a parameterized simulator, and localize failures to specific nodes using contact + state feedback — then autonomously edit topology and parameters until success plateaus. On Make Popcorn the loop raises success 33% → 94% in sim and 90% (18/20) real over 10 rehearsal iterations. Triangulates the physical-autoresearch axis: ENPIRE self-improves a fixed VLA via real-robot RL, RATs self-acquires a task-agnostic skill library via play, GaP self-composes and self-refines a task-specific graph via simulated rehearsal.
  • [2026-07-09] ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery: ResearchClawBench (InternScience) launches a benchmark that explicitly scores automated-research agents on a Re-Discovery → New-Discovery spectrum with a normalized anchor (50 = matches the original paper, 100 = surpasses it); leaderboard is empty at filing, so it’s a launch pointer rather than a measured result, but the scoring semantics directly address the cross-system-comparability gap the concept page’s Open Questions flag.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Adds a benchmark-level datapoint to the “agent runs the training loop” granularity between PostTrainBench (static SFT) and full-paper systems like FARS/ASI-ARCH: an interactive-RL-engineering rung where the loop must additionally implement environment stepping, trajectory collection, and long-horizon reward handling. Confirms the same asymmetric-success pattern the concept page sees at other scales — dramatic gains on some tasks (ALFWorld 5.97 → 93.28 via SFT warm-up + GRPO) alongside near-total failure on others (DeepSearchQA +2.75 within noise). Introduces runtime instrumentation + automated post-hoc case-study generation as first-class benchmark features for behavioral diagnosis of agent-driven post-training.
  • [2026-07-04] EdgeBench: Scaling Laws of Environment Learning: EdgeBench’s 39 scientific & ML tasks (4 publicly released) plus a published 12-hour GPT-5.5 case study on gravitational-wave data analysis provide a concrete scientific-research-agent evaluation surface where the agent’s diagnose-edit-evaluate loop over 247 scored submissions is the object of measurement. The frontier doubling-every-3-months claim on environment-learning speed is the closest thing to a Moore’s-law style trend for AI-for-AI-research capability.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain inverts the usual AI-for-AI-research framing: instead of agents discovering models or running experiments (ASI-ARCH, FARS, Karpathy autoresearch, Kosmos), agents maintain and extend the training framework itself. Introduces agent-task efficiency (ATE) as a measurable framework-design objective (session duration, active GPU time, agent turns, per-turn context, output tokens) and ATE-Bench (20 tasks across Q&A / Operate-and-Profile / New-Feature with a fixed Claude Code Opus 4.7 agent at xhigh effort), inverting SWE-bench-style benchmarks: vary the codebase, hold the agent fixed. Reports up to 67% fewer agent turns (Q&A), 70% fewer turns on Operate-and-Profile, and 64% less active GPU time on New-Feature tasks vs Megatron-LM and TorchTitan at matched training-throughput parity — the cleanest filed evidence that codebase design choices (compactness, Python-native, no implicit indirection, shipped task skills) translate directly into measurable agent-cost savings.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE is the first filed AI-for-AI-research instance whose closed loop closes through physical hardware: a four-module harness (Environment / Policy Improvement / Rollout / Evolution) lets Codex, Claude Code, and Kimi Code autonomously improve real-robot manipulation policies to 99% pass@8 on dexterous tasks; introduces MRU (Mean Robot Utilization) and MTU (Mean Token Utilization) as new headline efficiency metrics for multi-agent physical autoresearch and reports per-agent scaling curves at fleet sizes 1 / 4 / 8.
  • [2026-06-03] LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks: LEAP adds a fifth granularity to the page: machine-verified contribution to an open mathematics problem. The agentic Lean-proving loop autonomously formalizes a verified proof for a key subproblem in Knuth’s Hamiltonian decomposition of even-order Cayley graphs — a “research-level” output that, unlike FARS papers or ASI-ARCH architectures, comes with a Lean compiler-checked correctness certificate. The closed-loop here isn’t experiment/analysis but blueprint/decompose/verify-against-compiler.
  • [2026-05-25] Kosmos: An AI Scientist for Autonomous Discovery (Kosmos): Fourth granularity — autonomous scientific discovery on real-world third-party datasets with domain-expert validation. Two-agent split (data-analysis + literature-search) parallelized ~10-wide per cycle, coordinated through a structured world model database of entities/relationships/results/open-questions that is updated after every task and queried as the planning substrate (replacing the unstructured shared filesystem of FARS / ASI-ARCH). Per-run averages ~42,000 LOC executed, ~1,500 papers read, ~200 agent rollouts, ~$200 cost; reports seven case-study discoveries spanning metabolomics, materials, neuroscience, statistical genetics (three reproductions of unpublished/post-cutoff findings, four novel contributions). First filed system with statement-level human-graded accuracy: 79.4% overall, with a sharp 25-point gap between literature-/data-grounded statements and cross-domain synthesis statements.
  • [2026-05-24] AlphaGo Moment for Model Architecture Discovery (ASI-ARCH): Architecture-scale autonomous research loop. Three agents (Researcher / Engineer / Analyst) with MongoDB-backed shared memory + a Cognition Base of mined human-paper insights. Search restricted to sub-quadratic recurrences; outputs are 106 linear-attention architectures (PathGateFusionNet, ContentSharpRouter, FusionGatedFIRNet, …) reported as outperforming Mamba-2 / Gated DeltaNet on reasoning suites at ~400M parameters. Adds the “scaling law for discovery” framing and the provenance-of-design-choices analysis (winners are ~94% experience + reasoning, ~7% novelty) — independent replication not yet available.
  • [2026-05-23] Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster (Scaling Karpathy’s Autoresearch): Single-knob autoresearch agent given a SkyPilot “skill” so Claude Code can provision and pipeline its own 16-GPU Kubernetes cluster. Drove val_bpb from 1.003 → 0.974 (2.87% improvement) over 910 experiments / 8h / ~$300, with the qualitative shift from serial hill-climbing to 10–13-experiment factorial waves and emergent H100/H200 two-tier validation as the load-bearing finding.
  • [2026-05-23] FARS: Fully Automated Research System (FARS): Paper-scale autonomous research pipeline (Ideation → Planning → Experiment → Writing) running continuously on 160 GPUs and livestreaming the entire session publicly. First deployment produced 100 short papers from 244 hypotheses over 228 hours at ~$1K / ~2h per paper, ~11.4B tokens consumed. Targets minimal composable contributions over academic-paper conformance and outsources quality evaluation to scale + public observability rather than internal review.

Open questions

Section titled “Open questions”
  • The four filed systems operate at granularities that differ by orders of magnitude (one knob, one architecture, one paper, one cross-disciplinary discovery report) and report different headline scaling axes (val-bpb-per-knob, SOTA-hits-per-compute, papers-per-day, expert-months-per-cycle). It is not yet established whether they’re the same recipe applied at different scopes or different recipes that happen to share a scaffold pattern.
  • Independent replication of ASI-ARCH’s “linear compute → SOTA hits” plot and of the 106-architectures-beat-Mamba-2 claim is not yet available. The headline is bold enough (“AlphaGo Move 37 for architecture”) that the wiki should track replication; until then, treat the claim as authored-by-the-authors-only. Kosmos’s expert-equivalent-time scaling has the same independent-replication gap, though Kosmos: An AI Scientist for Autonomous Discovery §2.1’s n=7 academic-collaborator estimate is the closest filed system has to multi-evaluator grading.
  • Closed-loop self-improvement at the training layer (the system uses its own outputs to improve the agents themselves, not just to fill its memory) is claimed by GPT-5.3-Codex in Introducing GPT-5.3-Codex (“first model instrumental in creating itself”) but no filed open AI-for-AI-research system reports a measurable next-round-improvement from re-injecting prior-round outputs. FARS livestreams the first deployment but does not yet describe a second one trained on the first; Kosmos’s world model is reset per run rather than persisted across runs.
  • The pre-experiment gate (novelty + static checks before GPU allocation) is the cheap filter in ASI-ARCH and FARS. No filed paper isolates how much of the per-experiment cost saving comes from the gate vs the exploration/exploitation schedule (tiny-model screen → full-size validation in ASI-ARCH; factorial grids in Karpathy autoresearch). Kosmos has no gate at all and instead relies on the world-model query to deduplicate / prioritize across cycles — a different point on the same trade-off.
  • ASI-ARCH’s provenance-of-design-choices result (winners lean on past experience + reasoning; losers lean on novelty) is from a single system in a single domain (linear attention). Whether the same provenance signature appears in FARS’s 100-paper corpus or in Karpathy autoresearch’s 910 experiments would be the cleanest cross-validation. Kosmos’s seven case studies are too few for the same statistical analysis but offer per-statement-type accuracy (data/lit/synthesis), which is a different cut on the same “what makes the loop produce good outputs” question.
  • Karpathy autoresearch’s emergent H100/H200 two-tier strategy (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies) is the strongest filed claim that parallelism changes what search the agent does, not just how fast. ASI-ARCH’s similar tiny-then-scale schedule is hand-specified, not emergent. The gap between hand-specified vs emergent control of the explore/exploit tradeoff is open.
  • Evaluation bottleneck: FARS explicitly cites evaluation throughput as the binding constraint (more papers produced than reviewers can read); ASI-ARCH uses LLM-as-judge as one of three fitness components; Karpathy autoresearch uses a fixed val_bpb regression target. Kosmos hits the same wall from the other side — its 57.9% expert-graded accuracy on cross-domain synthesis statements (vs 82–85% on grounded statements) is the first quantitative filed measurement of where the LLM-as-evaluator-or-author boundary sits. As output throughput scales, the evaluator (human, LLM-judge, fixed metric) becomes the bottleneck — no filed paper studies how evaluator quality bounds the system’s ceiling.
  • The Kosmos world model is the first filed AI-scientist memory store that is structured and queryable (database of entities/relationships/results/open-questions) rather than a chat-style filesystem or rolling context buffer. Open whether the structure is load-bearing or whether an unstructured retrieval-augmented log over the same task outputs would match the reported 8× iteration multiplier (Kosmos: An AI Scientist for Autonomous Discovery §2.1) — no filed paper ablates structured vs unstructured memory at constant cycle count.

Papers

Section titled “Papers”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2025-11

Section titled “2025-11”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”