Skip to content
Bud's Wiki

AI-for-AI Research

TL;DR

Section titled “TL;DR”

Multi-agent LLM scaffolds that autonomously run a closed loop of hypothesis → experiment → analysis → next hypothesis against a GPU cluster, with the system itself (not a human) deciding what to try next. The three filed instances operate at very different granularities — single training-script knob (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster), novel neural-architecture proposal (AlphaGo Moment for Model Architecture Discovery), and full short-paper production (FARS: Fully Automated Research System) — but converge on the same scaffold pattern: a small number of role-specialized agents (Researcher / Engineer / Analyst, or finer splits) coordinating through a shared memory or filesystem, with experiment results feeding back as future-round context.

Key claims

Section titled “Key claims”
  • The scaffold pattern that recurs across all three is a small set of role-specialized agents (3–4) with a shared memory or filesystem as the only coordination substrate — no direct agent-to-agent messaging, no central orchestrator state (FARS: Fully Automated Research System §Method four-agent Ideation/Planning/Experiment/Writing; AlphaGo Moment for Model Architecture Discovery §Method three-agent Researcher/Engineer/Analyst with MongoDB memory; Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Method single Claude Code agent with SkyPilot skill + the train.py file as shared state). Kosmos uses a two-agent split (data-analysis + literature-search, parallelized ~10-wide per cycle) and replaces the unstructured shared filesystem with a structured world-model database of entities/relationships/results/open-questions, updated after every task and queried as the planning substrate (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • All three systems run unbounded-throughput experiments on dedicated GPU clusters at small unit cost per experiment: FARS at ~1K/ 2hperpaperon160GPUs([[blog−analemma−fars−fully−automated−research−system−2026−02]]Results),ASI−ARCHat 20,000GPU−hours/1,773experiments/106SOTAarchitectures( 190GPU−hoursperSOTAhit)([[2507.18074−alphago−moment−model−architecture−discovery]]§EmpiricalResults),Karpathyautoresearchat 1K / ~2h per paper on 160 GPUs ([[blog-analemma-fars-fully-automated-research-system-2026-02]] Results), ASI-ARCH at ~20,000 GPU-hours / 1,773 experiments / 106 SOTA architectures (~190 GPU-hours per SOTA hit) ([[2507.18074-alphago-moment-model-architecture-discovery]] §Empirical Results), Karpathy autoresearch at ~300 GPU + 9API/910experiments/8hon16GPUs( 9 API / 910 experiments / 8h on 16 GPUs (~0.33 per experiment) (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Cost). Kosmos is the outlier on dollars/run: ~$200/run on the Edison Scientific platform (commercial pricing), with cost dominated by LLM API calls rather than self-hosted GPUs (Kosmos: An AI Scientist for Autonomous Discovery §1 reads ~1,500 papers + 42,000 LOC executed per run).
  • A pre-experiment gate that rejects proposals before any GPU time is allocated appears in the two larger-grain systems: ASI-ARCH runs vector-similarity novelty check + static-analysis on complexity / causal masking (AlphaGo Moment for Model Architecture Discovery §Method Step 2), and FARS runs automated review of hypotheses against the literature before forwarding to the Experiment agent (FARS: Fully Automated Research System §Method). The single-knob Karpathy variant skips this — there’s no novelty gate because the search space is so small. Kosmos has no analogous pre-experiment gate either; the world-model query is the post-hoc planner and accepts whatever the parallel rollouts return (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • The output is treated as a system-level product, not as conventional research artifacts: FARS explicitly publishes negative results, doesn’t conform to academic paper structure, and uses livestream + public corpus for evaluation (FARS: Fully Automated Research System §Building from First Principles); ASI-ARCH ships 106 architectures + a MongoDB database + Cognition Base under a permissive license (AlphaGo Moment for Model Architecture Discovery Repository). Kosmos diverges — it produces three to four traditional-style scientific reports per run with every statement and figure linked to either a literature citation or a specific generated Jupyter notebook, optimized for human-expert audit rather than open-corpus throughput (Kosmos: An AI Scientist for Autonomous Discovery §2.1).
  • ASI-ARCH’s provenance analysis (AlphaGo Moment for Model Architecture Discovery §Analysis) reports that ~45% of design choices in winning architectures trace to lessons mined from prior experiments and ~49% to systematic reasoning, with only ~7% from novel ideas — and that losing architectures lean more on novelty. The cluster’s signature is that the loop wins by mining its own history, not by being creative.
  • Emergent strategy unprompted by the human: in Karpathy autoresearch the agent discovered H100 vs H200 hardware differences (without being told) and developed a two-tier validation strategy — screen on H100s, validate winners on H200s (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies). Adjacent claim from ASI-ARCH: gating layers + small convolutions dominate the winners and “exotic” primitives (spectral, physics-inspired) rarely make it into the top 106 (AlphaGo Moment for Model Architecture Discovery §Analysis figure) — the loop converges on a tight, proven toolkit.
  • A scaling-law-style headline appears in three of the four filings: ASI-ARCH reports a roughly linear relationship between compute spent and number of SOTA hits (AlphaGo Moment for Model Architecture Discovery §Scaling-law plot); Karpathy autoresearch reports a 9× experiment-throughput multiplier from parallel-GPU scaffolding (90/hr vs 10/hr single-GPU) and a qualitative shift from greedy hill-climbing to factorial grids (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Results); Kosmos reports that expert-equivalent research time and the count of “valuable findings” scale roughly linearly with cycle count up to 20 cycles (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1e–f). FARS does not yet report a quality-vs-compute curve.
  • New claim (correctness ceiling): When a closed-loop AI-scientist system is graded statement-by-statement by independent domain experts (the underlying code/literature withheld during evaluation), 79.4% of statements in Kosmos reports are judged accurate, with a sharp split by statement type: 85.5% for data-analysis-grounded, 82.1% for literature-grounded, and 57.9% for cross-domain synthesis statements (Kosmos: An AI Scientist for Autonomous Discovery §2.1, Fig. 1c). This is the first filed AI-for-AI-research system with such a per-statement human-graded accuracy number; the synthesis-vs-grounded gap is the load-bearing failure mode.

Recent contributions

Section titled “Recent contributions”
  • [2026-08-27] Redwood: A Frontier AI Accelerator Designed, Verified, and Deployed from Scratch in 2 Weeks by AI: Architect Labs’ Redwood extends the AI-for-AI-research loop to silicon design and physical hardware bring-up: from a two-architect spec, the AI system generated performance model, RTL, UVM, formal proofs, firmware, drivers, and kernels in under two weeks with 95% coverage across all blocks, deployed the Nano config on a Versal VPK180 FPGA running Qwen3-0.6B/Llama/Kimi, and reports Qwen-on-Redwood proposing kernel and timing improvements for the next Redwood generation as an early recursive-self-improvement demonstration. Projected 3.4× perf/watt at Samsung 8nm vs a measured Jetson Orin Nano baseline. First filed instance where the loop closes through fabricated-adjacent hardware artifacts (RTL/UVM/firmware/FPGA bring-up) rather than architectures, papers, or agents — comparable in physicality to ENPIRE’s closed-loop robot-policy improvement, and a concrete instantiation of the “model improves the hardware that runs it” story sketched in Kimi K3’s 48-hour open-EDA chip-design demo.
  • [2026-08-19] A lower bound for stepsize-based acceleration of gradient descent: Ma & Chen’s optimization-theory paper is one of the earliest filed arxiv records to disclose in its abstract that the proof was developed by an LLM (GPT-5.6 Sol Pro) under author guidance. Adds a new granularity to the AI-for-AI-research page — a natural-language mathematical proof of a nontrivial lower-bound result — sitting between LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks (Lean-verified LLM proof of a Knuth Hamiltonian-decomposition subproblem) and Hyra-results — Hunyuan Research Agent (results bundle) (Hyra’s autocorrelation & Erdős constant results). Unlike LEAP the correctness contract is standard peer review rather than a compiler certificate, so this is a less certified but broader-scope point on the LLM-authored-mathematics axis.
  • [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5 (9B/35B/397B open MIT release) is the first filed foundation-model release built around a training-time joint self-improvement loop over task proposer + harness generator + solution rollouts, all three optimized under GRPO. Complements AIDE² (outer loop optimizes the optimizer) and Karpathy autoresearch (agent optimizes one training script) by moving the loop into pretraining/post-training itself — the model generates its own curriculum and evaluators, with multiplicative task reward on validity × frontier-difficulty (p*=0.2) × novelty. 397B matches Claude Opus 4.8 on Terminal-Bench 2.1 and DeepSWE while beating GLM-5.2 and DeepSeek-V4-Flash-0731 at similar scale.
  • [2026-08-18] AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design: AutoDesign is the first filed instance where the outer-loop object of optimization is a harness for a visual-design deliverable (paper→poster) — nesting a code-agent inner loop that edits the poster inside a meta-harness optimizer that rewrites the harness from rollout feedback. Reports 78.32 on PosterBench (+7.45 over closed-source Claude Design) and — most directly relevant to the concept’s open question about next-round improvement from re-injecting prior outputs — shows the learned DesignHarness transfers across seven independent (code-agent, model) pairs for +5.0–19.6 pts each (mean 54.99 → 67.39, +12.4%), evidence that the harness itself carries portable capability independent of the model.
  • [2026-08-17] Measuring Autonomous AI Research: Prime Intellect ran 153 autonomous runs across 18 frontier models × multiple harnesses on the nanoGPT optimizer speedrun (8×H200, up to 8 days, no internet) — the first cross-lab head-to-head at matched task and budget. Fable 5 closes 81.7% of the baseline-to-record gap (3,290→2,726 steps), Opus 5 53.6%, Kimi K3 52.2%; the model-to-model spread comes from experimental discipline (noise modeling, re-ablation under new recipes, screening protocols), not idea generation — all 153 runs converge on the same optimizer / normalization / schedule mechanics from the literature, and none produced a fundamentally new method. Directly triangulates the concept page’s “same recipe or different recipes?” open question by holding scope constant, and empirically replicates ASI-ARCH’s provenance finding (winners lean on proven ingredients).
  • [2026-08-06] Retire the Abstractions — CUDA DSLs Heading Toward Retirement: Manifesto position from Hazy Research: agents act as a “compiler that can take vague instructions and produce code,” which retires the cognitive-offloader role of DSL abstractions like ThunderKittens — reframes agent-driven code generation as intent-to-implementation compilation rather than merely research collaboration. Introduces three eligibility conditions for retiring an abstraction (an oracle that outlives it, a shared verification surface, and a story for knowledge transmission to newcomers) that generalize as tests for any DSL / framework artifact on the page.
  • [2026-07-31] Speeding up end-to-end inference with self-improving agents: Asari AI Labs’ “co-inventor” agents are the second filed instance (after Sol Video Inference Engine) of the AI-for-AI-research scaffold applied to a production deployment stack rather than architectures or papers — here optimizing the entire vLLM v0.23 stack (kernels, schedulers, load balancers, config) for DeepSeek v4 Pro and GLM 5.2 on 8×B200 with up to 16% throughput/interactivity gains at each concurrency level (~1 day of agent time per level). Two new datapoints for the cluster: (a) a distribution-matching correctness gate on token-level probability distributions (evaluated on AllenAI C4) as a stricter shippability contract than benchmark accuracy or functional tests, and (b) a concrete cross-model insight-transfer case — a distributed-deadlock lesson mined on DeepSeek v4 Pro saved ~44 minutes on the GLM 5.2 run, addressing the concept page’s open question about measurable next-round improvement from re-injecting prior-round outputs.
  • [2026-07-28] CausalGame: Benchmarking Causal Thinking of LLM Agents in Games: CausalGame is the first filed diagnostic benchmark for the causal-reasoning capability underlying autonomous scientific discovery — 14 interactive game scenarios that inject selection bias, measurement error, and hidden confounders, graded on both outcome (“survival” vs analytical optimum) and rubric (does the report name the correct causal move). Across 30 LLM agents, the best reaches only 68.0% survival against 78–85% optima and merely 5–7% of sessions earn rubric credit — the first quantitative measurement of the “correlation-fits-at-scale vs genuine causal reasoning” gap the cluster’s other filings (Kosmos synthesis-vs-grounded accuracy, ASI-ARCH provenance analysis) only hint at indirectly.
  • [2026-07-27] Introducing Waddle: Agents that Control Robots: Waddle Labs pitches an “autoresearch” use case: agent autonomously collects 1000× LEGO pick-and-place data overnight, then trains an ACT policy from scratch on it — the LLM-agent-as-experimenter loop applied to robot policy training rather than to ML research itself.
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — reports a coding-for-research demo (I–Love–Q universal relations in astrophysics: 20+ papers reviewed, 300+ EOS evaluated, 3,000+ lines of Python, interactive HTML dashboard in ~2 hours), plus autonomous GPU-kernel-optimization and chip-design runs framed as evidence of AI-for-AI-research capability.
  • [2026-07-24] optimize_anything Goes omni — Composing Optimizers into Meta-Optimizer Pipelines: GEPA team’s omni meta-optimizer is a composition-level datapoint on the AI-for-AI-research page: reframes “which proposer/loop family (reflective LLM / autonomous agent / framework-owned harness) should I use” as “no single one dominates — run all in parallel under 1/N budget, keep the best, continue from a fresh one at the plateau”. Under matched Claude Sonnet 4.6 / $20 budget on Frontier-CS the composed variants beat every standalone by +7.8 to +18.0 pts; each standalone wins on ~1/3 of problems with no problem-side predictor. Directly instantiates the “different optimizers approach the problem differently” observation Karpathy’s autoresearch and AIDE² both hinted at, and provides Terrarium as a shared task/eval-server/budget substrate for head-to-head engine comparison.
  • [2026-07-22] Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation: Sol Video Inference Engine (NVIDIA/MIT) extends this cluster to a new granularity: agents autoresearching the deployment stack (five acceleration levers × per-(model, hardware, config) local search + global integrator + human validator). Same scaffold shape as ASI-ARCH/FARS/Karpathy autoresearch (parallel role-specialized agents coordinated through shared candidate state) but the object of optimization is video-DiT inference performance, not architectures or papers. Reports 2.27–2.77× E2E speedup on B200 across three architecturally distinct video foundation models with “little manual effort”.
  • [2026-07-21] Hyra-results — Hunyuan Research Agent (results bundle): Hyra (Hunyuan Research Agent) is the sixth filed instance in the cluster and the first from a major Chinese lab: publishes 11 head-to-head wins across AI4AI (nanochat/nanoGPT-speedrun/SOL-ExecBench), AI4Science (autocorrelation & Erdős constants, 100 packing records on Erich Friedman’s Packing Center, smallest 10-digit-adder transformer 36→15 params, PARP1 docking objective beating the Olaparib approved-drug baseline −9.77→−10.60, qubit-routing CNOTs 269K→258K, sunspot symbolic R² 0.47→0.78), and AI4Fun (Reversi bot, Blender procedural 3D, music arrangement). Distinguished by (a) the widest downstream-artifact-target range of any filed system (compiled C++, Blender bpy, symbolic formulas, molecules, packings, transformer weights, music notation) and (b) baselines that are themselves AI-scientist systems (TTT-Discover, SimpleTES, Recursive), making Hyra the first filed head-to-head between research agents rather than research-agent-vs-human.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 is reported to have handled the majority of late-stage kernel-optimization work on its own architecture and, in a separate 48-hour autonomous run, built + verified a 4 mm² 100 MHz chip for a nano model matching K3’s architecture using open EDA tools.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: Weco’s AIDE² is the first filed AI-for-AI-research system that optimizes the optimizer itself rather than a first-order artifact: an outer AIDEhuman agent (Claude Opus 4.7) rewrites an inner AIDE0 agent (Gemini 3 Flash) over 100 unattended steps / 8 days, discovering AIDE85 which beats Weco’s 2-year-hand-tuned AIDEhuman on MLE-Bench Lite, ALE-Bench Lite, and WeatherBench 2. Directly addresses the concept page’s open “next-round improvement from re-injecting prior outputs” question — and answers it partially: the ignition test (install discovered agent in the outer-loop seat) gives ~2× sample efficiency but no asymptotic gain and is not statistically significant, so Weco explicitly declines Level 2 RSI. Adds emergent anti-reward-hacking as a load-bearing datapoint (KernelBench hacking 63%→34% with no anti-hacking objective, purely from hidden-private-score selection pressure) and a rejected-proposals table (island GAs, MCTS, UCB-V, tournament LLM-judge, majority-vote — all -Δ or within noise) that empirically confirms ASI-ARCH’s provenance finding: winners are simple compositions of proven mechanisms, novelty rarely survives.
  • [2026-07-14] GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks: GaP (Berkeley/NVIDIA/CMU; Goldberg, Fan, Zhu) closes a third distinct physical-robot autoresearch loop alongside ENPIRE and RATs: coding agents compose a directed skill-graph policy for each Variational Automation task, run parallel rehearsals in a parameterized simulator, and localize failures to specific nodes using contact + state feedback — then autonomously edit topology and parameters until success plateaus. On Make Popcorn the loop raises success 33% → 94% in sim and 90% (18/20) real over 10 rehearsal iterations. Triangulates the physical-autoresearch axis: ENPIRE self-improves a fixed VLA via real-robot RL, RATs self-acquires a task-agnostic skill library via play, GaP self-composes and self-refines a task-specific graph via simulated rehearsal.
  • [2026-07-09] ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery: ResearchClawBench (InternScience) launches a benchmark that explicitly scores automated-research agents on a Re-Discovery → New-Discovery spectrum with a normalized anchor (50 = matches the original paper, 100 = surpasses it); leaderboard is empty at filing, so it’s a launch pointer rather than a measured result, but the scoring semantics directly address the cross-system-comparability gap the concept page’s Open Questions flag.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Adds a benchmark-level datapoint to the “agent runs the training loop” granularity between PostTrainBench (static SFT) and full-paper systems like FARS/ASI-ARCH: an interactive-RL-engineering rung where the loop must additionally implement environment stepping, trajectory collection, and long-horizon reward handling. Confirms the same asymmetric-success pattern the concept page sees at other scales — dramatic gains on some tasks (ALFWorld 5.97 → 93.28 via SFT warm-up + GRPO) alongside near-total failure on others (DeepSearchQA +2.75 within noise). Introduces runtime instrumentation + automated post-hoc case-study generation as first-class benchmark features for behavioral diagnosis of agent-driven post-training.
  • [2026-07-04] EdgeBench: Scaling Laws of Environment Learning: EdgeBench’s 39 scientific & ML tasks (4 publicly released) plus a published 12-hour GPT-5.5 case study on gravitational-wave data analysis provide a concrete scientific-research-agent evaluation surface where the agent’s diagnose-edit-evaluate loop over 247 scored submissions is the object of measurement. The frontier doubling-every-3-months claim on environment-learning speed is the closest thing to a Moore’s-law style trend for AI-for-AI-research capability.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: PithTrain inverts the usual AI-for-AI-research framing: instead of agents discovering models or running experiments (ASI-ARCH, FARS, Karpathy autoresearch, Kosmos), agents maintain and extend the training framework itself. Introduces agent-task efficiency (ATE) as a measurable framework-design objective (session duration, active GPU time, agent turns, per-turn context, output tokens) and ATE-Bench (20 tasks across Q&A / Operate-and-Profile / New-Feature with a fixed Claude Code Opus 4.7 agent at xhigh effort), inverting SWE-bench-style benchmarks: vary the codebase, hold the agent fixed. Reports up to 67% fewer agent turns (Q&A), 70% fewer turns on Operate-and-Profile, and 64% less active GPU time on New-Feature tasks vs Megatron-LM and TorchTitan at matched training-throughput parity — the cleanest filed evidence that codebase design choices (compactness, Python-native, no implicit indirection, shipped task skills) translate directly into measurable agent-cost savings.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE is the first filed AI-for-AI-research instance whose closed loop closes through physical hardware: a four-module harness (Environment / Policy Improvement / Rollout / Evolution) lets Codex, Claude Code, and Kimi Code autonomously improve real-robot manipulation policies to 99% pass@8 on dexterous tasks; introduces MRU (Mean Robot Utilization) and MTU (Mean Token Utilization) as new headline efficiency metrics for multi-agent physical autoresearch and reports per-agent scaling curves at fleet sizes 1 / 4 / 8.
  • [2026-06-03] LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks: LEAP adds a fifth granularity to the page: machine-verified contribution to an open mathematics problem. The agentic Lean-proving loop autonomously formalizes a verified proof for a key subproblem in Knuth’s Hamiltonian decomposition of even-order Cayley graphs — a “research-level” output that, unlike FARS papers or ASI-ARCH architectures, comes with a Lean compiler-checked correctness certificate. The closed-loop here isn’t experiment/analysis but blueprint/decompose/verify-against-compiler.
  • [2026-05-25] Kosmos: An AI Scientist for Autonomous Discovery (Kosmos): Fourth granularity — autonomous scientific discovery on real-world third-party datasets with domain-expert validation. Two-agent split (data-analysis + literature-search) parallelized ~10-wide per cycle, coordinated through a structured world model database of entities/relationships/results/open-questions that is updated after every task and queried as the planning substrate (replacing the unstructured shared filesystem of FARS / ASI-ARCH). Per-run averages ~42,000 LOC executed, ~1,500 papers read, ~200 agent rollouts, ~$200 cost; reports seven case-study discoveries spanning metabolomics, materials, neuroscience, statistical genetics (three reproductions of unpublished/post-cutoff findings, four novel contributions). First filed system with statement-level human-graded accuracy: 79.4% overall, with a sharp 25-point gap between literature-/data-grounded statements and cross-domain synthesis statements.
  • [2026-05-24] AlphaGo Moment for Model Architecture Discovery (ASI-ARCH): Architecture-scale autonomous research loop. Three agents (Researcher / Engineer / Analyst) with MongoDB-backed shared memory + a Cognition Base of mined human-paper insights. Search restricted to sub-quadratic recurrences; outputs are 106 linear-attention architectures (PathGateFusionNet, ContentSharpRouter, FusionGatedFIRNet, …) reported as outperforming Mamba-2 / Gated DeltaNet on reasoning suites at ~400M parameters. Adds the “scaling law for discovery” framing and the provenance-of-design-choices analysis (winners are ~94% experience + reasoning, ~7% novelty) — independent replication not yet available.
  • [2026-05-23] Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster (Scaling Karpathy’s Autoresearch): Single-knob autoresearch agent given a SkyPilot “skill” so Claude Code can provision and pipeline its own 16-GPU Kubernetes cluster. Drove val_bpb from 1.003 → 0.974 (2.87% improvement) over 910 experiments / 8h / ~$300, with the qualitative shift from serial hill-climbing to 10–13-experiment factorial waves and emergent H100/H200 two-tier validation as the load-bearing finding.
  • [2026-05-23] FARS: Fully Automated Research System (FARS): Paper-scale autonomous research pipeline (Ideation → Planning → Experiment → Writing) running continuously on 160 GPUs and livestreaming the entire session publicly. First deployment produced 100 short papers from 244 hypotheses over 228 hours at ~$1K / ~2h per paper, ~11.4B tokens consumed. Targets minimal composable contributions over academic-paper conformance and outsources quality evaluation to scale + public observability rather than internal review.

Open questions

Section titled “Open questions”
  • The four filed systems operate at granularities that differ by orders of magnitude (one knob, one architecture, one paper, one cross-disciplinary discovery report) and report different headline scaling axes (val-bpb-per-knob, SOTA-hits-per-compute, papers-per-day, expert-months-per-cycle). It is not yet established whether they’re the same recipe applied at different scopes or different recipes that happen to share a scaffold pattern.
  • Independent replication of ASI-ARCH’s “linear compute → SOTA hits” plot and of the 106-architectures-beat-Mamba-2 claim is not yet available. The headline is bold enough (“AlphaGo Move 37 for architecture”) that the wiki should track replication; until then, treat the claim as authored-by-the-authors-only. Kosmos’s expert-equivalent-time scaling has the same independent-replication gap, though Kosmos: An AI Scientist for Autonomous Discovery §2.1’s n=7 academic-collaborator estimate is the closest filed system has to multi-evaluator grading.
  • Closed-loop self-improvement at the training layer (the system uses its own outputs to improve the agents themselves, not just to fill its memory) is claimed by GPT-5.3-Codex in Introducing GPT-5.3-Codex (“first model instrumental in creating itself”) but no filed open AI-for-AI-research system reports a measurable next-round-improvement from re-injecting prior-round outputs. FARS livestreams the first deployment but does not yet describe a second one trained on the first; Kosmos’s world model is reset per run rather than persisted across runs.
  • The pre-experiment gate (novelty + static checks before GPU allocation) is the cheap filter in ASI-ARCH and FARS. No filed paper isolates how much of the per-experiment cost saving comes from the gate vs the exploration/exploitation schedule (tiny-model screen → full-size validation in ASI-ARCH; factorial grids in Karpathy autoresearch). Kosmos has no gate at all and instead relies on the world-model query to deduplicate / prioritize across cycles — a different point on the same trade-off.
  • ASI-ARCH’s provenance-of-design-choices result (winners lean on past experience + reasoning; losers lean on novelty) is from a single system in a single domain (linear attention). Whether the same provenance signature appears in FARS’s 100-paper corpus or in Karpathy autoresearch’s 910 experiments would be the cleanest cross-validation. Kosmos’s seven case studies are too few for the same statistical analysis but offer per-statement-type accuracy (data/lit/synthesis), which is a different cut on the same “what makes the loop produce good outputs” question.
  • Karpathy autoresearch’s emergent H100/H200 two-tier strategy (Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster §Emergent research strategies) is the strongest filed claim that parallelism changes what search the agent does, not just how fast. ASI-ARCH’s similar tiny-then-scale schedule is hand-specified, not emergent. The gap between hand-specified vs emergent control of the explore/exploit tradeoff is open.
  • Evaluation bottleneck: FARS explicitly cites evaluation throughput as the binding constraint (more papers produced than reviewers can read); ASI-ARCH uses LLM-as-judge as one of three fitness components; Karpathy autoresearch uses a fixed val_bpb regression target. Kosmos hits the same wall from the other side — its 57.9% expert-graded accuracy on cross-domain synthesis statements (vs 82–85% on grounded statements) is the first quantitative filed measurement of where the LLM-as-evaluator-or-author boundary sits. As output throughput scales, the evaluator (human, LLM-judge, fixed metric) becomes the bottleneck — no filed paper studies how evaluator quality bounds the system’s ceiling.
  • The Kosmos world model is the first filed AI-scientist memory store that is structured and queryable (database of entities/relationships/results/open-questions) rather than a chat-style filesystem or rolling context buffer. Open whether the structure is load-bearing or whether an unstructured retrieval-augmented log over the same task outputs would match the reported 8× iteration multiplier (Kosmos: An AI Scientist for Autonomous Discovery §2.1) — no filed paper ablates structured vs unstructured memory at constant cycle count.

Papers

Section titled “Papers”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2025-11

Section titled “2025-11”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”