AlphaGo Moment for Model Architecture Discovery

TL;DR

ASI-ARCH is a fully autonomous, three-agent LLM research loop (Researcher → Engineer → Analyst) that proposes, implements, trains, and analyzes novel neural architectures end-to-end, with a shared memory of prior experiments feeding back into the next proposal. Over 1,773 experiments and ~20,000 GPU-hours, the system discovered 106 linear-attention architectures (e.g. PathGateFusionNet, ContentSharpRouter, FusionGatedFIRNet) reported as state-of-the-art against human-designed baselines like Mamba-2 and Gated DeltaNet on reasoning suites at ~400M-parameter scale. The headline claim is empirical: a roughly linear relationship between compute spent and number of SOTA hits, framed as the first scaling law for architecture discovery itself — turning research progress from a human-bandwidth problem into a compute-bound one.

Key claims

A three-role agent stack (Researcher proposes code + motivation, Engineer trains and self-fixes failures, Analyst extracts lessons and updates a shared memory) runs an end-to-end autonomous architecture-discovery loop with no human in the inner loop [§Method, “ASI-Arch framework” figure].
1,773 experiments over ~20,000 GPU-hours produced 106 architectures the paper reports as outperforming Mamba-2 and Gated DeltaNet on reasoning benchmarks at ~400M parameters [§Empirical Results].
The number of SOTA hits scales approximately linearly with compute spent (the “scaling law for discovery” framing) [§Scaling-law plot].
Pre-training novelty + complexity gates filter proposals before any GPU is allocated: a similarity search against prior experiments rejects reruns, and automated static checks enforce sub-quadratic complexity + correct causal masking; failures bounce back to the Researcher for rewrite [§Method, Step 2].
A composite fitness score combines (a) sigmoid-scaled loss improvement, (b) benchmark gain, and (c) LLM-as-judge quality rating into a single survival metric per experiment [§Method, Step 4].
A two-stage explore/exploit schedule trains thousands of tiny models for coverage, then scales only the most promising designs to full size for validation — the GPU-budget allocator inside the loop [§Method, Step 6].
Component-usage analysis across 106 winning vs 1,667 discarded architectures: gating layers (~14%) and small convolutions (~12%) dominate both sets; exotic primitives (physics-inspired, spectral) are rare among winners [§Analysis figure].
Source-of-contribution analysis: ~45% of design choices in winning architectures trace to lessons mined from prior experiments, ~49% to systematic reasoning, only ~7% to novel ideas; less successful models lean more on novelty, suggesting cumulative experience-mining (not creativity) drives the empirical scaling [§Analysis table].
The authors compare the AI-found designs to AlphaGo’s Move 37 — i.e. the claim is that the loop reaches regions of architecture space humans had not explored, not just that it accelerates existing search [§Discussion].
All 106 architectures, the agent framework, the MongoDB-backed Architecture Database, and the Cognition Base (paper-insight knowledge store) are open-sourced at github.com/GAIR-NLP/ASI-Arch [Repository README].

Method

The system is composed of three LLM-backed agent roles sharing a single MongoDB-backed memory.

The Researcher agent reads (a) the running memory of prior experiments and (b) a “Cognition Base” of mined human-paper insights, then writes both a natural-language motivation and a working PyTorch implementation of a candidate architecture. Before training, the proposal passes a similarity-novelty check (vector retrieval against the experiment memory) and a static-analysis pass that verifies sub-quadratic complexity and proper causal masking; failures are returned to the Researcher for rewrite.

The Engineer agent submits the model to the training environment, watches for crashes, OOMs, or stalls, reads the error logs, patches its own code, and retries until the run completes. Training is bucketed into a tiny-model exploration phase (cheap broad search) and a scaled-up validation phase (full-size confirmation for top candidates).

The Analyst agent reads the training logs and benchmark numbers, writes a short structured report on what helped and what hurt, and commits both the raw metrics and the synthesized insight back to the shared memory. The next Researcher round retrieves both, closing the loop. A composite fitness score (sigmoid-loss + benchmark + LLM-judge) decides which experiments are promoted into the “winning” set used for downstream pattern-mining.

The framework is instantiated for linear-attention discovery specifically: the search space is restricted to sub-quadratic recurrences (linear attention, gated recurrences, state-space variants), and the target benchmarks are reasoning suites where Mamba-2 / Gated DeltaNet are the public baselines.

Results

1,773 experiments × ~20,000 GPU-hours → 106 architectures reported as outperforming Mamba-2 / Gated DeltaNet on reasoning benchmarks at ~400M parameters [§Empirical Results].
Roughly linear scaling between compute spent and number of SOTA-passing designs found (the “scaling law for discovery” plot) — the headline framing.
Component frequency in winners is dominated by gating + small convolutions (~14% / ~12%); residual links and feature pooling follow. Exotic spectral / physics primitives are rare in winners but appear more in the discarded pool.
Provenance of winning-architecture design choices: ~45% experience-driven (from memory of prior runs), ~49% systematic reasoning, ~7% novel — and losers actually lean more on novelty.
Named example architectures (PathGateFusionNet, ContentSharpRouter, FusionGatedFIRNet) ship in the public repo with full PyTorch implementations.
No quantitative reproduction by an independent third party at filing time; both the result and the “Move 37” framing are the authors’ own.

Why it’s interesting

This is the wiki’s first datapoint where the artifact being discovered is a model architecture, putting it at an intermediate granularity between FARS: Fully Automated Research System (FARS produces full short papers) and Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster (single-train.py-knob optimization). The three-agent role split (Researcher / Engineer / Analyst with a shared memory as the only coordination substrate) is structurally close to FARS’s four-agent pipeline, suggesting a converging scaffold pattern for AI-for-AI research. The provenance-of-design-choices result (winners are 94% experience + reasoning, only 7% novelty; losers lean more on novelty) is the load-bearing finding: the loop wins by mining its own history, not by being creative — same closed-loop-synthesis pattern as Agentic Software Engineering (model → synthesizer → next model), just one level up at the level of architecture proposals rather than code patches. Luma’s research note “Either LK-99 or huge” is the right calibration: the “linear compute → SOTA hits” framing is bold enough that the wiki should track independent replication, and the “106 SOTA architectures” claim depends on (a) Mamba-2 / Gated DeltaNet being the reference SOTA (not Mamba-3: Improved Sequence Modeling using State Space Principles) and (b) the ~400M scale generalizing to the range the rest of the wiki cares about.