Skip to content
Bud's Wiki

Epoch AI launches benchmark reviews — rubric-based Flawed / Verified / Not Enough Info verdicts for 15 benchmarks

TL;DR

Section titled “TL;DR”

Epoch AI is launching a benchmark-reviews service: each reviewed benchmark gets a Flawed, Verified, or Not Enough Info verdict against a stated rubric, alongside a full-length review documenting weaknesses and limitations. The launch batch is 15 benchmarks, with an explicit conflict-of-interest carve-out — Epoch will not review its own benchmarks — and an “ongoing basis” cadence targeting the highest-impact and most-cited public evals. Reviews are pinned to a specific benchmark version and re-runnable when the benchmark is updated to fix errors; developers get advance findings and their published response is linked.

Key claims

Section titled “Key claims”
  • Verdicts come from a rubric with three explicit levels — Flawed, Verified, Not Enough Info — with Verified meaning “can broadly be interpreted as described, errors do not substantially affect results” [tweet §2, §3].
  • Not Enough Info is reserved for benchmarks Epoch cannot access enough of to review; Epoch commits to working with private-benchmark authors to conduct reviews while keeping tasks outside public knowledge [tweet §5].
  • Reviews are versioned against a specific benchmark release — updates get re-reviews, and developer responses are published as linked companions [tweet §6].
  • Epoch will not review Epoch-created benchmarks and welcomes external reviews of those [tweet §2].
  • Launch batch is 15 benchmarks with ongoing additions prioritized by impact/reach and by high-quality-but-overlooked candidates [tweet §7].

Method

Section titled “Method”

The tweet describes the review scheme rather than a specific benchmark study. Structurally: (a) a public rubric defines the three verdicts; (b) reviews consist of a full-text assessment plus the verdict; (c) each review binds to a benchmark version identifier; (d) findings are shared with developers before publication, and developer responses are linked from the review. Related Epoch material at the time of the tweet is consistent with this shape — the SWE-bench Verified review documents an 11-percentage-point GPT-5 delta from scaffold choice and a 15-point delta for Kimi K2 Thinking on the same underlying benchmark, and Epoch’s gradient-updates/why-benchmarking-is-hard explicitly attributes scaffold choice as “the single biggest impact on overall performance.” No embedded URLs were resolvable from the tweet at filing time, so the specific 15 benchmarks in the launch batch are not enumerated here.

Results

Section titled “Results”

Quantitative launch numbers stated in the tweet: 15 benchmarks in the initial batch. No verdict distribution (how many Flawed vs Verified vs Not Enough Info) is stated in the tweet text; that lives in the individual reviews linked from the launch thread. External context (Epoch’s own SWE-bench Verified review) reports 5–10% flawed-instance rate on that specific benchmark, well below the ~1/3 fatal-error rate Epoch’s May 2026 AI-assisted FrontierMath audit flagged on Tiers 1–4 — an internal benchmark that would fall under the conflict-of-interest carve-out for this new review track.

Why it’s interesting

Section titled “Why it’s interesting”

This is a bet on standardized meta-evaluation as a public good. The wiki already tracks the judge-side fragility of evals (VLM-as-Evaluator — where a benchmark is only as good as the VLM judge or rubric behind it, cf. IGenBench’s finding that Gemini-2.5-Pro is the only public MLLM judge clearing r=0.8 with humans) and the scaffold-side fragility (Agentic Software Engineering — SWE-Bench Verified numbers moving 11–15 points from a scaffold swap, per Epoch’s own review). Epoch’s launch operationalizes the natural next layer: a persistent, versioned, third-party audit of the benchmarks themselves, with an explicit no-self-review rule that concedes the FrontierMath-style conflict-of-interest problem. Complements Sophon — AI Research Catalog of Evals, Tools, and Labs (which catalogs saturation and evaluator lift on a per-benchmark basis) and Implications of Large-Scale Test-Time Compute (Noam Brown) (which argues that benchmark scores are ill-defined without disclosed inference budgets — an axis the review rubric will need to cover). Also sits alongside ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation as a methodology for improving benchmark reliability rather than a tool for it.

See also

Section titled “See also”