EdgeBench: Scaling Laws of Environment Learning

TL;DR

EdgeBench (ByteDance) is a 134-task ultra-long-horizon benchmark built to measure not what an agent already knows but how much it learns when given 12+ hours of continuous interaction with a real-world environment (some tasks extend past 72 h). Task families span six areas — science/ML, systems & SE, optimization, knowledge, formal math, games — with 51 tasks and the full evaluation framework released for public use and the remainder held for the leaderboard. Averaging 402 learning curves per model across all tasks in interaction time collapses noisy per-task trajectories into a clean log-sigmoid law (S(t) = S_{\max}/(1 + (t_{\mathrm{mid}}/t)^{\beta})) with mean (R^2 = 0.998), which the authors derive from a mean-field frontier-percolation argument on a self-similar task graph. Applying the fit to frontier releases from Sep 2025 to May 2026 gives the paper’s headline empirical claim: AI learning speed from environments roughly doubles every three months.

Key claims

EdgeBench is the first benchmark explicitly designed to score environment-learning (long-horizon feedback loops) rather than static-knowledge recall, with 134 tasks running ≥12 h and 320 h of expert-verified curation effort per task at the largest end [§Benchmark].
The 51/134 public split covers all six task families (4/39 scientific & ML, 13/36 systems & SE, 14/19 optimization, 4/19 knowledge, 8/13 formal, 8/8 games), with the remaining 83 tasks reserved for the private leaderboard accessed via email submission [§Benchmark].
Averaging 402 per-model learning curves point-by-point in interaction time collapses jagged per-task trajectories into a log-sigmoid (S(t) = S_{\max}/(1 + (t_{\mathrm{mid}}/t)^{\beta})) with mean (R^2 = 0.998) across 38,000 hours of recorded interaction [§38,000 hours of noisy curves].
The log-sigmoid law is derived from a frontier-percolation model on a self-similar task graph: each task is a graph of small unlockable units, learning spreads from unlocked to locked neighbors, so the frontier speed is proportional to (x(1-x)); self-similarity forces the correct time axis to be (u = \ln t), giving (\frac{dx}{d\ln t} = \beta x(1-x)) which solves to the fitted form [§A theory of the log-sigmoid law].
Isolating environment-learning from prior knowledge on 18 tasks with matched initial performance and 2-hour evaluation windows across Sep 2025 – May 2026 model releases produces the headline scaling claim: learning speed from environments roughly doubles every three months [§AI learns from environments roughly twice as fast].
A single 12-hour trace of GPT-5.5 on a gravitational-wave analysis task climbs 42.8 → 67.0 across 247 scored attempts with only 27 improving submissions, revealing a sparse diagnose-edit-evaluate loop with seven qualitative turning points (make-measurable → decompose → isolate-bottleneck → keep-core-repair-residual) [§Inside a single 12-hour run].

Method

Each of the 134 tasks is authored and reviewed by domain experts (mean 57.2 h of expert effort per task, up to 320 h) and packaged with a workspace, feedback signals, and an automated judge that approximates a real-world review loop. Agents run for 12–72 h against the task and can submit repeatedly; every submission is scored, producing a dense per-task learning curve of best-so-far score over interaction time. Across 402 curves per model, the authors average point-by-point in interaction time to obtain a smooth trend, then fit the log-sigmoid (S(t) = S_{\max}/(1 + (t_{\mathrm{mid}}/t)^{\beta})) with three per-model parameters (asymptote (S_{\max}), midpoint (t_{\mathrm{mid}}), sharpness (\beta)).

The theoretical companion models a task as a graph of small “unlockable units”, assumes learning spreads from unlocked to adjacent locked units, and shows via a mean-field argument that the normalized progress (x(t) = S(t)/S_{\max}) obeys (\frac{dx}{d\ln t} = \beta x(1-x)) — a logistic in (\ln t), which is exactly the log-sigmoid. Self-similarity of the task graph forces the logarithmic time axis. The population-level fit emerges as an averaging effect over many finite jagged individual-task graphs. To disentangle environment-learning from prior-knowledge, the authors pick 18 tasks where models start from similar initial performance and use the 2-hour improvement inside that window as a learning-speed metric — the same metric applied across Sep 2025 – May 2026 model releases produces the doubling-every-three-months frontier claim.

Results

Log-sigmoid fits reach mean (R^2 = 0.998) across 38,000 recorded interaction-hours and 402 per-model curves [§38,000 hours of noisy curves].
Public release covers 51/134 tasks and the full evaluation framework; the remaining 83 tasks are held for the leaderboard [§Benchmark].
Frontier learning-speed metric doubles every ~3 months across Sep 2025 – May 2026 model releases evaluated on the matched-initial-performance 18-task subset [§AI learns from environments roughly twice as fast].
The GPT-5.5 gravitational-wave case study: 247 scored attempts over 12 h, 27 improve best-so-far by ≥0.1 pp, best-so-far climbs 42.8 → 67.0 with the largest jump appearing at the source-mass calibration step after component-level feedback isolated velocity/separation as the dominant gap [§Inside a single 12-hour run].
Full 134-task leaderboard is not publicly disclosed at filing time; evaluation on the full set requires contacting zhongshu@bytedance.com [§Leaderboard].

Why it’s interesting

EdgeBench is the first filed benchmark whose scoring axis is learning speed, not final capability — which turns the RL Environment Platforms design pattern (verifier-cheap, action-expensive, Docker-packaged tasks) into a measurement pattern rather than a training pattern. Where SETA: Scaling Environments for Terminal Agents (env repo), Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents, and OpenReward — 330+ RL Environments through one API (General Reasoning announcement) use env-packages as GRPO training substrates and RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots uses them as embodied-policy eval, EdgeBench uses them to sample learning-curve dynamics at 12–72 h horizons. This is the direct evaluation-side counterpart to MiroThinker’s MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling interaction-depth scaling claim (Inference-Time Scaling axis (b)): where MiroThinker shows monotone gains up to several hundred tool calls, EdgeBench actually fits a functional form to the gain curve and turns “more interaction = better” into a testable scaling law with clean (R^2). The theoretical frontier-percolation derivation is a rare instance of a bottom-up scaling-law justification, complementing empirical curves like How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size on the training-side and Scaling Laws, Carefully‘s general treatment. The three-month doubling claim, if it holds under third-party replication, is the first frontier-progress metric anchored on environment-learning capability rather than static leaderboard scores or scaling-law extrapolations from pretraining loss.