How to run SWE-bench Verified in one hour on one machine

TL;DR

Epoch AI publishes a public Docker image registry for SWE-bench, reducing the unique-layer storage from 684 GiB → 67 GiB for the full 2290-image set (10× reduction) and 189 GiB → 30 GiB for SWE-bench Verified (6× reduction). The savings come from moving shared work (git clone, heavy apt installs like matplotlib’s TeX stack) from per-instance layers down into shared env layers, exploiting that 850/2294 issues are Django and that most repo-level dependencies are constant across nearby commits. The result: SWE-bench Verified runs in 62–73 minutes on a single 32-core / 128 GB GitHub Actions VM (~8 s/sample), reported as 77× faster than the All Hands OpenHands baseline on one machine and 2.4× faster than their 32-machine parallel run.

Key claims

Common reported SWE-bench storage of ~2,000 GB is incorrect when accounting for Docker layer sharing — true unique-layer total for the unoptimized registry is 684 GiB full / 189 GiB Verified [§Impact on size].
37% of SWE-bench instances (850/2294) come from a single repo (Django), so moving git clone from the per-instance setup_repo.sh into the shared env image is a high-leverage optimization — drops a representative Django instance’s top layer from 330 MB to 40 MB, with .git history (291 MB) being the dominant cost [§Moving the git clone operation].
The matplotlib pre_install apt block (imagemagick, ffmpeg, full TeX Live) was sitting in per-instance layers despite depending only on the matplotlib version key, not the instance — relocating it to env cuts a representative top layer from 1,900 MB to 110 MB (17× reduction) [§The matplotlib 1.9 GB top layer].
Disabling pip cache via PIP_NO_CACHE_DIR=1 silently crashes pip <19 (used by older SWE-bench instances with Python 3.5 via Anaconda); the backward-compatible escape hatch is PIP_NO_CACHE_DIR=0, which works on both old and modern pip [§Disabling the pip cache].
Single-machine SWE-bench Verified wall-clock with the optimized registry: gemini-2.0-flash-001 62 min, gpt-4o-2024-11-20 70 min, claude-3-7-sonnet-20250219 63 min on one 32-core / 128 GB VM, ~8 s/sample [§Running SWE-bench Verified in about an hour].
Reported throughput is 77× faster than the OpenHands single-machine baseline (10 min/sample) and 2.4× faster than OpenHands’ 32-machine parallel configuration (~20 s/sample); hardware is not strictly comparable [§Running SWE-bench Verified in about an hour].
Coverage: 2290/2294 (99.8%) x86_64 images successfully built; full 500/500 SWE-bench Verified coverage; 1819/2294 best-effort arm64 builds untested [§Image registry usage].
Reproducibility caveat for from-source builds: SWE-bench Dockerfiles do not pin dependency versions, so images built at different times depend on apt/PyPI resolution at build time — a public pre-built registry is necessary, not just convenient, for auditable evaluations [§SWE-bench and Docker].
A residual integrity concern not addressed by the optimization: even after git reset --hard + git remote remove origin, post-PR git history may remain accessible via version tags or dangling commits reachable through git fsck --lost-found, which a sufficiently sophisticated cheating model could exploit [§Should the git history be included?].

Method

The SWE-bench Dockerfiles are built in three stages (base → env → instance) but do not exploit Docker layer caching: per-instance steps duplicate work that is either repo-constant (git clone) or version-constant (apt pre-install for matplotlib’s TeX stack). Adamczewski profiles every layer with dive (github.com/wagoodman/dive), identifies layers whose contents depend only on coarser keys than the instance, and pushes those operations down into the env stage so they are cached and shared. The two highest-impact moves are (a) doing git clone once per repo in env and letting setup_repo.sh just git reset --hard <sha> to the instance commit, and (b) moving repo-version-level apt installs (matplotlib’s pre_install) into the env stage. A PIP_NO_CACHE_DIR=0 workaround is applied uniformly to keep older pip versions (from Python 3.5 conda envs for 2015-era issues) from crashing. Build success is checked against the original SWE-bench image hashes to preserve evaluation identity. The optimized images are published under ghcr.io/epoch-research/swe-bench.eval.<arch>.<instance_id> (MIT-licensed) with x86_64 coverage at 2290/2294.

Results

Full SWE-bench (2290 images): 684 GiB → 67 GiB (10× reduction) [§Impact on size, table].
SWE-bench Verified (500 images): 189 GiB → 30 GiB (6× reduction) [§Impact on size, table].
Wall-clock for full SWE-bench Verified on a single GitHub Actions VM (32 cores, 128 GB RAM, 300K-token per-sample budget): 62 min for gemini-2.0-flash-001, 70 min for gpt-4o-2024-11-20, 63 min for claude-3-7-sonnet-20250219, with each run consuming 100–150M tokens (~2M tok/min, majority cached) [§Running SWE-bench Verified in about an hour].
~8 s/sample on one machine vs OpenHands’ reported 10 min/sample on one machine (77× speedup) and 20 s/sample on 32 machines (2.4× speedup); hardware not strictly matched [§Running SWE-bench Verified in about an hour].
Per-layer concrete examples: Django 13371 final layer 330 MB → 40 MB after relocating git clone; matplotlib 23913 final layer 1,900 MB → 110 MB after relocating pre_install apt block.

Why it’s interesting

SWE-bench is the load-bearing evaluation for Agentic Software Engineering — Qwen3-Coder-Next, IQuest-Coder-V1, GPT-5.3-Codex, Step 3.5 Flash all report SWE-bench Verified (or Pro) as their headline metric — but every filed paper on that concept page glosses over how the eval is actually run. This post makes the eval infrastructure first-class: a public 30 GiB Verified registry plus an hour-long single-machine recipe drops the operational cost of running these benchmarks by ~10× in storage and ~10×–77× in wall-clock, which is what makes RL-with-execution-feedback loops (the dominant training recipe in Agentic Software Engineering) tractable at lab scale. It also complements the env-as-package pattern from RL Environment Platforms — SETA, Toolathlon-GYM, and OpenReward all use per-task Docker sandboxes with the same “verifier-cheap, action-expensive” asymmetry, but treat the registry-size problem as out-of-scope; this post is the missing companion piece showing that registry deduplication is itself an evaluation-infrastructure design problem. The git-history side-note (sufficiently capable agents could reward-hack via git fsck --lost-found or version-tag checkouts) is a concrete leakage vector that none of the agentic-SWE training recipes filed so far address.