Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase

ListenGrade the Agent, Not the Hype: Databricks Benchmarks Coding Agents on Its Own Code10:49

Show notes

Databricks built an internal coding-agent benchmark out of its own merged pull requests against a multi-million-line codebase, then graded every mainstream model-and-harness combination with hand-audited, executable tests. The headline surprises: per-token price is a poor predictor of per-task cost, the harness can swing cost by more than two times at the same quality, and the open model GLM 5.2 landed statistically tied with Opus 4.8 at lower cost. We unpack the method, the numbers, and what it means for teams building their own evals. **Sources** - Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase (blog) — Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, Matei Zaharia

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Databricks built an internal coding-agent benchmark from their own merged PRs against a multi-million-line codebase spanning Python, Go, TypeScript, and Scala, then ran every mainstream model/harness combination through it under sealed-git-history conditions with hand-audited tests. Three capability tiers emerged, per-token cost is a poor predictor of per-task cost (Sonnet 5 is 1.7× cheaper per token than Opus 4.8 but ends up more expensive per task because it reads more), and the harness choice can shift per-task cost by >2× at the same model+quality by controlling how much context is fed per turn. GLM 5.2 lands in the top tier statistically tied with Opus 4.8 on quality at ~2/3 the cost, and Pi’s harness sends ~3× less context per turn than Claude Code / Codex at equal quality — reframing “which coding agent is best” as a joint choice over model, harness, and task complexity, all measured on an in-house corpus that no vendor has trained on.

Key claims

Model/harness combinations cluster into three distinct capability tiers on real Databricks tasks rather than forming a smooth Pareto frontier, and the intra-tier variance is smaller than the tier gap [Fig. 2]. Medium- and lower-intelligence models (Haiku, GPT-5.4 Mini class) remain highly effective on common operational tasks like flag flips and config edits at substantially lower cost, arguing against always defaulting to the most-intelligent model [§Three tiers].
GLM 5.2 lands in the top capability tier, statistically tied with Opus 4.8 on quality but at $1.28/task vs Opus's$ 1.94/task, consistent with qualitative internal-developer daily-driver feedback [§GLM]. First reported evidence in the concept lineage of a fully-open model reaching parity with Anthropic’s frontier on a proprietary benchmark.
Per-token pricing is a poor proxy for per-task cost because reasoning efficiency varies across models: Sonnet 5 is ~1.7× cheaper per token than Opus 4.8 but costs $2.09/task vs Opus's$ 1.94/task while scoring six points lower on completion (81% vs 87%), because Sonnet 5 consumed 1.9× more tokens working longer and reading more [§Token vs task cost].
Harness choice moves per-task cost by more than 2× at the same model with the same thinking effort while holding quality constant. Comparing Claude Code / Codex vs Pi, Pi sent ~3× less context per turn, kept a tighter working set, and finished in fewer runs — the harness’s context-management policy, not model choice, is the dominant cost knob at fixed capability [§Harness comparison].
Task-complexity distribution in real Databricks engineering logs is roughly a quarter low-complexity and ~60% medium-complexity, yet expensive models were the engineer default for all of it — a mismatch that motivates intelligent per-task routing and the Omnigent meta-harness [§Task-complexity mix].
Sealed git history is a necessary hardening step for benchmarks constructed from merged PRs: early runs looked “too good to be true” because agents with shell access were walking forward through git history to recover the original merged implementation. Cutting the working copy off from the repository for the duration of each run fixed inflated scores [§Sealing git history].
LLM-as-judge is explicitly rejected as the correctness evaluator on this benchmark because it “rewards sounding right over being right”; each task is graded by executing held-out patched tests written or manually rewritten by humans [§Evaluation].
Public benchmarks (SWE-Bench, TerminalBench) are insufficient to answer deployment questions like which harness/model to make the default because they don’t reflect the codebase’s task distribution, language mix, or task-complexity variance — motivating per-organization benchmarks built from merged PRs [§Why not public benchmarks].

Method

Databricks used Unity AI Gateway to capture all internal coding-agent interactions, then mined merged PRs for benchmark task candidates. PR-to-task construction pipeline: (1) filter for high-quality PRs with rich commit iteration, human review, and tests; (2) generate candidate task specifications from the PR body via scripting + AI assistance; (3) manually audit every sample — many required rewriting tests to accept alternative correct implementations (e.g. moving from exact-string-match to behavior-grading, illustrated in Fig. 3) or tightening task descriptions. Each task ships with a hidden test suite that is patched in at evaluation time.

Agent execution: harnesses and models were instantiated with their standard out-of-the-box setup and the common tool set Databricks engineers use. Each run terminates when the agent declares completion, at which point the working copy is checkpointed, held-out tests are patched in, and pass/fail is determined by test execution — never by an LLM judge. The critical hardening step from early experiments: sealing git history for the length of each run, cutting the working copy off from the repository so agents can’t walk forward through commits to recover the merged solution.

Cost accounting is done per-task (not per-token), integrating over the entire agent trajectory including all context read, tool calls, and reasoning tokens.

Results

Three capability tiers emerged in the overall model/harness scoring [Fig. 2]. GLM 5.2 and Opus 4.8 statistically tied in the top tier on quality; Opus 4.8 scored 87% task completion vs Sonnet 5’s 81%.
Cost efficiency at frontier quality: GLM 5.2 $1.28/task vs Opus 4.8$ 1.94/task at tied quality [§GLM]. Sonnet 5 $2.09/task while scoring 6 points below Opus 4.8, despite ~1.7× cheaper per-token pricing [§Token vs task cost].
Harness effect at fixed model: Same model + same thinking effort ran through Claude Code / Codex vs Pi, per-task cost differed by more than 2× in some cases with quality held constant; Pi’s per-turn context was ~3× smaller [§Harness comparison].
Task-complexity distribution on the Databricks coding-agent traffic: ~25% low-complexity, ~60% medium-complexity, with expensive models used as the default across the whole distribution — quantifying the routing-headroom opportunity that motivated the internal push toward Haiku- and GPT-5.4-Mini-class defaults for common operational tasks [§Task-complexity mix].

Why it’s interesting

This is the first datapoint in the wiki where a large industrial engineering team publishes a benchmark constructed from their own merged PRs and grades models on real work they actually did — directly addressing the “public benchmarks don’t reflect our workload” gap that FrontierSWE — benchmarking coding agents at the limits of human ability framed at the ultra-long-horizon end, and complementing PithTrain: A Compact and Agent-Native MoE Training System‘s ATE-Bench which held the agent fixed and varied the codebase (Databricks holds the codebase fixed and varies the agent). Two findings sharpen the Agentic Software Engineering concept: (a) harness choice is a first-class cost knob independent of model choice, contradicting the implicit assumption in most SWE-Bench Verified / Terminal-Bench comparisons that harness is a fixed substrate — Pi’s 3× smaller per-turn context at equal quality is the cleanest published evidence that the tool-use-agent context-management policy dominates cost at fixed capability, generalizing the “learned experiences reduce required interaction depth” observation from Training-Free Group Relative Policy Optimization to harness-level context engineering; (b) GLM 5.2 at frontier-Anthropic parity on proprietary tasks is a stronger open-model result than SWE-Bench numbers alone suggest, because the benchmark is not in any vendor’s training set and the tasks span four languages against a real multi-million-line codebase.

Also methodologically notable: the “sealed git history” gotcha (Agentic Software Engineering Open questions had flagged execution-grounded verification shortcuts abstractly; this is a concrete production instance where the shortcut was walking forward through git history) and the explicit rejection of LLM-as-judge in favor of hand-audited executable tests — both align with the Qwen3-Coder-Next Technical Report recipe of separate detector + QA agent for shortcut exploitation but at benchmark-construction time rather than training-data time.