Context-Bench: Benchmarking LLMs on Agentic Context Engineering

TL;DR

Letta releases Context-Bench, a contamination-proof benchmark that scores LLMs on agentic context engineering — the ability of an agent to decide on its own what to load into the context window, via open_files and grep_files, over a fully fictional SQL-generated entity-relationship corpus. Questions are produced by sampling SQL queries against a synthetic database of people / pets / addresses / projects with ground-truth answers, then converting both the schema and the queries to natural-language text and questions. Headline numbers: Claude Sonnet 4.5 tops the live leaderboard at 74.0% / $24.58, GPT-5 trails at 72.67% /$ 43.56, and the best open-weight entries (GLM-4.6 at 56.83%, Kimi K2 at 55.13% / $12.08) are within ~17 points of the frontier. Even the top model misses ~25–30% of questions.

Key claims

The benchmark targets agentic context engineering — the agent (not the human) chooses what to retrieve via tools, with the explicit framing that too much context causes “context rot” and too little causes hallucination [§Introduction].
Questions are generated from a synthetic SQL database with verified ground-truth answers and fictional entities, so training-data contamination is structurally impossible [§Constructing Context-Bench].
Difficulty is controllable by varying SQL-query complexity — the benchmark is parameterized rather than fixed-size, so it can be re-tuned upward as models improve rather than saturating [§Introducing Context-Bench].
Tasks require multi-hop, multi-turn tool use: the agent must chain open_files and grep_files calls to trace relationships between entities, with no single-shot solution path [§Evaluating Agents on Context-Bench].
Claude Sonnet 4.5 leads with 74.0% at $24.58 total cost; GPT-5 reaches 72.67% but at$ 43.56 (nearly 2× the cost for −1.3 pp accuracy); GPT-5-mini hits 64.33% at $12.45 [§Models Trained for Context Engineering Excel].
Open-weight frontier: GLM-4.6 56.83%, Kimi K2 55.13% at $12.08 — Kimi K2 has the best cost-per-point ratio among open models reported [§Open-Weight Models Closing Gap to Closed-Weight Models].
“Nano” tier and some open-weight models lag substantially: GPT-4.1-nano 16.2%, DeepSeek V3 11.97%, GPT-OSS family 6.67–20.2% — context engineering is not uniform across scale or vendor [§Challenges Remain].
Total cost (not per-token price) is the production-relevant metric: a more expensive per-token model that uses fewer tokens to solve the task can be cheaper end-to-end [§Evaluating Agents on Context-Bench].

Method

A synthetic SQL database is populated with entities (people, pets, addresses, medical records, projects, collaborators) and explicit relations. An LLM question-generator (a) explores the schema and the existing question pool, (b) ideates a candidate question by writing a SQL query, (c) executes the query to obtain the ground truth, and (d) phrases the SQL as a natural-language question while serializing the database tables to semi-structured text files. The agent under test is then given exactly two tools — open_files (read full contents) and grep_files (pattern search) — and must answer the natural-language question by interleaving tool calls and reasoning. Scoring is exact-match against the SQL-derived ground truth, and total dollar cost across the benchmark run is logged alongside accuracy. The benchmark is built on the open-source Letta Evals framework and the live leaderboard is hosted at leaderboard.letta.com.

Results

Headline accuracy/cost (selected): Claude Sonnet 4.5 — 74.0% / $24.58; GPT-5 — 72.67% /$ 43.56; GPT-5-mini — 64.33% / $12.45; GLM-4.6 — 56.83%; Kimi K2 — 55.13% /$ 12.08; GPT-5-nano — 44.83%; GPT-4.1-nano — 16.2%; DeepSeek V3 — 11.97%; GPT-OSS family — 6.67–20.2% [§Models Trained for Context Engineering Excel; §Open-Weight Models Closing Gap; §Challenges Remain].
Cost dispersion at fixed accuracy is large: Kimi K2 at 55.13% / $12.08 is ≈2× cheaper than GPT-5-mini at 64.33% /$ 12.45 per accuracy point, and GPT-5 at $43.56 is the most expensive top-tier entry [§Models Trained for Context Engineering Excel].
No model exceeds 75% — the 25–30% headroom at the top is the authors’ main “remaining-difficulty” signal [§Key Takeaways].

Why it’s interesting

Context-Bench operationalizes the “agentic context engineering” framing from Memory in the Age of AI Agents — specifically the retrieval leaf of the Dynamics axis — as a measurable benchmark, distinct from how the existing tool-use benchmarks the wiki tracks (BFCL, WebWalkerQA, BrowseComp, GAIA) are scored. The closest filed neighbors are the parameter-space vs context-space split documented under Tool-Use Agents: Sonnet 4.5’s lead is consistent with the page’s claim that “models post-trained explicitly for context engineering” out-perform models that aren’t, and the open-weight gap closing (GLM-4.6, Kimi K2) is the same pattern flagged in GLM-4.5: Agentic Foundation Model from Z.ai and Kimi K2: Open Agentic Intelligence. The contamination-proof SQL-generated design also rhymes with ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence‘s “controllable difficulty via procedural generation” approach for ARC. The benchmark complements Training-Free Group Relative Policy Optimization‘s findings on context-space agent optimization — where Training-Free GRPO showed cost asymmetry of ~500× between parameter-space and context-space recipes, Context-Bench is the first benchmark that prices the inference-side half of that asymmetry directly.