GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

TL;DR

GDPval is an OpenAI-introduced benchmark of 1,320 real-world knowledge-work tasks (220 open-sourced as a “gold subset”) drawn from 44 occupations across the top 9 U.S. GDP sectors, constructed by industry professionals with an average of 14 years of experience. The headline metric is blinded pairwise expert preference between a model deliverable and a real industry-expert deliverable, on artifacts averaging ~7 hours of expert work and using up to 17–38 reference files spanning slides, spreadsheets, CAD, audio, video, and code. At first measurement (Claude Opus 4.1) 47.6% of model outputs won or tied a human expert’s; reasoning effort, agent scaffolding, and prompt engineering all monotonically increase win rate. An experimental automated grader (66% agreement with human experts vs 71% inter-human) is available at evals.openai.com.

Key claims

GDPval covers the top 9 sectors contributing to U.S. GDP, with 5 occupations per sector × 44 occupations total (one sector contributes only 4 occupations), at least 30 tasks per occupation (1,320 tasks full set, 220 in the gold subset) [§2, Table 1].
Occupations were selected by total wages × “predominantly digital” classification: GPT-4o classified each ONET task as digital or non-digital, and occupations were kept if ≥60% of tasks (weighted by ONET relevance/importance/frequency) were digital [§2.1].
Tasks are long-horizon: average ~7 hours of expert work, up to multiple weeks on the high end, with up to 17 reference files in the gold subset and 38 in the full set [§1, §A.4].
Primary evaluation is blinded pairwise human expert ranking; each gold-subset comparison took >1 hour to grade and experts wrote detailed justifications used to compute headline win rates [§2.5].
Claude Opus 4.1 was the top performer at first evaluation: 47.6% of its gold-subset deliverables were graded better than or as good as the human expert’s [§3.1, Fig. 5]. Per-modality split: Claude wins on .pdf / .xlsx / .pptx (aesthetics); GPT-5 wins on pure text (instruction-following, calculation accuracy) [§A.2.4, §3.1, Fig. 8].
Frontier models’ GDPval win rate is improving roughly linearly over time [Abstract].
Failure-mode clustering across GPT-5-high, Claude Opus 4.1, Gemini 2.5 Pro, Grok 4 [Fig. 8]: Claude/Grok/Gemini most often lose to instruction-following failures; GPT-5 high loses to formatting errors but has the fewest instruction-following issues; Gemini/Grok frequently promise but fail to provide a deliverable, ignore reference data, or use the wrong format; GPT-5 and Grok show the fewest accuracy errors.
Higher reasoning effort improves performance: o3 and GPT-5 sweeps at low/medium/high reasoning all show monotone improvement [§3.4].
A general (not task-specific) prompt that pushes GPT-5 to render files as images, avoid nonstandard Unicode, and check layouts eliminates black-square artifacts from PDFs (previously affecting >50%), reduces egregious PowerPoint formatting errors from 86% to 64%, and improves human-preference win rate by 5 percentage points [§3.4, Fig. 9(b)]. The fraction of agent turns using multimodal capabilities to inspect deliverables jumps 15% → 97% [§3.4].
Best-of-N=4 with a GPT-5 judge plus enabling GET in the agent container further improves win rate beyond reasoning + prompting [§3.4].
Frontier models paired with human oversight can complete GDPval tasks cheaper and faster than unaided experts under a “try the model first, fall back to humans if unsatisfactory” workflow [§3.2, Fig. 7]; the paper does not have cost numbers for Claude, Gemini, or Grok.
The automated grader (a model trained to mimic expert pairwise comparisons) reaches 66% agreement with human expert graders vs 71% human inter-rater agreement on the gold subset [§2.5, §A.6].
“Under-contextualized” GDPval (less context provided up front) degrades model performance, isolating the assumption that real-world deployment requires the agent to gather its own context [§A.2.7].

Method

GDPval is constructed top-down. Sectors are ranked by share of U.S. GDP using Q2 2024 BEA Value Added by Industry data; the 9 sectors contributing >5% are kept. Inside each sector, the 5 occupations with highest total compensation that pass a “predominantly digital” filter (GPT-4o-rated tasks from ONET, weighted by relevance × importance × frequency, threshold 60%) are kept, validated for representativeness against the Acemoglu–Autor non-routine-cognitive task content framework [§2.1, §A.7.1]. Industry-expert contributors (average 14 years of experience, plus video interview, background check, training, and quiz) write tasks based on actual work product from their occupations, classifying each task against ONET Work Activities so coverage is auditable. Each task ships with a request (prompt + reference files, up to 38 files in the full set) and a reference deliverable (the expert’s work product). Each task receives an average of 5 human reviews before inclusion.

Evaluation is blinded pairwise expert ranking — two anonymized deliverables (one model, one human expert; or two models) are ranked by an additional occupational expert who writes a justification. Each gold-subset prompt × model pair is sampled 3 times and graded by 3 human experts → 9 comparisons per model per task across 220 tasks. To enable scalable downstream research, an experimental automated grader (a model trained on expert pairwise justifications) is released at evals.openai.com; it reaches 66% agreement with humans vs the 71% human inter-rater agreement.

Models tested at first evaluation: GPT-4o, o4-mini, o3, GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, Grok 4. OpenAI models were sampled with the web-search tool and code interpreter enabled; Claude was sampled through its UI to enable “Upgraded file creation and analysis”; agent harness scaffolds were configured per model to be roughly comparable.

Results

Headline win-or-tie rate on the gold subset: Claude Opus 4.1 at 47.6% [§3.1, Fig. 5] — i.e., over half of human-expert deliverables were preferred, but Claude was the closest. GPT-5 sits below Claude on aggregate but wins on text-only tasks. (Note: a follow-up amaarora.github.io review of GPT-5.2 reports 70.9% win rate; this is not in the original arxiv paper but is consistent with the “roughly linear improvement over time” claim.)
Frontier-model win rates “are approaching industry experts in deliverable quality” [Abstract].
Reasoning effort: o3 and GPT-5 at low → medium → high reasoning each step monotonically improve [§3.4]. Quantitative deltas are reported in §A.2 but not in the truncated body.
Prompting fix on GPT-5 [§3.4, Fig. 9(b)]: PDF black-square artifacts eliminated (was >50% of generated PDFs), egregious PowerPoint formatting errors reduced 86% → 64%, +5 percentage points on human win rate, multimodal-self-inspection rate 15% → 97%.
Speed/cost (under “try-then-fix” workflow) [§3.2, Fig. 7]: expert time and money saved relative to unaided experts; absolute numbers reported only for OpenAI models.
Failure-mode clustering [Fig. 8]: instruction-following failures dominate for Claude/Grok/Gemini, formatting errors dominate for GPT-5, all models occasionally hallucinate or miscalculate.
The 220-task gold subset, all reference files, and the automated grader are publicly available at evals.openai.com.

Why it’s interesting

GDPval is the missing benchmark in the Agentic Software Engineering “four-benchmark quadruple” (SWE-Bench Pro / Terminal-Bench 2.0 / OSWorld / GDPval) that Introducing GPT-5.3-Codex framed as the new evaluation surface — but unlike the other three it is not coding-shaped, which is exactly why no open model has reported it yet. The win-rate-vs-human-expert design is structurally identical to LMArena-style pairwise judging applied to deliverables instead of chat turns; it shares the methodology of RISE-Video: Can Video Generators Decode Implicit World Rules? and IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation under the VLM-as-Evaluator umbrella, but with human experts as judges (and an experimental VLM judge available as a cheaper proxy). The most consequential empirical finding for Luma is the prompting-and-scaffolding result in §3.4: a general, non-task-specific prompt eliminated GPT-5’s PDF black-square artifacts and roughly halved PowerPoint formatting errors, and best-of-N=4 with a GPT-5 judge stacks on top — concrete evidence that Inference-Time Scaling still buys substantial real-world-task quality at the current frontier, in line with the broader pattern documented across Reasoning with Sampling: Your Base Model is Smarter Than You Think and pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation but on a benchmark where the deliverable is graded by an industry expert rather than a metric.