ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

TL;DR

ResearchRubrics is a 101-prompt benchmark for Deep Research (DR) agents — systems like OpenAI Deep Research, Gemini Deep Research, and Perplexity Deep Research that do multi-step web exploration and long-form synthesis. Every prompt is paired with 20–43 expert-written rubric criteria (2,593 total) covering explicit/implicit requirements, synthesis, references, communication quality, and instruction following; rubrics are weighted, split into mandatory vs. optional, and include negative criteria that penalize specific failure modes. A model-as-judge then assigns ternary verdicts (Satisfied / Partially / Not) per criterion. The headline result: none of the three commercial DR systems exceeds 68% average compliance, with implicit reasoning and synthesis failures jointly accounting for 45–50% of all failures.

Key claims

Existing DR benchmarks either rely on static answer keys, narrow domains (academic/related-work writing), or LLM-generated rubrics — leaving a gap for human-written, domain-diverse, open-ended evaluation [§2, Table 1].
DR queries can be characterized along three orthogonal complexity axes: conceptual breadth, logical nesting depth, and exploration level, and existing simple-QA benchmarks fail to cover this tri-axial space [§3.2, Table 2].
All 2,593 rubric criteria are written and reviewed by three-expert pipelines (propose → review → independent final review) with no LLM-seeded rubrics, avoiding the anchoring bias that human-reviewed-but-LLM-generated rubric benchmarks suffer from [§3.1].
Rubrics are split into mandatory (weights in {±1.0}) and optional ({±0.25, ±0.5}) criteria mapped to a six-level human preference scale (Critically Important → Critically Detrimental) [§3.3, Tables 3–4].
Gemini Deep Research achieves 0.677 ternary / 0.615 binary compliance, OpenAI DR 0.664 / 0.597, Perplexity DR 0.566 / 0.487 — all below 68% under ternary grading [§4.2, Table 5].
Implicit reasoning and synthesis of information jointly account for 45–50% of all failures across systems; explicit factual retrieval and communication quality fail under 20% of the time [§4.2, Fig. 5].
Performance degrades monotonically with logical nesting depth — shallow single-/two-hop queries are handled well, deeper multi-step queries are not [§4.2, Fig. 6].
Mandatory criteria drive failures on explicit requirements and synthesis; optional criteria drive most implicit-reasoning failures — i.e. agents satisfy basic implicit context but miss nuanced quality signals [§4.2, Fig. 8].
Ternary grading (with Partially Satisfied as a middle bucket) is proposed as a better fit for rubric-based open-ended evaluation than binary met/not-met, supporting partial credit and improving human alignment [§3.4, §4.1].

Method

The benchmark is built by a three-expert pipeline: Expert 1 proposes a prompt + rubric, Expert 2 iterates with Expert 1 until approval, Expert 3 does an independent final review. Prompt seeds come from user forums, Q&A sites, and brainstorming, then adapted for realism. Each prompt is annotated with a (breadth, depth, exploration) complexity triplet and tagged into one of nine domains (AI/ML, historical analysis, technical documentation, business planning, current events, creative writing, general consumer research, etc.). Each rubric criterion is assigned a numerical weight in [-1, 1] (mandatory at ±1.0, optional at ±0.25/±0.5) and falls under one of six evaluation axes: Explicit Requirements, Implicit Requirements, Synthesis, References, Communication Quality, Instruction Following.

Evaluation is LLM-as-judge: GPT-5, Claude-Sonnet-4.5, and Gemini-2.5-Pro grade each model response against every rubric criterion with a ternary verdict {Satisfied, Partially Satisfied, Not Satisfied}; binary grading collapses Partially → Not. The final task score is a weighted sum normalized by the maximum positive weight: S = Σ w_c · v(c) / Σ max(w_c, 0), where v is the ternary indicator (1, 0.5, 0). Failure-rate decomposition averages per-task per-category failure ratios across only those tasks in which the category appears, to avoid imbalanced-distribution distortion. Human–model alignment is measured by Macro-F1 against nine expert annotators over 303 graded responses.

Results

Best system is Gemini DR at 0.677 ternary / 0.615 binary compliance, then OpenAI DR (0.664 / 0.597), then Perplexity DR (0.566 / 0.487). The 68%-ceiling pattern matches LiveResearchBench (<74% on comprehensive metrics) and DeepResearch Bench (<50%), which the authors read as evidence of architectural rather than benchmark-specific limitations. Failure rates decompose to ~45–50% from implicit reasoning + synthesis, <20% from explicit factual retrieval and communication. Mandatory-vs-optional split: mandatory failures concentrate in explicit requirements and synthesis; optional failures concentrate in implicit reasoning. Performance degrades monotonically with logical-nesting depth; the breadth and exploration axes show weaker but visible degradation.

Why it’s interesting

ResearchRubrics is the kind of evaluation harness Luma will eventually need an analog of for any agentic generation pipeline (video-research agents, multi-turn creative-direction agents). The transferable ideas are not the specific DR result — it’s that (i) rubrics-with-weights + ternary grading + mandatory/optional split is a more honest evaluation primitive than averaged scalar scores for any open-ended generative task, (ii) the breadth/depth/exploration complexity framework is a cheap way to stratify a test set so degradations are visible per-axis, and (iii) the failure-rate decomposition formula (averaging per-task category ratios only across tasks where that category appears) is a useful trick for handling imbalanced rubric distributions in any heterogeneous benchmark. The paper is also a direct foil to MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling: MiroThinker argues open-source research agents now match closed systems on BrowseComp/HLE/GAIA, but those benchmarks reward short, verifiable answers; ResearchRubrics evaluates the long-form synthesis side where even the strongest closed systems are stuck below 70%. Worth tracking whether MiroThinker-class open agents catch up here or whether the gap widens.