ArxivQA: Training Retrieval Agents for arXiv Search

TL;DR

alphaXiv post-trains Qwen3-8B as a multi-hop retrieval agent over the arXiv corpus (~150K AI papers) using a deep-research–style ReAct loop with keyword_search / semantic_search / read_page / finish tools. They tried straight GRPO with a verifiable recall reward (RLVR) on ~200 hand-labelled real-world queries from the alphaXiv assistant, found it collapsed to single-shot single-tool-call rollouts at step 100, then switched to Rubrics-as-Rewards (RaR): an LLM judge scores the entire trajectory across criteria (search diversity, thoroughness, thoughtfulness, document relevance). The RaR-trained 8B agent outperforms every frontier model on judge-graded eval except Opus 4.6, and matches GPT-5.4 on recall. The headline lesson: for open-ended retrieval queries, RLVR’s recall reward is too noisy to teach intelligent search behavior, and rubric rewards give much more direct control over rollout shape with far less reward engineering.

Key claims

~200 hand-labelled real-world queries are sufficient training data for a multi-hop search agent when the reward signal is low-noise and precisely aligned with the desired skill [§Dataset & Environment].
Pure RLVR (GRPO + an over-engineered recall reward with duplication / submission / citation / limit gates plus a redundant-query penalty and a multi-turn bonus) plateaus around step 100 and collapses to a single semantic_search + single keyword_search call per rollout; entropy collapses by step 200 [§RLVR].
RLVR rollouts under this reward exhibit four named failure modes: lack of systematic exploration/refinement, superfluous reasoning grounded in latent knowledge instead of search results, zero reasoning before turns 2–3 tool calls, and search queries that are mere rephrasings of the user query [§RLVR].
Curriculum learning (easy/medium/hard splits) does not rescue RLVR on open-ended queries: on the medium split the agent still hill-climbs by repeating the original query, because the recall reward gives lucky-keyword-overlap rollouts the same score as well-reasoned ones [§Curriculum Learning].
Rubrics-as-Rewards (RaR-Implicit) with a single hand-crafted rubric — judge scores 0-5 on criteria (search diversity, thoroughness, thoughtfulness, document relevance) then a holistic 1-10 — produces “more thorough search patterns” across virtually every judge configuration [§Rubrics, §Results].
Final reward formulation under RaR uses the gates from RLVR plus a stepped length penalty against trajectory token count and the judge’s holistic score linearly mapped to [0, 1] [§Rubrics setup].
The RaR-trained Qwen3-8B (judged by Gemini 3 Flash during training) outranks every frontier model on average judge score across Opus 4.6 / Gemini 3.1 Flash / GPT 5.2 except Opus 4.6 itself, and scores just above GPT-5.4 on recall on a held-out test set of mostly medium/hard queries [§Results].
Choice of training judge is not strongly correlated with downstream recall (Olmo-32B and Gemini 3 Flash give nearly identical results) — but eval-judge scores diverge, suggesting performance on judge-graded eval is driven more by alignment between training and evaluation judges than by the underlying capability of the training judge [§Results, judge ablation].
The deep-research labeller — Claude Sonnet 4.5 + Gemini 3 Flash with parallel-search subagents reading entire candidate documents, ~3 min/query — still required manual re-labelling on almost every query, because open-ended research queries don’t have unambiguous ground truth [§Dataset & Environment].
Recall is chosen as the primary metric because in production the retriever feeds a downstream synthesis LLM, where missing a critical document is more damaging than including unnecessary ones [§Training].
Reported as a generalizable thesis: the optimal RL strategy is dataset-dependent — RLVR works on factoid multi-hop benchmarks (HotpotQA) where labels carry strong signal, but real-world open-ended retrieval benefits from rubric rewards that grade the process rather than the outcome [§Results conclusion].

Method

The agent is post-trained from Qwen3-8B using SkyRL-Agent (GRPO with parallel tool calls and document-ID obfuscation in the orchestrator to prevent ID memorization). Tools: keyword_search and semantic_search (return arXiv ID + publication date + upvotes + abstract or matching snippets), read_page (full page text by ID), and finish (submits up to 10 final IDs). The corpus + search backend is a sandbox API serving ~150K recent AI papers with page text + abstract embeddings.

Training data is ~200 queries sampled from over 1M production alphaXiv assistant queries, each labelled with up to 10 ground-truth arXiv IDs. Labels are bootstrapped by a deep-research agent (parallel subagents reading candidate documents end-to-end over ~3 min per query) then almost always manually re-curated.

Two training regimes are compared. RLVR uses a recall reward R_search (fraction of ground-truth IDs appearing across all tool-call results, not just the final answer) minus a Jaccard-similarity redundancy penalty across emitted queries, plus a multi-turn bonus, all gated by duplication / submission / citation / limit gates. RaR-Implicit replaces the outcome reward with an LLM-judge score over the full trajectory using a single hand-crafted rubric (the judge first emits per-criterion 0-5 scores, then a holistic 1-10 score), linearly mapped to [0, 1] and combined with a stepped trajectory-length penalty. The same gates carry over. Hyperparameters: LR 1e-6, batch size 8, 4 update epochs per batch, 250 epochs, KL coefficient 0.02, PPO clip [0.2, 0.28], 8 samples per prompt (the post says rich rubric reward let them use 8 rollouts rather than the 16 RLVR needed).

Judge ablations during training: Olmo-32B, Gemini 3 Flash, GLM-5, Claude Sonnet 4.5, GPT 5.2. Eval judges: Opus 4.6, Gemini 3.1 Flash, GPT 5.2, and GPT 4.1 Mini as a small-model judge probe.

Results

The RaR-trained Qwen3-8B (Gemini-3-Flash-judged) scores above every frontier baseline on average eval-judge score except Claude Opus 4.6 — and just above GPT-5.4 on recall. Frontier models on the held-out test set top out around 0.58 recall (Opus 4.6), illustrating how hard real-world deep-research queries are. The judge-ablation result is the most surprising finding: training-judge choice barely moves recall (Olmo-32B and Gemini 3 Flash produce near-identical retrieval behavior, both open- and closed-source judges work), but training-judge choice does move downstream eval-judge scores — with models trained against GLM-5 and Arcee judging worst on the frontier eval judges. The authors’ interpretation is that this is alignment between training and evaluation judges rather than underlying judge capability, since GLM-5/Arcee-trained models retain competitive recall.

Training dynamics: RLVR mean reward plateaus around step 100 and entropy collapses around step 200, consistent with the model converging to a single keyword_search + semantic_search + finish rollout. Under RaR with the same base model, dataset, and gates, training is “much more straightforward” and produces visibly richer multi-turn refinement.

Future-work axes the authors flag: query-specific (LLM-generated) rubrics for cheap factoid queries where the global rubric over-incentivizes exploration; latency reduction (less explicit reasoning between tool calls; parallel-search-strategy batching à la Native Parallel Reasoner); continual learning from production user click signal.

Why it’s interesting

This is the cleanest filed datapoint that rubric rewards beat verifiable recall rewards specifically because of open-endedness, not because rubrics are inherently better RL signals — the authors are explicit that for HotpotQA-style factoid multi-hop, RLVR is fine. That framing matches the design-space tension already on the wiki: MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling and OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis both train deep-research agents with SFT/RL on synthesized trajectories against verifiable correctness, and both make a bet that the bottleneck is environment scale, not reward shape; this post argues the opposite — that for production research queries where ground truth is not a tight set, the reward formulation is the binding constraint. The named RLVR failure modes (single-tool-call collapse, latent-knowledge reasoning, query-rephrasing) complement the failure modes already documented for Tool-Use Agents (serial collapse, spurious parallelism) and add a new one: reward-aligned reformulation — the agent learns to hill-climb a recall reward by keyword-overlap rather than reasoning. The judge-alignment finding (training judge ≈ eval judge dominates downstream score) is the most actionable lesson for anyone running rubric RL at Luma — pick the eval judge first, then the training judge to match. The setup also contrasts sharply with Training-Free Group Relative Policy Optimization: same underlying GRPO primitive, but Training-Free GRPO keeps the LM frozen and writes the advantage to a natural-language experience library, whereas this work updates parameters and pays the price of needing a rich rubric reward to keep rollouts intelligible.