Cut the Bill, Keep the Turns: Affordable Multi-Turn Search RL

TL;DR

A Tencent AIPD report on training a small (Qwen3-8B) multi-turn search agent that reliably executes 10+ retrieval turns, using the open-source rLLM framework, offline Wikipedia / BrowseComp-Plus corpora, and the synthetic multi-turn data from ASearcher. The headline claims are that SFT is not strictly necessary (GRPO from the base model is enough), that long-horizon capability requires a dedicated multi-turn dataset (HotpotQA-style single/two-hop data does not transfer), that training stability is mostly an engineering problem (train/inference mismatch, token-in-token-out alignment, abnormal trajectory handling), and that summarizing retrieved documents with an auxiliary LLM improves both stability and horizon — while generalizing back to raw-document retrieval at evaluation time.

Key claims

A small open base (Qwen3-8B) + GRPO on offline Wikipedia / BrowseComp-Plus + ASearcher’s synthetic multi-turn data is sufficient to train a search agent that reliably performs 10+ retrieval turns [§TL;DR].
SFT is not strictly necessary for multi-turn search RL: GRPO from the base model alone builds stable multi-turn search capability; SFT can give a better starting point for long-horizon training but locks the model into a fixed pattern and causes overuse of search on simple questions [§TL;DR].
A dedicated multi-turn synthetic dataset is critical — standard single-hop / two-hop QA datasets (HotpotQA, etc.) do not produce long-horizon multi-turn search behavior [§TL;DR].
Training stability is dominated by three engineering issues: train/inference mismatch, strict token-in-token-out alignment, and handling abnormal trajectories [§TL;DR].
Summarizing retrieved documents with an auxiliary LLM during training improves stability, supports longer horizons, and the resulting policy still generalizes to setups that return raw search results directly [§TL;DR].
Smaller open-source models still lag closed-source commercial models on multi-turn long-horizon search-and-verify settings — both in effective use of turns and in final accuracy — and this is the gap the report targets [§1].

Method

The recipe is GRPO post-training of Qwen3-8B inside the rLLM training framework, retrieving against an offline Wikipedia / BrowseComp-Plus corpus, with multi-turn trajectories synthesized following the ASearcher pipeline. An auxiliary LLM is used to summarize retrieved documents inside the rollout, shortening the effective context per turn and stabilizing long-horizon optimization. The engineering surface is built around eliminating train/inference mismatch, enforcing strict token-in-token-out alignment between the rollout and gradient steps, and detecting / discarding abnormal trajectories so they do not poison the GRPO advantage estimates. SFT is treated as an optional warm-start rather than a required stage.

Results

The report does not present a clean benchmark table in the fetched section, but the operational claim is that the resulting 8B agent reliably executes 10+ retrieval turns on multi-turn search tasks at low training cost (“cut the bill, keep the turns”) [§TL;DR]. The auxiliary-summarizer trained agent transfers to evaluation setups that return raw search results, suggesting the summarizer is a training-time stability tool rather than a runtime crutch [§TL;DR].

Why it’s interesting

Sits squarely inside the multi-turn search RL design space mapped by Tool-Use Agents and Reasoning RL. The “SFT is not necessary” finding directly contradicts the SFT → DPO → online GRPO three-stage pattern in MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling and the “SFT alone underperforms but is load-bearing for GRPO to work” finding in Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning — the disagreement is worth tracking. The “synthetic multi-turn data is critical” claim aligns with Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing‘s closed-loop synthesis recipe but uses ASearcher’s data instead of self-synthesis, and provides a third datapoint that single-hop QA does not transfer to long-horizon search. The auxiliary-summarizer trick is a new training-time stability primitive on this page — orthogonal to MiroThinker’s streaming rollouts and Training-Free GRPO’s Training-Free Group Relative Policy Optimization frozen-base-plus-experience-library approach.