ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration

TL;DR

ToolOrchestra trains an 8B “orchestrator” LLM (Qwen3-8B base) with GRPO to coordinate a heterogeneous toolkit of basic tools (web search, code interpreter, local search) and other LLMs as tools (specialized + generalist, from Qwen3-8B up to GPT-5). The reward is a three-term composite — outcome correctness (GPT-5-judged), efficiency penalties on monetary cost and wall-clock latency, and a user-preference alignment term over tool-usage vectors — enabling a small orchestrator to outperform monolithic frontier models at substantially lower cost. Orchestrator-8B reaches 37.1 on HLE (text-only subset), beating GPT-5 (35.1) at roughly 30% of the cost and 2.5× faster, and generalizes to unseen tools / pricing configurations. Released as nvidia/Nemotron-Orchestrator-8B on HuggingFace alongside the ToolScale synthetic training dataset.

Key claims

An 8B Qwen3-based orchestrator coordinating a toolkit that includes larger models (GPT-5, Claude Opus 4.1, Qwen3-235B-A22B) outperforms each of those models used monolithically: 37.1 HLE vs GPT-5 35.1 (basic tools) and 21.2 (GPT-5 as orchestrator over the same expanded toolkit) [Table 1]. [§5]
Prompting an off-the-shelf LLM to act as orchestrator exhibits two systemic failure modes that motivate trained orchestration: self-enhancement bias (GPT-5 disproportionately delegates to GPT-5-mini) and default-to-strongest (Qwen3-8B over-routes to GPT-5 regardless of cost or utility). [§1, Fig. 3]
The reward is a weighted sum of outcome correctness $r_o$ (binary, GPT-5 judge), an efficiency term $\alpha r_c + \beta r_l$ (per-trajectory monetary cost via TogetherAI/proprietary-API pricing + wall-clock latency, batch-normalized), and a preference term $\sum_i p_i \tilde r^{u_i}$ over a per-user preference vector $p$ indicating which tools to up- or down-weight. [§3.2, Eq. 2]
GRPO is the optimizer. Training stabilization requires three filters during backprop: homogeneity filtering (skip batches whose reward std is below threshold), format consistency filtering (drop malformed tool calls), and invalid output filtering (drop trajectories without a valid answer) — without these the KL loss is reported to explode in this agentic setup. [§3.2]
ToolScale is a synthetic multi-turn tool-use training corpus generated by a two-step LLM pipeline: (1) generate per-domain database schema + tools, (2) generate diverse tasks with golden function calls + required-mentioned-info, filter on execution correctness + process fidelity + operation completeness + multi-LLM pass@N failure. ToolScale spans 10 domains and is released. [§3.3, Table 5]
Generalization to unseen tool sets is non-trivial. On a held-out tool configuration (Claude Opus 4.1 / o3-mini / GPT-4o as query writers; Codestral as code writer; OpenMath-Llama-2-70b / DeepSeek-Math-7B as math models; Claude Opus 4.1 / Claude Sonnet 4.1 / Gemma-3-27B as generalists) Orchestrator-8B still gets 22.0 HLE / 73.8 FRAMES / 48.8 τ²-Bench at lowest cost among all baselines (Qwen3-8B: 12.6; GPT-5 prompted-orchestrator: 16.4). [Table 2, §6.3]
Preference adherence: when the task instruction is concatenated with a per-user preference vector $p$ , Orchestrator-8B scores 46.7 / 68.4 / 79.5 on the Preference-aware HLE / FRAMES / τ²-Bench test split — beating GPT-5 (34.6 / 62.3 / 70.3) and Claude Opus 4.1 (40.2 / 63.4 / 73.5) on a metric defined by the same reward formula used in training. [Table 3, §6.4]
Cost-vs-accuracy frontier: at the 10/20/50/100 maximum-turn budgets, Orchestrator-8B Pareto-dominates GPT-5, Claude Opus 4.1, and Qwen3-235B-A22B on HLE — both higher accuracy at fixed budget and lower budget for matched accuracy. [Fig. 6, §6.2]
Tool-call distribution: Orchestrator-8B calls GPT-5 ~40% of steps and routes the rest to cheaper models or basic tools, while Claude Opus 4.1 (as orchestrator) over-relies on GPT-5 and GPT-5 (as orchestrator) routes mostly to GPT-5-mini — i.e. the trained policy avoids both bias failure modes named in §1. [§6.1, Fig. 5]

Method

The agent loop is a standard ReAct-style multi-turn rollout: at each turn the orchestrator emits a reasoning trace, picks one tool from a unified JSON-schema catalog (which uniformly covers deterministic tools like Tavily web search / Python sandbox / Faiss local search and “intelligent tools” — other LLMs exposed by name + capability description), receives an observation, and continues until termination or 50 turns. The training innovation is in the reward, not the loop: per-trajectory tally vectors over $\{r_o, r_c, r_l, r^{u_1}, …, r^{u_n}\}$ are batch-normalized via min–max scaling, multiplied by a preference vector $p$ , and summed into a scalar advantage that GRPO normalizes group-relatively (Eq. 3) and updates against a clipped surrogate (Eq. 4).

Tool descriptions for LLMs-as-tools are auto-generated: sample 10 training tasks, get an LLM’s trajectory on each, ask another LLM to write a capability description from the trajectories + outcomes. Pricing for open-source models is sourced from TogetherAI; pricing for proprietary models from official APIs. To prevent overfitting to a fixed toolkit, each training instance gets a random subset of tools and randomized pricing — this is what enables the unseen-tool / unseen-pricing generalization in §6.3.

The 8B model trains on Qwen3-8B (warm-started from a GeneralThought-430K subset) on 16 H100s with a 24K input context and 8K generation budget, batch size 16, rollout batch 8, learning rate 1e-6, max 50 turns/task.

Results

Three benchmarks, three settings (no tools / basic tools / basic + specialized + generalist LLMs):

HLE (text-only subset): Orchestrator-8B 37.1, beating the strongest prior at any cost (GPT-5 + basic tools 35.1, Claude Opus 4.1 + tools+models 34.6). On the cost frontier (Fig. 6), Orchestrator-8B beats GPT-5 at every budget tested. Monetary cost 9.2 vs GPT-5’s 30.2 with basic tools and Claude Opus 4.1’s 76.2 — roughly 3× and 8× cheaper. [Table 1]
FRAMES: Orchestrator-8B 76.3 vs GPT-5 + basic tools 74.0, Qwen3-235B-A22B + tools+models 74.2.
τ²-Bench: Orchestrator-8B 80.2 vs GPT-5 + basic tools 77.7. Orchestrator-8B beats GPT-5 + basic tools while invoking GPT-5 in only ~40% of steps (§6.1).
Unseen tools (held-out model configuration, §6.3): Orchestrator-8B 22.0 / 73.8 / 48.8 HLE / FRAMES / τ²-Bench at cost 34.8 and latency 8.2 — still Pareto-best vs GPT-5 (16.4 / 54.8 / 44.8 at cost 81.3, latency 14.6).
Preference adherence (§6.4): Orchestrator-8B 46.7 / 68.4 / 79.5 vs GPT-5 34.6 / 62.3 / 70.3 on the preference-conditioned reward.

Why it’s interesting

ToolOrchestra is the cleanest filed datapoint for small-orchestrator-over-big-tools as a deployment pattern, and it lands in the middle of several active threads on the wiki. It is the parameter-space dual to Training-Free Group Relative Policy Optimization — both papers use GRPO’s group-relative-advantage primitive over multi-turn tool-use rollouts, but ToolOrchestra updates the orchestrator’s weights against a multi-objective reward while Training-Free GRPO updates a natural-language experience library on a frozen frontier model; ToolOrchestra spends GPU on training and saves API spend at inference, Training-Free GRPO does the opposite. ToolOrchestra is also the first filed paper to make cost and latency first-class terms in the RL objective alongside correctness — a concrete realization of the latency-in-reward pattern that Kimi K2.5: Visual Agentic Intelligence‘s PARL Critical-Steps metric opened up, and it adds the user-preference vector axis on top, which neither MiroThinker nor PARL trains for. The paper also sharpens MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling‘s “interaction depth scales” thesis: ToolOrchestra holds the 50-turn budget fixed and instead scales tool diversity (calling LLMs as tools, not just APIs), suggesting interaction depth and tool intelligence are independent levers on the same Pareto frontier.