Training frontier knowledge work agents: A 397B RL training guide with SkyRL

TL;DR

Mercor Research post-trained Qwen3.5-397B-A17B and Qwen3.6-35B-A3B with GRPO-family RL (no SFT warmup) on 1,928 expert-created APEX-Agents tasks — realistic knowledge-work simulations with MCP tool servers over dockerized company filesystems — using public SkyRL, and releases the full training script, model weights, and eval traces. The 397B run improves Pass@1 by 70% relative (16.11% → 27.29%) on a 480-task held-out benchmark; the 35B run gains ~10–12 pts. The blog’s throughline is that the RL algorithm choices (DPPO vs GLM-5 loss, prompt_mean vs token_mean, context-nudge, overlong-filtering) each move scores by 1–4 pts, while environment/harness de-risking (fixing MCP tool bugs, TITO token accounting, timeout discipline, error classification) moves the base model by ~6 pts before any training — the data and infrastructure, not the RL objective, is the load-bearing lever. Gains transfer across harnesses (Archipelago MCP → OpenCode code-only) and to Terminal-Bench 2.1.

Key claims

Post-training moved Qwen3.5-397B-A17B Pass@1 on APEX-Agents from 16.11% → 27.29% (+70% relative) and Qwen3.6-35B-A3B by 10–12 pts across three domains (corporate law, management consulting, investment banking); the 397B gains most in management consulting, the 35B most in corporate law [§Background, Figs. 1–2].
Algorithm ablations on the 35B (Table 1, three-pass mean over 480 held-out tasks) find prompt_mean token aggregation (DAPO/ScaleRL-style, equal weight per rollout group) beats the token_mean default by +3.9 pts because trajectories range 2k–128k tokens and token_mean lets the longest rollouts dominate the gradient [§Step 4, Table 1 run 4 vs 1].
DPPO (binary total-variation-divergence masking on train-inference logprob divergence) and the GLM-5 loss (truncated importance ratio) score within noise of each other at epoch 1; both avoid the extra forward pass that traditional TIS requires — a real cost saving at 100k-token trajectories. DPPO’s clearest effect is behavioral: 21→32 turns and 834→588 assistant tokens per turn (more, shorter turns) [§Step 4, run 3 vs 1].
A “context nudge” harness change — inject a wrap-up notice when 20% of the context budget remains — adds +3.0 pts as a pure training-time effect (all evals ran without the nudge), by reducing zeroed-across-the-board over-length rollouts and increasing usable signal per batch [§Step 4, run 4 vs 1].
Overlong filtering (mask over-length rollouts out of the loss) cost 1.5 pts in this regime (train 160k / eval 256k, both without compaction), and adaptive length penalty was neutral-to-negative — pushing back on both as default recipe ingredients [§Step 4].
Environment/harness de-risking before RL raised base Qwen3.6-35B-A3B from 22.74% → 28.69% mean reward with zero training — roughly one epoch’s worth of gain — via three patches: fixing a PowerPoint MCP tool that returned None on every success, nudging the model to use pdfplumber instead of a layout-flattening PDF-reader MCP, and preinstalling Python packages the agent otherwise burned turns discovering [§Step 1.2].
Token-in-Token-Out (TITO) accounting is treated as a hard prerequisite: re-tokenizing chat-completions text output causes two misalignments (LLM-generated vs trainer-observed tokens, and turn-N-output vs later-turn-input tokens) that silently make training off-policy. Three options — rewrite the harness to use /completions, ask /chat/completions for input+output token IDs, or an RL-framework-provided proxy — are named; Mercor adopted the first [§Step 1.3].
Fully-async RL cluster splits: 12:4 inference:train nodes for the 35B run and 12:8 for the 397B run. Rollout concurrency is bounded by min(KV-cache capacity ÷ trajectory length, (max_staleness_steps+1) × mini_batch_size × n_samples_per_prompt); Mercor set 550 (35B) and 300 (397B) — well under the 1024 algorithmic ceiling, so the systems ceiling bound [§Step 2].
Train-inference logprob mismatch is validated as a routine check post-configuration; the check caught a correctness issue in the combination of vLLM CPU offloading + GDN models + in-flight weight updates. Mean logprob diff <0.03 without rollout router replay [§Step 2 step 4, Fig. 4].
The overfitting run (32 tasks, batch 32, 8 samples/prompt, synchronous) is used as a de-risking gate before spending compute on the hero run — if you cannot overfit a small subset, an end-to-end run cannot work. Tasks graded by diffing files pre/post-rollout were much harder to overfit than tasks graded on the final response alone; a third-party file-diffing tool made them learnable [§Step 3, Fig. 5].
Gains transfer across harnesses despite training on Archipelago (MCP-based): swapping to OpenCode (code-only, no MCP servers, only bash/glob/read/grep/write/edit/todowrite) preserves most of the improvement, with the 35B transferring noticeably better than the 397B [§Step 6, Table 2].
The 35B increasingly relies on code execution over MCP tool calls during training (rising fraction of code-execution tool calls), while the 397B keeps the same MCP preference throughout — attributed to Qwen3.6 having received heavier agentic post-training [§Step 6, Fig. 6].
Terminal-Bench 2.1 (Terminus harness, held out from training) also improves after post-training, and again the 35B transfers better than the 397B — consistent with the code-execution-preference hypothesis [§Step 6, Table 3].
HLE and GPQA differences before/after post-training all fall inside error bars — no reasoning regression on non-agentic benchmarks [§Step 6, Table 4].
Overall framing: “Algorithm choices mattered less than the data: the best of five knobs gave +3.9 points, while post-training as a whole moved both models 10 to 12 points” [§Lessons].

Method

Full pipeline is a six-step recipe explicitly ordered to minimize compute waste:

Step 1 (de-risk environment + harness) — build on Harbor for rollout lifecycle (Terminal-Bench team’s framework, @retry primitives, sandbox providers Modal/Daytona). Each task ships as an image.tar pushed to ECR; Modal pulls the image at trial time to boot the sandbox, which runs MCP servers (Docs/PDF/Email/Chat) over the world filesystem. Timeouts on everything (downloads, MCP calls, teardown). LLM-judge API keys round-robined with backoff to avoid rate limits at 800+ concurrent rollouts. Each agent loop runs as its own Ray task (isolated MCP clients). Every failure classified fail-vs-retry. Trace-reading via coding agents (systematic per-tool failure-rate analysis) surfaced the PowerPoint-None and PDF-flatten bugs.

Step 1.3 (TITO) — the harness is rewritten to use /completions instead of /chat/completions so token IDs flow through the trainer without re-tokenization.

Step 2 (RL systems) — vLLM inference engine + Megatron training backend under SkyRL’s fully-async loop with in-flight NCCL weight sync. Overnight Claude-Code Megatron knob sweep (TP/EP/PP/CP, CPU offload granularity, dynamic max_tokens_per_microbatch). Inference:train GPU split tuned so wait_for_generation_buffer ≈ 0. Rollout concurrency at min(systems ceiling, algorithmic ceiling (max_staleness+1) × mini_batch × n_samples). Train-inference logprob diff verified <0.03.

Step 3 (overfitting) — 32-task subset with non-zero reward variance, batch 32, 8 samples/prompt, synchronous. If it doesn’t overfit, walk back to Step 1.

Step 4 (ablations on 35B) — five knobs swept, epoch-1 checkpoints, 3-pass eval on 480 tasks: token aggregation (token_mean vs sequence_mean vs prompt_mean), policy loss (DPPO vs GLM-5), context nudge (harness change), overlong filtering, adaptive length penalty. Landed configuration: DPPO + prompt_mean + context nudge, no OLF, no ALP.

Step 5 (hero run) — same knobs applied to Qwen3.5-397B-A17B; only the Step 2 systems work differed (12:8 vs 12:4 GPU split, 300 vs 550 concurrency).

Step 6 (evaluation) — three surfaces: same-task-same-harness (APEX-Agents on Archipelago), same-task-cross-harness (APEX-Agents on OpenCode with no MCP servers), cross-task-cross-harness (Terminal-Bench 2.1 on Terminus). Plus HLE + GPQA regression check.

Results

APEX-Agents Pass@1, Archipelago harness: Qwen3.5-397B-A17B 16.11% → 27.29% (+70% relative); Qwen3.6-35B-A3B baseline → +10–12 pt gain (exact baseline harness-dependent; harness fixes alone: 22.74% → 28.69%).
APEX-Agents Pass@16: reported curves show co-scaling with Pass@1 for both models; initial dip attributed to async RL dynamics where easier tasks finish first.
Cross-harness (OpenCode): improvement largely transfers for both models; 35B transfers more than 397B [Table 2].
Terminal-Bench 2.1 (Terminus harness, held out): baselines recreate Artificial Analysis reported scores (35B ~44.6%, 397B ~50.6%) within error; post-training improves both, with the 35B again transferring more [Table 3].
HLE + GPQA: all deltas inside error bars — no non-agentic-reasoning regression [Table 4].
Ablation effect sizes (35B, epoch-1, 3-pass mean over 480 tasks): prompt_mean vs token_mean +3.9 pts; context nudge +3.0 pts; DPPO vs GLM-5 loss within noise; OLF −1.5 pts; ALP neutral-to-negative.
Harness fix effect size (no training): base Qwen3.6-35B-A3B 22.74% → 28.69% mean reward.

Why it’s interesting

The Mercor recipe is a rare end-to-end published case of the Reasoning RL recipe applied to a non-SWE, non-coding, professional-knowledge-work task distribution — investment banking, corporate law, management consulting — with a public benchmark (APEX-Agents), open training script, released weights, and traces. That fills a specific gap on the Agentic Software Engineering page: the OpenAI GPT-5.3-Codex framing named a four-benchmark agentic quadruple (SWE-Bench Pro, Terminal-Bench 2.0, OSWorld, GDPval) but the open cluster had covered only SWE-Bench-* + Terminal-Bench; APEX-Agents is a GDPval-adjacent axis and this is the first open recipe targeting it.

Two of the reported findings sharpen live debates on this wiki. First, the +3.9 pt gain from prompt_mean over token_mean token aggregation is direct evidence for the DAPO/ScaleRL framing that Reasoning RL has been circling but had not yet seen quantified on a long-horizon agentic task with 2k–128k trajectory spread — token_mean’s bias toward the longest rollouts dominating the gradient is the specific mechanism. Second, the “environment/harness de-risking moved the base model +6 pts before any training” observation is the same shape as Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase‘s finding that harness context-management policy moves per-task cost by >2× independent of the model, and complements AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design‘s “harness > model” thesis — but here the harness change is a training-time lever, not just deployment-time.

The DPPO vs GLM-5 within-noise result is also new data on a specific open question: prior filings (TMax: A Simple Recipe for Terminal Agents TMax reports DPPO, GLM-5: from Vibe Coding to Agentic Engineering introduced the GLM-5 loss) had not been directly compared on a shared task and shared 35B base — Mercor’s ablation is the first side-by-side.