Forge: Scalable Agent RL Framework and Algorithm

TL;DR

Forge is MiniMax’s in-house agent-RL framework, presented as the deferred technical companion to the MiniMax-M2.5 release. The post frames large-scale agentic RL as an “impossible triangle” between system throughput, training stability, and agent flexibility, and structures the framework around a three-module decoupling — Agent / Middleware / Train+Inference — that lets arbitrary black-box or white-box agent scaffolds plug into the same training loop. Key throughput contributions are Windowed FIFO asynchronous scheduling (a sliding visibility window over the rollout queue that trades a fixed off-policyness for elimination of head-of-line blocking) and Prefix Tree Merging (folding rollouts that share long context-managed prefixes into a single forward pass via attention primitives like Magi-Attention, claimed ~40× training speedup with strict mathematical equivalence). Algorithmically Forge sticks with CISPO (MiniMax’s MoE-stable RL objective) and layers three reward terms over it: process rewards, a task-completion-time reward, and a reward-to-go reformulation for variance reduction. The MiniMax M2.5 release post already used these as headline ingredients; this is the architectural and algorithmic exposition behind that release.

Key claims

Large-scale agent RL faces an “impossible triangle” of system throughput vs training stability vs agent flexibility, formalized as maximizing Effective Agent Training Yield $J = \text{Throughput} \times \text{Sample Efficiency}$ subject to update-variance and convergence bounds across an arbitrary-agent space $\Omega_\text{agent}$ [§Optimization Objective].
Existing RL paradigms hit a “glass ceiling” on agent complexity from two structural flaws: restricted agent autonomy (white-box assumptions about shared state between agent and trainer that block complex context-management / multi-agent cognitive architectures) and a token consistency barrier (TITO architectures couple agent logic with token-level training, making CM consistency between inference and training prohibitively expensive) [§Glass Ceiling].
Asynchronous scheduling faces a deadlock between strict FIFO (suffers Head-of-Line blocking from stragglers) and greedy/FFFO modes (maximize throughput at the cost of a non-stationary distribution shift toward “easy” then “hard” tasks, causing optimization instability) [§Asynchronous Controller].
Forge decomposes into three modules — Agent Side (orchestrates env interactions; agent is a pure trajectory producer), Middleware Abstraction Layer (Gateway server + Data Pool; physically isolates agent from training/inference), and Training+Inference Side (Rollout Engine + Train Engine consuming from the Data Pool) [§Middleware Design].
The Middleware design is what enables black-box agent training: agents simply route completion requests to the RL service Gateway over standard protocols, and Forge handles data collection and policy updates transparently — supporting arbitrary context manipulations (memory compression, history rewriting) and any internal Agent Loop (Deep Think, Multi-Agent architectures) without modifying the agent [§Black-box Agent Experiment].
Forge integrates Context Management directly into the RL interaction loop as a functional agent action driving state transitions ( $S_t \to S_{t+1}$ implicitly encapsulates context-switching logic), training the policy to anticipate CM operations and prioritize “state-critical” tokens — closing the inference-training mismatch where CM-at-inference-only causes distribution shift [§CM-Driven State Transitions].
Windowed FIFO is the proposed asynchronous scheduling primitive: the training scheduler is restricted to a visibility window $W$ (e.g. $W = 0.3N$ ) over the rollout queue, allowing greedy disorder inside the window (mitigating HoL blocking) but strict blocking at the window boundary (preventing distribution drift toward fast/easy samples later in the batch) [§Windowed FIFO].
Prefix Tree Merging addresses prefix redundancy from context management: multiple completions sharing long common prefixes are merged into a single prefix-tree forward pass using attention primitives such as Magi Attention; the tree is deconstructed by metadata for loss computation, yielding strict mathematical equivalence to standard training [§Prefix Tree Merging].
Headline throughput claim: ~40× training speedup and significantly reduced memory overhead from prefix-tree merging, with zero impact on loss or metrics [§Prefix Tree Merging].
Generation-side optimizations are three: MTP-based speculative decoding (Multi-Token Prediction heads continuously fine-tuned via Top-K KL loss to stay aligned with the evolving RL policy), Heterogeneous PD Disaggregation (decoupling Prefill and Decode with independent parallelism strategies to eliminate PD interference under mixed-MoE scheduling), and a Global L3 KV Cache Pool (DFS-backed cache with a cost-aware scheduler that weighs queuing delay against cache migration costs for group-level rollout) [§Generation Pipeline].
Forge uses CISPO as the core RL algorithm, plus Unified Mixed-Domain Training: rather than multi-stage RL with negative transfer, Reasoning + General QA + Agent domains are mixed simultaneously in a single training stage [§Unified Mixed-Domain Training].
The reward framework is composite: Process Reward (dense feedback on intermediate behaviors like language mixing or tool invocation errors), Task Completion Time Reward (relative completion time as a reward signal that incentivizes parallelism over serial tool execution), and Reward-to-go (replacing sparse outcome rewards to reduce gradient variance and sharpen credit assignment over ultra-long 200K-context trajectories) [§Composite Reward].
Reported operational envelope behind MiniMax M2.5: over a hundred thousand distinct real-world agent scaffolds and environments, context lengths up to 200K, daily processing throughput on the scale of millions of samples, with consistent reward convergence [§Introduction].
Forge integrates over hundreds of types of scaffolds and thousands of distinct tool invocation formats, with the M2.5 OpenCode Agent trained entirely as a black box through Forge [§Black-box Agent Experiment].

Method

Forge is structured as a three-module decoupling. The Agent Side abstracts the General Agent (white-box or black-box) and its environment; the agent is treated as a pure trajectory producer that focuses on context management and reasoning chains, agnostic to the underlying training/inference mechanics. The Middleware Abstraction Layer physically isolates the Agent Side from training/inference via two components: a Gateway Server that serves as a standardized communication endpoint between agents and the LLM under a common completion protocol, and a Data Pool — a distributed asynchronous store of rollout trajectories that decouples generation from training and supports flexible data processing/batching. The Training and Inference Side consists of a Rollout Engine dedicated to high-throughput generation and a Train Engine that consumes processed sequences from the Data Pool and stays synchronized with the rollout policy distribution.

The system-level innovations sit at the Middleware/Train boundary. Windowed FIFO is the asynchronous-scheduling middle ground: the rollout queue $Q = [T_0, T_1, ..., T_{N-1}]$ is exposed to the training scheduler only through a sliding window of size $W$ . Inside $[T_i, T_{i+W-1}]$ the scheduler is free to grab any completed trajectory greedily (mitigating HoL blocking); outside the window completed tasks cannot be consumed even if ready (preventing data-distribution drift). The window slides forward only as the head consumes, forcing the scheduler to wait for stragglers within the current window and bounding the off-policyness contribution from greedy fetching.

Prefix Tree Merging addresses the structural overlap that arises in multi-turn agent dialogues under context management: many completions share long common prefixes after CM (truncation, summarization, history rewriting), and traditional training treats each as an independent sample, recomputing those prefixes redundantly. Forge replaces this linear processing with a tree-structured forward pass — completions are merged into a single prefix tree, attention primitives like Magi-Attention ensure the tree forward is logically identical to N separate forward passes, and metadata is used after the pass to deconstruct the tree for per-completion loss computation. Reported result: ~40× speedup, large memory savings, zero loss/metric impact.

Generation-side: MTP-based speculative decoding uses Multi-Token Prediction heads (rather than a static draft model) continuously fine-tuned via Top-K KL loss against the evolving RL policy, sustaining acceptance rates as policy distributions shift through training. Heterogeneous PD Disaggregation runs Prefill and Decode on independently parallelism-configured instances to eliminate the mixed-MoE PD interference. The Global L3 KV Cache Pool is a DFS-backed cache shared across rollout workers; a cost-aware scheduler routes requests by trading queuing delay against cache-migration cost to maximize prefix-cache hit rate at group-level rollout.

Algorithmically, Forge keeps CISPO (MiniMax’s earlier MoE-stable RL objective, used since M1/M2) and adds three reward components. Process Reward penalizes intermediate failure modes (language mixing, invalid tool calls) for dense feedback. Task Completion Time Reward brings wall-clock cost into the reward — multiple valid trajectories exist with different latencies due to tool-execution overhead and serial processing, and relative completion time as a reward signal incentivizes parallel tool use over sequential strategies. Reward-to-go replaces standard sparse outcome rewards with a normalized return formulation to reduce gradient variance and improve credit assignment over the 200K-context multi-thousand-action trajectories. Unified Mixed-Domain Training mixes Reasoning, General QA, and Agent training tasks in a single stage to avoid the negative transfer of multi-stage RL.

For black-box agent training (the deployment-relevant case where customers run proprietary scaffolds), the Middleware design is the whole story: agents call the Gateway as if it were a standard LLM endpoint, Forge collects trajectories transparently, and the only requirement is that completion requests flow through the Gateway. Forge thus supports arbitrary context manipulations (memory compression, history rewriting) and arbitrary internal agent loops (Deep Think, multi-agent) without intrusive integration — the M2.5 OpenCode Agent is reported as trained entirely as a black box through this mechanism.

Results

The post does not present standalone benchmark tables for Forge itself — those are folded into the MiniMax-M2.5 release post. The quantitative claims in the Forge post are:

~40× training speedup from Prefix Tree Merging via attention primitives, with strict mathematical equivalence to standard training (no loss/metric impact) [§Prefix Tree Merging].
Operational envelope: >100,000 distinct real-world agent scaffolds and environments, up to 200K context length, daily throughput on the scale of millions of samples [§Introduction].
Scaffold and tool coverage: hundreds of scaffold types, thousands of distinct tool-invocation formats integrated [§Black-box Agent Experiment].
Empirical claim on black-box generalization: “consistent, stable improvements even across completely opaque black-box systems” — including agents using aggressive context reduction such as Truncate BC, with M2.5 OpenCode trained entirely as a black box [§Black-box Agent Experiment].

For downstream model numbers attributable to this framework see the filed M2.5 page: 80.2 SWE-Bench Verified, 76.3 BrowseComp, 76.8 BFCL multi-turn, 37% faster end-to-end SWE-Bench Verified runtime (22.8 min vs 31.3 min on M2.1) — all reportedly achieved through Forge’s scaling rather than architectural changes from M2/M2.1.

Why it’s interesting

This is the system-and-algorithm exposition the MiniMax M2.5 release deferred; the M2.5 paper page on this wiki explicitly flagged “a separate technical blog post on RL scaling is promised” and this is that post. For the team it pins down what was previously hand-waved as “scaled agent RL”: Windowed FIFO as a specific asynchronous-scheduling primitive, Prefix Tree Merging via Magi-Attention as the source of the ~40× speedup, and a composite reward (process + completion-time + reward-to-go) over CISPO as the algorithmic surface. The Middleware-as-Gateway pattern is also notable as the cleanest filed example of training a third-party scaffold’s policy without intrusive integration — relevant if Luma ever wants to RL-train a model against an external scaffold (e.g. someone else’s coding agent harness).

The most interesting algorithmic claim for Luma’s agent-RL thinking is task-completion-time reward. Kimi K2.5 PARL put a critical-path-latency metric (Critical Steps) into the orchestrator’s reward to incentivize parallel sub-agent spawning; MiniMax Forge puts wall-clock completion time into a single-agent RL reward to incentivize parallel tool use. These are two distinct routes to “latency-in-the-objective” — orchestration-level vs tool-call-level — and Forge is now the second filed datapoint that latency in the RL reward is becoming a recipe component, not a research curiosity.

Windowed FIFO is the cleanest framing yet of the “off-policyness vs straggler-utilization” knob in long-horizon agent RL. MiroThinker’s “streaming rollouts” mechanism re-queues unfinished tasks each iteration; Windowed FIFO instead bounds which completed tasks the scheduler can see at any moment, fixing off-policyness as a tunable window-size parameter rather than a property of which tasks happened to finish. Worth seeing reproduced.

The promise of “strict mathematical equivalence” for Prefix Tree Merging is load-bearing for the 40× claim; this depends on Magi-Attention (a MiniMax attention primitive) doing the same forward computation as N independent passes through shared prefixes. Independent verification would be useful — if the equivalence is real and the speedup holds, this is a general training-throughput primitive for any multi-turn agent RL setup where context management produces shared prefixes (which is, by now, most of them).