Qwen3-Coder-Next Technical Report

TL;DR

Qwen3-Coder-Next is an 80B-total / 3B-active MoE coding-agent LLM, continued from the Qwen3-Next hybrid-attention+MoE base, post-trained specifically for coding-agent workflows (SWE-Bench, Terminal-Bench, multi-scaffold tool use). The central bet is that scaling agentic training — large-scale synthesis of verifiable executable tasks paired with Docker environments, plus a Kubernetes-based rollout infrastructure (MegaFlow) — pushes a 3B-active model into the same SWE-Bench Pro Pareto neighborhood as 32B-active GLM-4.7 and 37B-active DeepSeek-V3.2. Pipeline: continued pretraining on 600B repository-level tokens (262K context, 358 languages), SFT, distill from four domain experts (SWE, QA, WebDev, UX), then RL. Open weights (base + instruct + FP8 + GGUF) ship alongside the report.

Key claims

An 80B-total / 3B-active MoE with hybrid attention reaches 74.2 on SWE-Bench Verified, 66.2 on SWE-Bench Multilingual, 34.6 on SWE-Bench Pro, 32.6 on Terminal-Bench 2.0, and 61.0 on Aider — competitive with or beating DeepSeek-V3.2 (37B-active), GLM-4.7 (32B-active), and MiniMax M2.1 (10B-active) at an order-of-magnitude smaller active footprint [Fig. 2].
The pretraining corpus is updated through Sep 30, 2025 and expands programming-language coverage from 92 (Qwen2.5-Coder) to 370 languages; repository-level data is ~600B tokens, with the training context length extended from 32,768 to 262,144 tokens [§3.1.1].
Web-document reformatting (rewriting Common-Crawl docs into normalized Markdown via Qwen3-Coder-480B-A35B-Instruct) lifts mid-training scores from 54.38 → 63.09 on EvalPlus, 36.02 → 48.35 on MultiPL-E, and 57.13 → 58.94 on CRUX-Eval [Table 1].
Mid-training token scaling on multi-turn agentic data is monotone within the same agent scaffold (OpenHands trajectories → OpenHands eval, SWE-Agent → SWE-Agent), but cross-scaffold transfer is weak — OpenHands-trained checkpoints transfer poorly to SWE-Agent — implying agent-framework specialization is a real axis of training [§3.1.2, Fig. 4].
Approximately 800K verifiable software-engineering task instances are synthesized across 9+ programming languages by mining GitHub PRs (decomposed into buggy state + fix + test patch + Docker verifier) and by injecting controlled bugs into curated containerized repos (SWE-Smith / SWE-Flow / SWE-Rebench / Multi-SWE-RL as seeds) [§2.1].
A specialized environment-building agent constructs runnable Docker environments and verification scripts per PR, with an automated detector that filters non-functional verifiers and a separate QA agent that removes ambiguous tasks before inclusion — a curation gate explicitly motivated by agents “exploiting superficial verification shortcuts” [§2.1].
MegaFlow, an Argo-on-Kubernetes orchestration system co-locating the agent container with the execution-environment container per rollout pod, is the infrastructure layer that makes large-scale agentic RL/training feasible [§2.2].
Search-and-Replace FIM outperforms Chat-FIM at equivalent training scale on code-completion tasks, attributed to alignment with PR-style pretraining data [§3.1.4].
Best-fit packing (BFP) replaces concat-then-split as the sample-packing strategy, eliminating context hallucination and head-side truncation; ablation shows stable gains on long-horizon tasks at the cost of negligible padding [§3.2, Appendix A.3].
Post-training uses four domain experts (SWE, QA, WebDev, UX) distilled back into a single model; the WebDev expert is data-curated via a Playwright/Vite rendering pipeline with VLM-based static visual evaluation + DOM-driven dynamic interaction evaluation [§4.2.1].
A multi-format tool-chat training scheme — training on diverse tool-set definition, invocation, and response-wrapping conventions across many scaffolds (Cline, Qoder, OpenCode, Claude Code, Qwen Code) — is explicitly framed as a generalization recipe against the failure mode of single-template overfit, with rule-based validation of tool-call format used as a training signal [§4.2.2, Fig. 5].
SFT data is filtered via a Mini-SWE-agent user-simulator that executes the proposed assistant solution and gates training pairs on whether the execution meaningfully advances the user task — a closed-loop functional-correctness gate, not just static review [§4.1].

Method

The model is built on Qwen3-Next-80B-A3B-Base (hybrid attention + MoE; 80B total, 3B active per forward pass). Training is a staged pipeline: continued pretraining → SFT → multiple expert specializations → distillation into a single unified model → RL. Continued pretraining mixes natural data (≈600B tokens of repository-level GitHub code across 370 languages, plus reformatted Common-Crawl text–code grounding data) with a smaller synthetic portion (single-turn QA generated by Qwen3-Coder-480B-A35B-Instruct from CC seeds, and multi-turn agentic trajectories generated under six agent frameworks — SWE-agent, Mini-SWE-agent, OpenHands, Claude Code, Qwen Code, Terminus — with the same teacher). Context length is 262,144 tokens with best-fit packing.

The ~800K-task synthetic SWE corpus comes from two complementary pipelines: (a) mining real GitHub PRs and decomposing each into buggy state, fix, and test patch, then constructing a Docker environment + verifier per instance; (b) injecting bugs into curated containerized repos seeded from SWE-Smith / SWE-Flow / SWE-Rebench / Multi-SWE-RL via model-driven rewriting, semantic perturbations, and rule-based transformations, retaining only bugs that fail existing tests and are resolved by patch reversion. All environments are reusable Docker images. Rollouts run on MegaFlow, an Argo-workflow-on-Kubernetes system with rollout/evaluation/post-processing stages per task; the rollout pod co-locates the agent and environment containers.

Post-training trains four expert models (SWE, QA, WebDev, UX) from the post-SFT checkpoint, each with its own data recipe (e.g. WebDev uses Playwright/Vite rendering + VLM static evaluation + DOM dynamic interaction; UX expert is optimized for CLI/IDE tool-call format adherence with rule-based validation), then distills them back into a single model. Tool-calling generalization is targeted explicitly via training on diverse tool chat templates (varying tool-definition syntax, invocation format, response wrapping).

Results

Headline: on SWE-Bench Pro (Fig. 1, the model is positioned on the Pareto frontier between Claude-Opus-4.5 / Sonnet-4.5 and DeepSeek-V3.2 / GLM-4.7 / MiniMax-M2.1 / Kimi-K2.5, achieving competitive performance with ~10× smaller active parameter count than its open-weight peers.

On the coding-agent benchmark table (Fig. 2): Qwen3-Coder-Next 70.6 / 62.8 / 44.3 / 36.2 / 69.9 on SWE-Bench Verified / Multilingual / Pro / Terminal-Bench 2.0 / Aider, vs DeepSeek-V3.2 at 66.2 / 70.2 / 62.3 / 40.9 / 39.3 (note Verified disagreement vs Fig. 1 — Fig. 2 shows DeepSeek-V3.2 ahead on multilingual but Qwen3-Coder-Next ahead on Verified and Aider). GLM-4.7 reaches 74.2 / 63.7 / 40.6 / 37.1 / 52.1 and MiniMax M2.1 reaches 74.8 / 66.2 / 34.6 / 32.6 / 61.0. The pattern is consistent: Qwen3-Coder-Next sits at the small-active-footprint end of the Pareto curve, matching or beating much larger models on a subset of the agentic benchmarks.

Within-scaffold mid-training scaling (1B → 8B tokens) is monotone on both OpenHands and SWE-Agent (Fig. 4 left), but cross-scaffold transfer is weak (Fig. 4 right) — OpenHands → SWE-Agent transfer is near-zero, SWE-Agent → OpenHands is moderate. Web-document reformatting ablation: +8.7 EvalPlus, +12.3 MultiPL-E, +1.8 CRUX-Eval (Table 1).

Open-weights release packaging: base, instruct, FP8, and GGUF variants on Hugging Face and ModelScope, plus a 480B-A35B and 30B-A3B family pair released alongside.

Why it’s interesting

The “3B-active competes with 30+B-active” datapoint is the headline, but the more interesting structural claim — and the one Luma should track — is that agentic training scale, specifically (number of verifiable executable tasks × infrastructure to run them at scale × cross-scaffold tool-call format diversity), is now treated as a first-order training axis on par with parameter and token count. The 800K-Docker-environment corpus + Argo/Kubernetes rollout system + multi-format tool-call training is the most fully described open recipe for this. The cross-scaffold transfer failure result is also useful prior on any agent post-training Luma might run: training on one scaffold’s trajectories does not generalize to another’s, which means the choice of scaffold is a load-bearing data-design decision, not a deployment-time afterthought. Connects directly to InfTool’s Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing closed-loop tool-trajectory synthesis (smaller scale, BFCL-targeted) and to IQuest-Coder-V1 IQuest-Coder-V1 Model Family (the other recent open SWE-tuned LLM family — IQuest at 76.2 SWE-Bench Verified vs Qwen3-Coder-Next at 70.6 with 13× less active compute).