Skip to content
Bud's Wiki

Agentic Software Engineering

TL;DR

Section titled “TL;DR”

Agentic SWE is the post-training regime that turns a code LLM into a long-horizon coding agent: SWE-Bench(-Verified/-Multilingual/-Pro), Terminal-Bench, BigCodeBench, and Aider are the de facto evaluation suite, and the common recipe is large-scale synthesis of verifiable executable tasks (PR mining + bug injection + Docker environment + automated verifier) paired with multi-stage post-training (SFT → domain experts → distillation → RL with environment feedback). Filed papers converge on three load-bearing pieces: (a) execution-grounded reward / verification (not static review), (b) cross-scaffold tool-call format diversity at training time, and (c) closed-loop synthesis where the trained model is fed back into the next round of trajectory generation.

Key claims

Section titled “Key claims”
  • SWE-Bench Verified / Multilingual / Pro and Terminal-Bench are the common evaluation axis across the cluster, with the latest open models clustering on a Pareto frontier in (active parameters, SWE-Bench Pro) space — Qwen3-Coder-Next 80B-total/3B-active reaches 44.3 SWE-Bench Pro, DeepSeek-V3.2 (37B-active) 40.9, GLM-4.7 (32B-active) 40.6, MiniMax M2.1 (10B-active) 34.6, Kimi-K2.5 in the same neighborhood (Qwen3-Coder-Next Technical Report Fig. 1, Fig. 2). IQuest-Coder-V1 40B-Instruct reports 76.2 on SWE-Bench Verified (IQuest-Coder-V1 Model Family Highlights). Closed-source GPT-5.3-Codex sets the upper bound on this same axis at 56.8 SWE-Bench Pro per its release post (Introducing GPT-5.3-Codex). Step 3.5 Flash (11B-active / 196B-total) reports SWE-bench Verified 74.4 and a leading Terminal-Bench 2.0 51.0 against the same open-MoE peer set (Step 3.5 Flash Benchmark table).
  • SWE-Bench Pro is the emerging consensus benchmark replacing SWE-Bench Verified as the headline metric in this cluster, motivated by being multi-language (4 langs), more contamination-resistant, and more industry-relevant (Introducing GPT-5.3-Codex; Qwen3-Coder-Next Technical Report Fig. 1).
  • Large-scale synthesis of verifiable executable tasks is the dominant data-generation pattern: Qwen3-Coder-Next synthesizes ~800K SWE tasks across 9+ languages by mining GitHub PRs (decomposed into buggy state + fix + test patch + Docker verifier) and by injecting controlled bugs into curated containerized repos seeded from SWE-Smith / SWE-Flow / SWE-Rebench / Multi-SWE-RL (Qwen3-Coder-Next Technical Report §2.1). InfTool extends the same idea to tool-use with a three-agent role-play synthesizer (User Simulator + Tool-Calling Assistant + MCP Server) closing the loop into BFCL-tagged trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
  • Verification gates are explicitly designed against agents that “exploit superficial verification shortcuts”: Qwen3-Coder-Next runs a specialized environment-building agent + automated detector for non-functional verifiers + a separate quality-assurance agent that strips ambiguous tasks before inclusion (Qwen3-Coder-Next Technical Report §2.1). InfTool applies GRPO-with-gated-rewards over verified trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
  • Execution-grounded SFT filtering replaces static review at the SFT stage: Qwen3-Coder-Next deploys a Mini-SWE-agent user-simulator that executes the proposed assistant solution and gates training pairs on whether the execution meaningfully advances the user task (Qwen3-Coder-Next Technical Report §4.1) — the SFT-stage analog of the RL-stage executable reward.
  • Cross-scaffold tool-call format diversity at training time is treated as a generalization recipe, not a deployment afterthought: Qwen3-Coder-Next explicitly trains on diverse tool chat templates (varying tool-set definition, invocation, and response-wrapping syntax across Cline, Qoder, OpenCode, Claude Code, Qwen Code) with rule-based tool-call-format validation as a training signal (Qwen3-Coder-Next Technical Report §4.2.2, Fig. 5). This is motivated by an empirical mid-training scaling finding: within-scaffold token scaling is monotone but cross-scaffold transfer is weak — OpenHands-trained checkpoints transfer poorly to SWE-Agent (Qwen3-Coder-Next Technical Report §3.1.2, Fig. 4). Step 3.5 Flash makes a similar claim from the deployment side — Claude-Code compatibility is foregrounded as a release feature (Step 3.5 Flash §Agentic Coding).
  • Domain-expert distillation is becoming a standard post-training step: Qwen3-Coder-Next trains four expert models (SWE / QA / WebDev / UX) from the post-SFT checkpoint, each with its own data recipe — WebDev uses a Playwright/Vite rendering pipeline with VLM static evaluation + DOM dynamic interaction evaluation — then distills them back into a single unified model (Qwen3-Coder-Next Technical Report §4.2). IQuest-Coder-V1 splits post-training into Instruct (instruction-following) and Thinking (reasoning-driven RL) tracks at every size (IQuest-Coder-V1 Model Family Highlights). Closed-source GPT-5.3-Codex frames its release as the consolidation of separate coding (5.2-Codex) and reasoning (5.2) lines into a single unified model (Introducing GPT-5.3-Codex), suggesting the open trend toward distill-back-to-single-model is mirrored at the closed-source frontier. The subsequent GPT-5.4 launch tweet extends the same consolidation framing — “reasoning, coding, and agentic workflows into one frontier model” — across ChatGPT (Thinking, Pro), API, and Codex on the same day (GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement)).
  • Context length extension is co-required: Qwen3-Coder-Next extends from 32,768 to 262,144 tokens for repository-level pretraining, motivated explicitly by cross-file dependencies and multi-turn agentic trajectories (Qwen3-Coder-Next Technical Report §3.1.1). IQuest-Coder-V1 ships native 128K context across all sizes without rope-scaling tricks (IQuest-Coder-V1 Model Family Model Overview). Step 3.5 Flash supports 256K context via 3:1 SWA:Full hybrid attention (Step 3.5 Flash §Architecture).
  • Closed-loop synthesis (trained model → next round of synthesizer) is the data-generation primitive: InfTool’s 19.8% → 70.9% BFCL improvement over closed-loop iterations is the cleanest filed datapoint (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract). Qwen3-Coder-Next uses the prior model Qwen3-Coder-480B-A35B-Instruct as the teacher for trajectory and QA data (Qwen3-Coder-Next Technical Report §3.1.2). The recipe is now the same shape at two very different scales. GPT-5.3-Codex extends the pattern one layer up: early checkpoints assisted the training process itself (debugging the training run, deployment monitoring, alpha-test telemetry analysis), described as “the first model that was instrumental in creating itself” (Introducing GPT-5.3-Codex).
  • MoE with low activation ratios is the architectural common factor among the latest agentic-SWE models: Qwen3-Coder-Next 80B-total / 3B-active, DeepSeek-V3.2 37B-active, GLM-4.7 32B-active, MiniMax M2.1 10B-active (Qwen3-Coder-Next Technical Report Fig. 1). Step 3.5 Flash adds another datapoint at 11B-active / 196B-total with the lowest normalized inference cost in its peer table (Step 3.5 Flash Benchmark table). IQuest-Coder-V1 is the dense outlier in the cluster (7B/14B/40B) (IQuest-Coder-V1 Model Family Model Overview).
  • The benchmark frontier is shifting from pure-coding axes to a four-benchmark agentic quadruple — SWE-Bench Pro (coding), Terminal-Bench 2.0 (terminal skills), OSWorld (computer-use), GDPval (professional knowledge work) — under the framing that a coding agent is becoming “a general-purpose agent that can reason, build, and execute” (Introducing GPT-5.3-Codex). No filed open model yet reports the full quadruple; the open cluster currently covers SWE-Bench-* + Terminal-Bench but not OSWorld or GDPval.
  • Decoding-throughput / inference-cost is increasingly co-reported with capability metrics on this Pareto. Step 3.5 Flash reports 100–300 tok/s typical throughput (peak 350 tok/s for single-stream coding) on Hopper GPUs via MTP-3 + sparse-active MoE, and normalizes its peer table to estimated decoding cost at 128K context — claiming 1.0× vs DeepSeek V3.2 6.0×, Kimi K2 Thinking / GLM-4.7 18.9×, MiniMax M2.1 3.9×, MiMo-V2 Flash 1.2× (Step 3.5 Flash Benchmark table). First filed agentic-SWE entry that puts inference cost on the headline axis alongside SWE-Bench Verified / Terminal-Bench scores.

Recent contributions

Section titled “Recent contributions”
  • [2026-08-27] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident: METR’s Hugging Face incident audit — the same coding-agent + shell + package-manager stack that agentic-SWE post-training relies on was, in a natural cyber-eval, used at 1,200-agent scale to reverse-engineer HMAC flag generation, coordinate multi-day cheating R&D, and chain to a real-world RCE against a third party; concrete evidence that “isolation between sandboxed coding agents” is not a safe assumption for shared package-cache environments.
  • [2026-08-20] The Multimodal Intelligence of Muse Spark 1.2: Muse Spark 1.2 (MSL) is grounded on Design Arena’s image/video → web-code leaderboard (as of 2026-08-18) as its visual-coding evaluation; the model re-examines its own generated artifacts to drive a self-improvement loop, extending the code-agent inner loop to a multimodal deliverable substrate.
  • [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5 (397B MoE, open MIT) matches Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85.0) and DeepSWE (56 vs 59), with SWE-Bench Verified 86 / Pro 65.1 / Multilingual 79.6 — a new open-source Pareto point above GLM-5.2 (82.7/46.2) and DeepSeek-V4-Flash-0731 (82.7/54.4). Replaces the static PR-mining + bug-injection synthesizer of Qwen3-Coder-Next with a self-evolving proposer that targets a p*=0.2 empirical success frontier and jointly optimizes proposer + harness + rollouts under GRPO. Also reports 35B-A3B beating dense Gemma-4-31B / Muse-Glimmer-30B by wide margins (Terminal-Bench Claude Code 68.5 vs 43.4 / 51.7).
  • [2026-08-18] AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design: AutoDesign extends the “harness > model” thesis to a visual-design deliverable (paper→poster): the same code-agent inner loop, but with an outer meta-harness optimizer that rewrites the harness from rollout feedback. A learned DesignHarness dropped into seven independent (code-agent, model) pairs lifts mean PosterBench score 54.99 → 67.39 (+12.4 pts), and the full pipeline runs at 253 tool calls / 11 editing turns / <$3 / ~40 min — quantitative support for the concept’s claim that harness design is a first-class capability + cost knob independent of the underlying model.
  • [2026-08-17] Training AI Scientists to Replicate Research: Faraday (Inherent Labs, 27B) directs GPT-5.5 Codex as a coding-agent tool via long-horizon RL trained on paper-figure-replication tasks, and generalizes at test time to directing a stronger tool than the one it trained against (train with GPT-5.4-mini → eval on GPT-5.5 Codex without retraining). A datapoint that scientific-taste post-training on a smaller model can improve production coding-agent output over the coding agent used standalone in its own harness.
  • [2026-08-14] behavior-judge — compiling Agent Behavior specs into deterministic + semantic judges: behavior-judge (Khaitan) targets long-horizon coding/ops agents (worked examples: tax research, refund support, staged-rollout SRE) as the deployment surface — compiles a Braintrust Agent Behavior spec into deterministic + semantic checks over trajectory events, with every verdict grounded in a verbatim quote from the spec. Adds a process-supervision evaluator primitive to the outcome-eval-heavy benchmarks (SWE-Bench, Terminal-Bench, etc.) this concept has been tracking.
  • [2026-08-14] Qwen3.8 open weights release — 27B dense multimodal + 2.4T-A95B MoE (Alibaba Qwen): Qwen3.8-27B is positioned in the launch tweet as excelling in “real-world coding and office workflows” beyond overall benchmark gains over Qwen3.7-Plus — the Qwen release cadence continues to foreground agentic-coding as the headline capability for open dense mid-scale models.
  • [2026-08-14] Thea: Towards the Harness of Embodied Agents: Thea explicitly ports the coding-agent harness paradigm to embodied agents — inheriting context-lifetime tiers, on-demand skill loading, per-tool experience memory, and safety hooks — with the load-bearing observation that “harness > model” transfers cleanly across substrates as long as the two software-only abilities (world state readability, action-outcome exit codes) are supplied by adapter primitives.
  • [2026-08-10] Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device: Muse Glimmer (MSL, 30B open-weights) advertises end-to-end task-completion training against DeepSearch QA, MCP-Atlas, τ-Bench, and SWE-Bench, plus failure-recovery training (diagnose tool-call error and retry). First MSL entry in the agentic-coding cohort with actual open weights — comparators are Kimi K2.6, Cursor Composer 2.5, GPT-5.3-Codex, and MiniMax M2.7 (closed).
  • [2026-08-06] Retire the Abstractions — CUDA DSLs Heading Toward Retirement: Hazy Research’s manifesto on retiring CUDA DSLs frames agentic-SWE as compiler-of-intent: DSLs and frameworks are specifications for a dumb executor, and lose their ground once the executor is intelligent enough to fill gaps from underspecified prompts. Cites the Cursor Mixture of Kittens MoE megakernel as existence proof for building complex GPU code without an intermediate C++ abstraction — the qualitative counterpart to CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation‘s KernelBench numbers.
  • [2026-08-04] Assemble Benchmark — DAgger with Code-as-Policy (HUD.ai): HUD.ai’s Assemble benchmark post proposes DAgger-with-code-as-policy for coding-agent training — a distinct axis from the GRPO / on-policy-distillation recipes dominant in this cluster, where the expert relabels programs at every visited state instead of only rewarding terminal test-pass ratios. Filed from Slack pointer; blog content not retrievable at filing time (HTTP 429).
  • [2026-08-04] AgentOmnia: Scaling Agentic Models for Full-Scenario Applications: AgentOmnia extends the agentic post-training pattern beyond SWE-only benchmarks to a ToC/ToB/ToE task-space spanning 90 level-1 domains, evaluated on OmniaBench + τ²-Bench + DeepPlanning + VitaBench macro-average (22.86 → 41.69).
  • [2026-07-31] DeepSeek-V4-Flash-0731 — post-training upgrade of V4-Flash, benchmarks now beating GLM-5.2: DeepSeek ships DeepSeek-V4-Flash-0731 — a post-training-only refresh of the earlier 158B-chat V4-Flash checkpoint (same architecture and size), claiming the upgraded Flash now far surpasses V4-Pro-Preview on the attached benchmark image; Slack framing is that it beats GLM-5.2 across the board at ~3× smaller. New API-level datapoint that a post-training-only refresh at fixed active parameters can move a model past a much larger frontier peer on agentic benchmarks; also ships native OpenAI Responses API format and Codex compatibility.
  • [2026-07-30] Introducing Inkling-Small — Thinking Machines' efficient open-weights MoE reasoning model: Inkling-Small exceeds 80% on SWEBench Verified (above Inkling’s 77.6%) after two additional weeks of agentic-coding RL layered on top of on-policy distillation from Inkling — a small-model + focused-RL recipe for pushing coding benchmarks.
  • [2026-07-30] Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World-Centric Foundation GUI Agents: Qwen-UI-Agent (MAI-UI, Alibaba) reports Terminal-Bench 2.0 avg-5 of 50.1 from a Qwen3.5-27B-based mobile/desktop GUI agent — a non-SWE-focused release that nevertheless lands in the agentic-coding-adjacent Pareto zone, and highlights that GUI-specialist peers score 0.0–9.0 on the same benchmark.
  • [2026-07-28] DozenDucc — Waddle demos were pure code-as-policy, no robotics models under the hood: Confirms Waddle’s shipped demos were pure write-code → execute → revise with no learned robotics-model fallback — the agentic-SWE loop was load-bearing on its own for the demo suite.
  • [2026-07-27] Introducing Waddle: Agents that Control Robots: Waddle Labs extends the write-code → execute → revise agentic-SWE loop to a physical execution substrate; reports that larger reasoning models with larger inference budgets solve harder manipulation tasks (only Fable 5 and GPT-5.6 xhigh solve t-shirt folding).
  • [2026-07-27] Kimi K3 Technical Report — Open Frontier Intelligence: Kimi K3 tech report — reports SWE-Marathon 42.0 (best), FrontierSWE 81.2, Terminal-Bench 2.1 88.3, ProgramBench 77.8, and Kimi Code Bench 2.0 72.9, positioning K3 as the open-weights frontier for long-horizon coding behind Claude Fable 5 and GPT-5.6 Sol.
  • [2026-07-24] optimize_anything Goes omni — Composing Optimizers into Meta-Optimizer Pipelines: Uses Frontier-CS (open-ended competitive-programming judged by hidden tests) as the evaluation target for three LLM-driven proposers (GEPA reflective, AutoResearch coding-agent, Meta-Harness) and reports the first head-to-head comparison at matched Claude Sonnet 4.6 / $20 budget: standalone AutoResearch 55.4 > Meta-Harness 50.9 > GEPA 43.8 > zero-shot 7.72; per-problem winner is unpredictable. The composed meta-optimizer omni hits 63.2 on Frontier-CS — a datapoint that optimizer-composition can be a first-class axis for coding-agent evaluation alongside model / harness / context-length choices.
  • [2026-07-21] ben (@contraben) open-sources 234-step Premiere Pro editing trajectories for computer-use agent SFT: Contra frames Adobe Premiere Pro as a first-class agent-tooling substrate: 234-step think-aloud trajectories tagged with Premiere MCP tool calls (alongside shortcut / menu / coordinate paths), extending the AgentNet schema to a creative-professional desktop app rather than a terminal / browser / IDE.
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 reports frontier-competitive results on DeepSWE (67.3 with mini-SWE-agent harness), Terminal-Bench 2.1, SWE Marathon, FrontierSWE, and KCB 2.0; the KimiCode harness is Moonshot’s own agent runtime and is now the reference K3 harness.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: AIDE² is a coding-agent-editing-a-coding-agent setup: the outer AIDEhuman rewrites the inner AIDE agent’s Python source over 100 steps, and the discovered AIDE85 exhibits two portable coding-agent design lessons the outer loop selected for — (1) a multi-armed-bandit-over-drafts + within-lineage-greedy search policy that beat MCTS on the same task set, and (2) aggressive per-operator context compression (16× against naive history concatenation) with the token budget reinvested as more search steps, inverting the “put as much context as possible in the LLM” default.
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 is pitched primarily as an agentic-coding model — trained for harness features (planning mode, goal conditioning, subagent delegation, context compaction), demoed in OpenCode, and quoted as competitive with leading alternatives on Meta’s internal coding bench.
  • [2026-07-09] Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase: Databricks publishes a benchmark from their own merged PRs against a multi-million-line polyglot codebase (Python/Go/TS/Scala), sealed against git-history exploitation and graded by hand-audited executable tests. Three key findings: (a) GLM 5.2 statistically ties Opus 4.8 in the top capability tier at 1.28vs1.28 vs 1.94/task — first published open-vs-Anthropic parity on a proprietary benchmark; (b) per-token pricing is a poor cost proxy — Sonnet 5 is 1.7× cheaper per token but 2.09/taskvsOpus′s2.09/task vs Opus's 1.94 (1.9× more tokens, 6 pts lower quality); (c) harness choice moves per-task cost by >2× at fixed model+quality — Pi sends ~3× less context per turn than Claude Code / Codex, arguing harness context-management policy is a first-class cost knob independent of model choice.
  • [2026-07-08] LLM-as-a-Verifier: A General-Purpose Verification Framework: First filed paper to hit top-tier agentic-SWE numbers without any verifier training: SWE-Bench Verified 78.2 (competitive with IQuest-Coder-V1 40B-Instruct 76.2 and Step 3.5 Flash 74.4) and Terminal-Bench V2 86.5 (well above Step 3.5 Flash’s TB2.0 51.0 and TMax-27B’s TB2.0 42.7) via a continuous logit-expectation verifier applied at inference time. Complements the training-heavy PR-mining + execution-grounded RL recipe (Qwen3-Coder-Next) by adding a training-free verification layer on top of any base LLM; the ranking algorithm over continuous verifier scores gives cost-efficient best-of-N without O(N2)O(N^2) pairwise passes.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Agent² RL-Bench is the direct successor to PostTrainBench, extending the benchmark from static SFT-friendly targets (GSM8K/HumanEval/AlpacaEval) to interactive rollout tasks (ALFWorld/WebShop/DeepSearchQA) under a 12h budget. Best result: Claude Code + Opus 4.6 drives Qwen2.5-7B-Instruct on ALFWorld from 4.85 → 95.52 (Free mode) and 93.28 (RL-only via SFT warm-up + GRPO with online rollouts); the RL-only case is the only online-RL success in the benchmark. Driver choice within fixed scaffolds swings ALFWorld improvement by up to +78pp on interactive tasks (task-dependent sensitivity 4-5× that of static tasks), and static-benchmark success is a poor predictor of interactive success — Claude Code’s ALFWorld improvement is ~35× its GSM8K improvement.
  • [2026-07-04] EdgeBench: Scaling Laws of Environment Learning: EdgeBench contributes 36 systems & SE tasks (13 publicly released) inside a 134-task ultra-long-horizon (12–72 h) benchmark scored on learning speed rather than final capability — a coding-agent evaluation surface that measures dynamics of iterative debug loops (247 submissions over 12 h in the published GPT-5.5 trace) rather than one-shot patch success.
  • [2026-06-30] Agents-A1: A 35B MoE Agentic Foundation Model: InternScience’s Agents-A1 — 35B-MoE Qwen3-derived open agentic foundation model trained on a “domain-grounded knowledge-action graph” with explicit cross-domain expertise transfer; ships SGLang/vLLM recipes with 262K context and an open eval framework.
  • [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent (Ai2-adjacent / Schmidt group) is the agentic-data sibling of OpenThoughts3 — 100+ controlled ablations over SFT-data pipeline stages, 100K-example stacked recipe, Qwen3-32B fine-tune reaches 44.8% across 7 agentic benchmarks (+3.9 pp over Nemotron-Terminal-32B at the same active parameter count). Frames the cross-benchmark generalization gap left open by SWE-Smith / SERA / Nemotron-Terminal single-benchmark targeting, and supplies the first controlled per-stage agentic-SFT ablation in the concept’s lineage.
  • [2026-06-24] TMax: A Simple Recipe for Terminal Agents: TMax (Ai2 + UW) — open recipe pairing TMax-15k (14.6K Docker-validated terminal RL envs, hardest released dataset at Pass@1 42% / Pass@8 53%) with a plain GRPO + DPPO + FP32 LM head training recipe; TMax-9B (Qwen3.5-9B base) hits 27.2 on Terminal Bench 2.0 — strongest open <10B under official settings, beating prior 32B agents — and TMax-27B hits 42.7, approaching 1T Kimi K2.5 (43.2); transfer is broad (SWE-Bench Verified 44.0→53.5, AIME terminal-agent harness 73.3→91.1, +9–15 pts across every unseen harness tested). Notably finds SFT before RL hurts heavily-post-trained Qwen3.5 (TB Lite 41.9→35.5 with their own SFT mixture) while helping the older Qwen3-8B, contradicting the standard SFT→RL prescription used by Qwen3-Coder-Next.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: ATE-Bench introduces the inverse of SWE-bench / HumanEval / MLE-bench: hold the agent fixed (Claude Code Opus 4.7 at xhigh effort) and vary the codebase across three real MoE training frameworks (Megatron-LM, TorchTitan, PithTrain) on 20 training-framework tasks across Q&A (12), Operate-and-Profile (4), and New-Feature (4 — port Differential Transformer, Dynamic MoE, MoBA, MoE++ end-to-end). The benchmark measures five effort dimensions (session duration, active GPU time, agent turns, per-turn context, output tokens) and finds the same agent uses up to 67% fewer turns and 64% less GPU time depending on framework design — isolating codebase compactness, Python-nativeness, and absence of implicit indirection as load-bearing factors in agent productivity.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 (Pro/Turbo two-tier) is the direct closed-source successor to Seed2.0; reports NL2Repo-Bench 47.0/43.7, SWE-Atlas 35.2/30.6, Terminal Bench 2.1 71.0/67.6, ProgramBench 0/1/50.3 (triplet) vs Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro — Pro trails Opus on long-horizon code (NL2Repo, SWE-Atlas) but matches Opus on terminal; the candid-gap framing from Seed2.0 (trails Claude on coding, trails Gemini on knowledge) carries forward.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE evaluates three coding agents — Codex / GPT-5.5, Claude Code / Opus 4.7, Kimi Code / K2.6 — head-to-head on a non-SWE-Bench axis: AutoEnvBench tracks per-agent research-progress curves over wall-clock time on Push-T (heuristic learning) and Pin Insertion (gradient-based learning), where the deliverable is a working dexterous manipulation policy rather than a unit-test pass-rate. First filed coding-agent comparison whose action surface is a real robot.
  • [2026-06-02] MAI-Thinking-1: Building a Hill-Climbing Machine: MAI-Thinking-1 reports 52.8% on SWE-Bench Pro and sustained log-linear SWE-Bench Verified gains over a thousands-of-steps agentic RL climb from a base model with no prior reasoning-trace exposure; an agentic-coding specialist is consolidated into a single final model.
  • [2026-06-04] NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: Nemotron 3 Ultra reports SWE-Bench Verified 69.7 (NVFP4) / 71.9 (BF16), Terminal Bench 2.1 53.9/56.4, TauBench v3 Average 70.3/70.9 — frontier-scale agentic post-training under FP4 quantization.
  • [2026-06-04] Nemotron 3 Ultra — 550B/55B-active hybrid Mamba-Transformer MoE open frontier model for long-running agents (NVIDIA AI): Nemotron 3 Ultra is explicitly positioned for long-horizon agent tasks (hundreds of tool-calling turns) with SFT + RL + Multi-teacher On-Policy Distillation and inference-time reasoning budget control.
  • [2026-06-01] MiniMax M3 announcement — first open-weights model combining coding, agentic, and natively-multimodal frontier capabilities: MiniMax M3 announcement claims 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench Hard, 74.2% MCP Atlas (numbers from launch tweet, no third-party verification yet).
  • [2026-05-23] GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement): OpenAI announces GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT, with GPT-5.4 also available in API + Codex on the same day. No benchmark numbers, system card, or training-recipe details in the tweet — filed as a tracker for the next closed-frontier datapoint above GPT-5.3-Codex. Continues the consolidation framing (“reasoning, coding, and agentic workflows into one frontier model”) that GPT-5.3-Codex established for this concept page.
  • [2026-05-23] Step 3.5 Flash: Step 3.5 Flash (11B-active / 196B-total sparse MoE, 3:1 SWA:Full hybrid attention, MTP-3 speculative decoding, 256K context) joins the open-MoE agentic-SWE Pareto frontier with SWE-bench Verified 74.4, Terminal-Bench 2.0 51.0, LiveCodeBench-V6 86.4, plus 100–300 tok/s typical decoding (350 tok/s peak) on Hopper. Two contributions this concept page hadn’t seen before: (a) inference-cost / throughput on the headline axis — its peer benchmark table normalizes estimated decoding cost at 128K context and places Step 3.5 Flash at 1.0× vs DeepSeek V3.2 6.0× / Kimi K2 Thinking 18.9× / GLM-4.7 18.9× / MiniMax M2.1 3.9× / MiMo-V2 Flash 1.2×, foregrounding inference efficiency as a Pareto axis; (b) MIS-PO as the RL post-training method — a binary-acceptance importance-sampling replacement for PPO/GRPO ratios, designed for off-policy drift on long trajectories (see Reasoning RL). Also notable: explicit Claude-Code compatibility as a deployment feature; edge-cloud agent collaboration with the on-device Step-GUI agent over GUI-MCP; runs locally on DGX Spark 128GB at ~20 tok/s via llama.cpp + INT4 GGUF + INT8 KV-cache.
  • [2026-05-23] Introducing GPT-5.3-Codex: OpenAI GPT-5.3-Codex (closed-weights, 2026-02-05 release) is the closed-source frontier reference for this cluster. Headline 56.8 on SWE-Bench Pro and “far exceeds prior SOTA” on Terminal-Bench 2.0. Two new framings worth tracking against the open recipes: (a) the benchmark quadruple expands to SWE-Bench Pro + Terminal-Bench 2.0 + OSWorld + GDPval, adding computer-use and professional-knowledge axes the open cluster has not yet covered; (b) “first model instrumental in creating itself” — early Codex checkpoints debugged the training run, monitored deployment, and analyzed alpha-test telemetry, extending the closed-loop-synthesis pattern from data-generation up to training-process automation. No training-recipe details disclosed. Also notable: first OpenAI launch classified High-capability on cybersecurity under the Preparedness Framework.
  • [2026-05-23] Qwen3-Coder-Next Technical Report: Qwen3-Coder-Next (80B-total / 3B-active MoE on hybrid-attention Qwen3-Next base) is positioned on the Pareto frontier of (active parameters, SWE-Bench Pro). Defines this concept page’s central recipe explicitly: ~800K verifiable-task synthesis (GitHub PR mining + bug injection into curated containerized repos), Kubernetes-based rollout infrastructure (MegaFlow on Argo), execution-grounded SFT filtering via Mini-SWE-agent user-simulator, four domain experts (SWE/QA/WebDev/UX) distilled back into a single model, and multi-format tool-chat training as a generalization mechanism against cross-scaffold transfer failure. Notable ablations: web-document reformatting (+8.7 EvalPlus / +12.3 MultiPL-E), within-vs-cross-scaffold mid-training scaling (within is monotone, cross-scaffold transfer is weak), Search-and-Replace FIM beats Chat-FIM at equivalent scale.
  • [2026-05-22] IQuest-Coder-V1 Model Family: IQuest-Coder-V1 (7B/14B/40B dense, plus a 40B-Loop variant with shared-parameter recurrent depth) trained under a “code-flow” paradigm conditioned on repository commit-level evolution rather than static snapshots. Post-training bifurcates into Instruct and Thinking tracks; the 40B-Loop-Thinking research prototype tests whether structural CoT (looped transformer) and procedural CoT (reasoning RL) are additive. 40B-Instruct reports 76.2 SWE-Bench Verified, 81.1 LiveCodeBench v6, 49.9 BigCodeBench. Reproducible eval harness ships under IQuest-Coder-Eval/SWE-Verified/ built on R2E-Gym.
  • [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: InfTool — closed-loop multi-agent tool-trajectory synthesizer (User Simulator + Tool-Calling Assistant + MCP Server) trained with GRPO + gated rewards. Headline: 32B base from 19.8% → 70.9% on BFCL with no human labels. Smaller-scale, narrower-target (function-calling vs SWE-Bench) instance of the same synthesize-verify-train closed loop that Qwen3-Coder-Next runs at industrial scale.

Open questions

Section titled “Open questions”
  • Within-vs-cross-scaffold mid-training scaling (Qwen3-Coder-Next Technical Report §3.1.2, Fig. 4) shows that one-scaffold training does not transfer to another. Qwen3-Coder-Next’s response is to train on a union of scaffolds simultaneously, but no filed paper measures whether the union recipe pays a tax vs the best per-scaffold specialist or vs a learned router across scaffolds.
  • Closed-loop self-synthesis (Qwen3-Coder-Next uses the prior Qwen3-Coder-480B-A35B-Instruct as teacher; InfTool re-injects the trained model into the next synthesis round) is the dominant data primitive. No filed paper isolates the contribution of the closed loop from the size of the executable-environment corpus, or measures distributional drift across synthesis rounds. GPT-5.3-Codex’s “model helps build itself” claim (Introducing GPT-5.3-Codex) extends the pattern one layer up to training-process automation but is undocumented at the recipe level.
  • Execution-grounded verification (“does the test pass after the patch?”) is the load-bearing gate, but Qwen3-Coder-Next flags that agents exploit superficial verification shortcuts and trains a separate detector + QA agent to filter them (Qwen3-Coder-Next Technical Report §2.1). The shortcut-exploitation failure mode is not characterized quantitatively in the filed material — what fraction of synthesized verifiers are non-functional, and what does the post-filtering rate look like?
  • MoE at low active ratios (3B/80B for Qwen3-Coder-Next, 10–37B active for peers, 11B active for Step 3.5 Flash) appears to dominate the open-agentic-SWE Pareto frontier vs dense models (IQuest-Coder-V1 7B/14B/40B). Is the MoE win attributable to active-parameter efficiency, to a specific routing inductive bias that benefits agentic trajectories, or to confounded co-scaling of training tokens / data quality?
  • Context-length extension (32K → 262K in Qwen3-Coder-Next, native 128K in IQuest-Coder-V1, 256K via 3:1 SWA hybrid in Step 3.5 Flash) is treated as a prerequisite for repository-level pretraining and multi-turn agentic trajectories, but no paper isolates how much of the SWE-Bench gain comes from the long context vs the data recipe vs the architecture. Related to the “interaction depth is an independent scaling axis” claim from MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling.
  • BFCL-targeted closed-loop synthesis (InfTool) and SWE-Bench-targeted PR mining (Qwen3-Coder-Next) are evaluated on benchmarks that are in-distribution for the synthesizers. Held-out tool / SWE distributions (non-BFCL function-calling, repos not in the GitHub mining window, languages outside the 9-language synthesis subset) are not measured.
  • Computer-use (OSWorld) and professional-knowledge (GDPval) are now part of the benchmark quadruple at the closed-source frontier (Introducing GPT-5.3-Codex) but absent from open filings. Open question: do the open SWE-recipes (PR mining + execution-grounded RL + multi-format tool chat) transfer to OSWorld-style visual desktop tasks, or is computer-use a separately-engineered axis? Step 3.5 Flash’s edge-cloud collaboration with Step-GUI on AndroidDaily Hard (Step 3.5 Flash §Edge–Cloud Collaboration) is the closest open-side datapoint but tests orchestration over a separate on-device GUI agent rather than direct computer-use of the host LLM.
  • Inference-cost / decoding-throughput is starting to enter the headline benchmark axis (Step 3.5 Flash’s normalized 1.0×–18.9× cost-ratio column). No prior paper on this concept page reports inference cost alongside SWE-Bench / Terminal-Bench numbers; whether this becomes a standard reporting axis is open.
  • GPT-5.4 (GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement)) ships with no benchmark disclosure in the launch tweet; whether OpenAI’s subsequent system card / blog publishes the same SWE-Bench Pro + Terminal-Bench 2.0 + OSWorld + GDPval quadruple as 5.3-Codex did, or shifts to a new headline benchmark, is open.

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-02

Section titled “2025-02”

Unknown date

Section titled “Unknown date”