Skip to content
Bud's Wiki

Agentic Software Engineering

TL;DR

Section titled “TL;DR”

Agentic SWE is the post-training regime that turns a code LLM into a long-horizon coding agent: SWE-Bench(-Verified/-Multilingual/-Pro), Terminal-Bench, BigCodeBench, and Aider are the de facto evaluation suite, and the common recipe is large-scale synthesis of verifiable executable tasks (PR mining + bug injection + Docker environment + automated verifier) paired with multi-stage post-training (SFT → domain experts → distillation → RL with environment feedback). Filed papers converge on three load-bearing pieces: (a) execution-grounded reward / verification (not static review), (b) cross-scaffold tool-call format diversity at training time, and (c) closed-loop synthesis where the trained model is fed back into the next round of trajectory generation.

Key claims

Section titled “Key claims”
  • SWE-Bench Verified / Multilingual / Pro and Terminal-Bench are the common evaluation axis across the cluster, with the latest open models clustering on a Pareto frontier in (active parameters, SWE-Bench Pro) space — Qwen3-Coder-Next 80B-total/3B-active reaches 44.3 SWE-Bench Pro, DeepSeek-V3.2 (37B-active) 40.9, GLM-4.7 (32B-active) 40.6, MiniMax M2.1 (10B-active) 34.6, Kimi-K2.5 in the same neighborhood (Qwen3-Coder-Next Technical Report Fig. 1, Fig. 2). IQuest-Coder-V1 40B-Instruct reports 76.2 on SWE-Bench Verified (IQuest-Coder-V1 Model Family Highlights). Closed-source GPT-5.3-Codex sets the upper bound on this same axis at 56.8 SWE-Bench Pro per its release post (Introducing GPT-5.3-Codex). Step 3.5 Flash (11B-active / 196B-total) reports SWE-bench Verified 74.4 and a leading Terminal-Bench 2.0 51.0 against the same open-MoE peer set (Step 3.5 Flash Benchmark table).
  • SWE-Bench Pro is the emerging consensus benchmark replacing SWE-Bench Verified as the headline metric in this cluster, motivated by being multi-language (4 langs), more contamination-resistant, and more industry-relevant (Introducing GPT-5.3-Codex; Qwen3-Coder-Next Technical Report Fig. 1).
  • Large-scale synthesis of verifiable executable tasks is the dominant data-generation pattern: Qwen3-Coder-Next synthesizes ~800K SWE tasks across 9+ languages by mining GitHub PRs (decomposed into buggy state + fix + test patch + Docker verifier) and by injecting controlled bugs into curated containerized repos seeded from SWE-Smith / SWE-Flow / SWE-Rebench / Multi-SWE-RL (Qwen3-Coder-Next Technical Report §2.1). InfTool extends the same idea to tool-use with a three-agent role-play synthesizer (User Simulator + Tool-Calling Assistant + MCP Server) closing the loop into BFCL-tagged trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
  • Verification gates are explicitly designed against agents that “exploit superficial verification shortcuts”: Qwen3-Coder-Next runs a specialized environment-building agent + automated detector for non-functional verifiers + a separate quality-assurance agent that strips ambiguous tasks before inclusion (Qwen3-Coder-Next Technical Report §2.1). InfTool applies GRPO-with-gated-rewards over verified trajectories (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
  • Execution-grounded SFT filtering replaces static review at the SFT stage: Qwen3-Coder-Next deploys a Mini-SWE-agent user-simulator that executes the proposed assistant solution and gates training pairs on whether the execution meaningfully advances the user task (Qwen3-Coder-Next Technical Report §4.1) — the SFT-stage analog of the RL-stage executable reward.
  • Cross-scaffold tool-call format diversity at training time is treated as a generalization recipe, not a deployment afterthought: Qwen3-Coder-Next explicitly trains on diverse tool chat templates (varying tool-set definition, invocation, and response-wrapping syntax across Cline, Qoder, OpenCode, Claude Code, Qwen Code) with rule-based tool-call-format validation as a training signal (Qwen3-Coder-Next Technical Report §4.2.2, Fig. 5). This is motivated by an empirical mid-training scaling finding: within-scaffold token scaling is monotone but cross-scaffold transfer is weak — OpenHands-trained checkpoints transfer poorly to SWE-Agent (Qwen3-Coder-Next Technical Report §3.1.2, Fig. 4). Step 3.5 Flash makes a similar claim from the deployment side — Claude-Code compatibility is foregrounded as a release feature (Step 3.5 Flash §Agentic Coding).
  • Domain-expert distillation is becoming a standard post-training step: Qwen3-Coder-Next trains four expert models (SWE / QA / WebDev / UX) from the post-SFT checkpoint, each with its own data recipe — WebDev uses a Playwright/Vite rendering pipeline with VLM static evaluation + DOM dynamic interaction evaluation — then distills them back into a single unified model (Qwen3-Coder-Next Technical Report §4.2). IQuest-Coder-V1 splits post-training into Instruct (instruction-following) and Thinking (reasoning-driven RL) tracks at every size (IQuest-Coder-V1 Model Family Highlights). Closed-source GPT-5.3-Codex frames its release as the consolidation of separate coding (5.2-Codex) and reasoning (5.2) lines into a single unified model (Introducing GPT-5.3-Codex), suggesting the open trend toward distill-back-to-single-model is mirrored at the closed-source frontier. The subsequent GPT-5.4 launch tweet extends the same consolidation framing — “reasoning, coding, and agentic workflows into one frontier model” — across ChatGPT (Thinking, Pro), API, and Codex on the same day (GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement)).
  • Context length extension is co-required: Qwen3-Coder-Next extends from 32,768 to 262,144 tokens for repository-level pretraining, motivated explicitly by cross-file dependencies and multi-turn agentic trajectories (Qwen3-Coder-Next Technical Report §3.1.1). IQuest-Coder-V1 ships native 128K context across all sizes without rope-scaling tricks (IQuest-Coder-V1 Model Family Model Overview). Step 3.5 Flash supports 256K context via 3:1 SWA:Full hybrid attention (Step 3.5 Flash §Architecture).
  • Closed-loop synthesis (trained model → next round of synthesizer) is the data-generation primitive: InfTool’s 19.8% → 70.9% BFCL improvement over closed-loop iterations is the cleanest filed datapoint (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract). Qwen3-Coder-Next uses the prior model Qwen3-Coder-480B-A35B-Instruct as the teacher for trajectory and QA data (Qwen3-Coder-Next Technical Report §3.1.2). The recipe is now the same shape at two very different scales. GPT-5.3-Codex extends the pattern one layer up: early checkpoints assisted the training process itself (debugging the training run, deployment monitoring, alpha-test telemetry analysis), described as “the first model that was instrumental in creating itself” (Introducing GPT-5.3-Codex).
  • MoE with low activation ratios is the architectural common factor among the latest agentic-SWE models: Qwen3-Coder-Next 80B-total / 3B-active, DeepSeek-V3.2 37B-active, GLM-4.7 32B-active, MiniMax M2.1 10B-active (Qwen3-Coder-Next Technical Report Fig. 1). Step 3.5 Flash adds another datapoint at 11B-active / 196B-total with the lowest normalized inference cost in its peer table (Step 3.5 Flash Benchmark table). IQuest-Coder-V1 is the dense outlier in the cluster (7B/14B/40B) (IQuest-Coder-V1 Model Family Model Overview).
  • The benchmark frontier is shifting from pure-coding axes to a four-benchmark agentic quadruple — SWE-Bench Pro (coding), Terminal-Bench 2.0 (terminal skills), OSWorld (computer-use), GDPval (professional knowledge work) — under the framing that a coding agent is becoming “a general-purpose agent that can reason, build, and execute” (Introducing GPT-5.3-Codex). No filed open model yet reports the full quadruple; the open cluster currently covers SWE-Bench-* + Terminal-Bench but not OSWorld or GDPval.
  • Decoding-throughput / inference-cost is increasingly co-reported with capability metrics on this Pareto. Step 3.5 Flash reports 100–300 tok/s typical throughput (peak 350 tok/s for single-stream coding) on Hopper GPUs via MTP-3 + sparse-active MoE, and normalizes its peer table to estimated decoding cost at 128K context — claiming 1.0× vs DeepSeek V3.2 6.0×, Kimi K2 Thinking / GLM-4.7 18.9×, MiniMax M2.1 3.9×, MiMo-V2 Flash 1.2× (Step 3.5 Flash Benchmark table). First filed agentic-SWE entry that puts inference cost on the headline axis alongside SWE-Bench Verified / Terminal-Bench scores.

Recent contributions

Section titled “Recent contributions”
  • [2026-07-16] Kimi K3 — Open Frontier Intelligence (2.8T MoE with KDA + AttnRes): Kimi K3 reports frontier-competitive results on DeepSWE (67.3 with mini-SWE-agent harness), Terminal-Bench 2.1, SWE Marathon, FrontierSWE, and KCB 2.0; the KimiCode harness is Moonshot’s own agent runtime and is now the reference K3 harness.
  • [2026-07-14] AIDE²: First Evidence of Recursive Self-Improvement: AIDE² is a coding-agent-editing-a-coding-agent setup: the outer AIDEhuman rewrites the inner AIDE agent’s Python source over 100 steps, and the discovered AIDE85 exhibits two portable coding-agent design lessons the outer loop selected for — (1) a multi-armed-bandit-over-drafts + within-lineage-greedy search policy that beat MCTS on the same task set, and (2) aggressive per-operator context compression (16× against naive history concatenation) with the token budget reinvested as more search steps, inverting the “put as much context as possible in the LLM” default.
  • [2026-07-09] Introducing Muse Spark 1.1: Muse Spark 1.1 is pitched primarily as an agentic-coding model — trained for harness features (planning mode, goal conditioning, subagent delegation, context compaction), demoed in OpenCode, and quoted as competitive with leading alternatives on Meta’s internal coding bench.
  • [2026-07-09] Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase: Databricks publishes a benchmark from their own merged PRs against a multi-million-line polyglot codebase (Python/Go/TS/Scala), sealed against git-history exploitation and graded by hand-audited executable tests. Three key findings: (a) GLM 5.2 statistically ties Opus 4.8 in the top capability tier at 1.28vs1.28 vs 1.94/task — first published open-vs-Anthropic parity on a proprietary benchmark; (b) per-token pricing is a poor cost proxy — Sonnet 5 is 1.7× cheaper per token but 2.09/taskvsOpus′s2.09/task vs Opus's 1.94 (1.9× more tokens, 6 pts lower quality); (c) harness choice moves per-task cost by >2× at fixed model+quality — Pi sends ~3× less context per turn than Claude Code / Codex, arguing harness context-management policy is a first-class cost knob independent of model choice.
  • [2026-07-08] LLM-as-a-Verifier: A General-Purpose Verification Framework: First filed paper to hit top-tier agentic-SWE numbers without any verifier training: SWE-Bench Verified 78.2 (competitive with IQuest-Coder-V1 40B-Instruct 76.2 and Step 3.5 Flash 74.4) and Terminal-Bench V2 86.5 (well above Step 3.5 Flash’s TB2.0 51.0 and TMax-27B’s TB2.0 42.7) via a continuous logit-expectation verifier applied at inference time. Complements the training-heavy PR-mining + execution-grounded RL recipe (Qwen3-Coder-Next) by adding a training-free verification layer on top of any base LLM; the ranking algorithm over continuous verifier scores gives cost-efficient best-of-N without O(N2)O(N^2) pairwise passes.
  • [2026-07-05] Agent² RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?: Agent² RL-Bench is the direct successor to PostTrainBench, extending the benchmark from static SFT-friendly targets (GSM8K/HumanEval/AlpacaEval) to interactive rollout tasks (ALFWorld/WebShop/DeepSearchQA) under a 12h budget. Best result: Claude Code + Opus 4.6 drives Qwen2.5-7B-Instruct on ALFWorld from 4.85 → 95.52 (Free mode) and 93.28 (RL-only via SFT warm-up + GRPO with online rollouts); the RL-only case is the only online-RL success in the benchmark. Driver choice within fixed scaffolds swings ALFWorld improvement by up to +78pp on interactive tasks (task-dependent sensitivity 4-5× that of static tasks), and static-benchmark success is a poor predictor of interactive success — Claude Code’s ALFWorld improvement is ~35× its GSM8K improvement.
  • [2026-07-04] EdgeBench: Scaling Laws of Environment Learning: EdgeBench contributes 36 systems & SE tasks (13 publicly released) inside a 134-task ultra-long-horizon (12–72 h) benchmark scored on learning speed rather than final capability — a coding-agent evaluation surface that measures dynamics of iterative debug loops (247 submissions over 12 h in the published GPT-5.5 trace) rather than one-shot patch success.
  • [2026-06-30] Agents-A1: A 35B MoE Agentic Foundation Model: InternScience’s Agents-A1 — 35B-MoE Qwen3-derived open agentic foundation model trained on a “domain-grounded knowledge-action graph” with explicit cross-domain expertise transfer; ships SGLang/vLLM recipes with 262K context and an open eval framework.
  • [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent (Ai2-adjacent / Schmidt group) is the agentic-data sibling of OpenThoughts3 — 100+ controlled ablations over SFT-data pipeline stages, 100K-example stacked recipe, Qwen3-32B fine-tune reaches 44.8% across 7 agentic benchmarks (+3.9 pp over Nemotron-Terminal-32B at the same active parameter count). Frames the cross-benchmark generalization gap left open by SWE-Smith / SERA / Nemotron-Terminal single-benchmark targeting, and supplies the first controlled per-stage agentic-SFT ablation in the concept’s lineage.
  • [2026-06-24] TMax: A Simple Recipe for Terminal Agents: TMax (Ai2 + UW) — open recipe pairing TMax-15k (14.6K Docker-validated terminal RL envs, hardest released dataset at Pass@1 42% / Pass@8 53%) with a plain GRPO + DPPO + FP32 LM head training recipe; TMax-9B (Qwen3.5-9B base) hits 27.2 on Terminal Bench 2.0 — strongest open <10B under official settings, beating prior 32B agents — and TMax-27B hits 42.7, approaching 1T Kimi K2.5 (43.2); transfer is broad (SWE-Bench Verified 44.0→53.5, AIME terminal-agent harness 73.3→91.1, +9–15 pts across every unseen harness tested). Notably finds SFT before RL hurts heavily-post-trained Qwen3.5 (TB Lite 41.9→35.5 with their own SFT mixture) while helping the older Qwen3-8B, contradicting the standard SFT→RL prescription used by Qwen3-Coder-Next.
  • [2026-06-23] PithTrain: A Compact and Agent-Native MoE Training System: ATE-Bench introduces the inverse of SWE-bench / HumanEval / MLE-bench: hold the agent fixed (Claude Code Opus 4.7 at xhigh effort) and vary the codebase across three real MoE training frameworks (Megatron-LM, TorchTitan, PithTrain) on 20 training-framework tasks across Q&A (12), Operate-and-Profile (4), and New-Feature (4 — port Differential Transformer, Dynamic MoE, MoBA, MoE++ end-to-end). The benchmark measures five effort dimensions (session duration, active GPU time, agent turns, per-turn context, output tokens) and finds the same agent uses up to 67% fewer turns and 64% less GPU time depending on framework design — isolating codebase compactness, Python-nativeness, and absence of implicit indirection as load-bearing factors in agent productivity.
  • [2026-06-23] Seed2.1: A Next-Generation Agent for Real-World Productivity: Seed2.1 (Pro/Turbo two-tier) is the direct closed-source successor to Seed2.0; reports NL2Repo-Bench 47.0/43.7, SWE-Atlas 35.2/30.6, Terminal Bench 2.1 71.0/67.6, ProgramBench 0/1/50.3 (triplet) vs Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro — Pro trails Opus on long-horizon code (NL2Repo, SWE-Atlas) but matches Opus on terminal; the candid-gap framing from Seed2.0 (trails Claude on coding, trails Gemini on knowledge) carries forward.
  • [2026-06-16] ENPIRE: Agentic Robot Policy Self-Improvement in the Real World: ENPIRE evaluates three coding agents — Codex / GPT-5.5, Claude Code / Opus 4.7, Kimi Code / K2.6 — head-to-head on a non-SWE-Bench axis: AutoEnvBench tracks per-agent research-progress curves over wall-clock time on Push-T (heuristic learning) and Pin Insertion (gradient-based learning), where the deliverable is a working dexterous manipulation policy rather than a unit-test pass-rate. First filed coding-agent comparison whose action surface is a real robot.
  • [2026-06-02] MAI-Thinking-1: Building a Hill-Climbing Machine: MAI-Thinking-1 reports 52.8% on SWE-Bench Pro and sustained log-linear SWE-Bench Verified gains over a thousands-of-steps agentic RL climb from a base model with no prior reasoning-trace exposure; an agentic-coding specialist is consolidated into a single final model.
  • [2026-06-04] NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: Nemotron 3 Ultra reports SWE-Bench Verified 69.7 (NVFP4) / 71.9 (BF16), Terminal Bench 2.1 53.9/56.4, TauBench v3 Average 70.3/70.9 — frontier-scale agentic post-training under FP4 quantization.
  • [2026-06-04] Nemotron 3 Ultra — 550B/55B-active hybrid Mamba-Transformer MoE open frontier model for long-running agents (NVIDIA AI): Nemotron 3 Ultra is explicitly positioned for long-horizon agent tasks (hundreds of tool-calling turns) with SFT + RL + Multi-teacher On-Policy Distillation and inference-time reasoning budget control.
  • [2026-06-01] MiniMax M3 announcement — first open-weights model combining coding, agentic, and natively-multimodal frontier capabilities: MiniMax M3 announcement claims 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench Hard, 74.2% MCP Atlas (numbers from launch tweet, no third-party verification yet).
  • [2026-05-23] GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement): OpenAI announces GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT, with GPT-5.4 also available in API + Codex on the same day. No benchmark numbers, system card, or training-recipe details in the tweet — filed as a tracker for the next closed-frontier datapoint above GPT-5.3-Codex. Continues the consolidation framing (“reasoning, coding, and agentic workflows into one frontier model”) that GPT-5.3-Codex established for this concept page.
  • [2026-05-23] Step 3.5 Flash: Step 3.5 Flash (11B-active / 196B-total sparse MoE, 3:1 SWA:Full hybrid attention, MTP-3 speculative decoding, 256K context) joins the open-MoE agentic-SWE Pareto frontier with SWE-bench Verified 74.4, Terminal-Bench 2.0 51.0, LiveCodeBench-V6 86.4, plus 100–300 tok/s typical decoding (350 tok/s peak) on Hopper. Two contributions this concept page hadn’t seen before: (a) inference-cost / throughput on the headline axis — its peer benchmark table normalizes estimated decoding cost at 128K context and places Step 3.5 Flash at 1.0× vs DeepSeek V3.2 6.0× / Kimi K2 Thinking 18.9× / GLM-4.7 18.9× / MiniMax M2.1 3.9× / MiMo-V2 Flash 1.2×, foregrounding inference efficiency as a Pareto axis; (b) MIS-PO as the RL post-training method — a binary-acceptance importance-sampling replacement for PPO/GRPO ratios, designed for off-policy drift on long trajectories (see Reasoning RL). Also notable: explicit Claude-Code compatibility as a deployment feature; edge-cloud agent collaboration with the on-device Step-GUI agent over GUI-MCP; runs locally on DGX Spark 128GB at ~20 tok/s via llama.cpp + INT4 GGUF + INT8 KV-cache.
  • [2026-05-23] Introducing GPT-5.3-Codex: OpenAI GPT-5.3-Codex (closed-weights, 2026-02-05 release) is the closed-source frontier reference for this cluster. Headline 56.8 on SWE-Bench Pro and “far exceeds prior SOTA” on Terminal-Bench 2.0. Two new framings worth tracking against the open recipes: (a) the benchmark quadruple expands to SWE-Bench Pro + Terminal-Bench 2.0 + OSWorld + GDPval, adding computer-use and professional-knowledge axes the open cluster has not yet covered; (b) “first model instrumental in creating itself” — early Codex checkpoints debugged the training run, monitored deployment, and analyzed alpha-test telemetry, extending the closed-loop-synthesis pattern from data-generation up to training-process automation. No training-recipe details disclosed. Also notable: first OpenAI launch classified High-capability on cybersecurity under the Preparedness Framework.
  • [2026-05-23] Qwen3-Coder-Next Technical Report: Qwen3-Coder-Next (80B-total / 3B-active MoE on hybrid-attention Qwen3-Next base) is positioned on the Pareto frontier of (active parameters, SWE-Bench Pro). Defines this concept page’s central recipe explicitly: ~800K verifiable-task synthesis (GitHub PR mining + bug injection into curated containerized repos), Kubernetes-based rollout infrastructure (MegaFlow on Argo), execution-grounded SFT filtering via Mini-SWE-agent user-simulator, four domain experts (SWE/QA/WebDev/UX) distilled back into a single model, and multi-format tool-chat training as a generalization mechanism against cross-scaffold transfer failure. Notable ablations: web-document reformatting (+8.7 EvalPlus / +12.3 MultiPL-E), within-vs-cross-scaffold mid-training scaling (within is monotone, cross-scaffold transfer is weak), Search-and-Replace FIM beats Chat-FIM at equivalent scale.
  • [2026-05-22] IQuest-Coder-V1 Model Family: IQuest-Coder-V1 (7B/14B/40B dense, plus a 40B-Loop variant with shared-parameter recurrent depth) trained under a “code-flow” paradigm conditioned on repository commit-level evolution rather than static snapshots. Post-training bifurcates into Instruct and Thinking tracks; the 40B-Loop-Thinking research prototype tests whether structural CoT (looped transformer) and procedural CoT (reasoning RL) are additive. 40B-Instruct reports 76.2 SWE-Bench Verified, 81.1 LiveCodeBench v6, 49.9 BigCodeBench. Reproducible eval harness ships under IQuest-Coder-Eval/SWE-Verified/ built on R2E-Gym.
  • [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: InfTool — closed-loop multi-agent tool-trajectory synthesizer (User Simulator + Tool-Calling Assistant + MCP Server) trained with GRPO + gated rewards. Headline: 32B base from 19.8% → 70.9% on BFCL with no human labels. Smaller-scale, narrower-target (function-calling vs SWE-Bench) instance of the same synthesize-verify-train closed loop that Qwen3-Coder-Next runs at industrial scale.

Open questions

Section titled “Open questions”
  • Within-vs-cross-scaffold mid-training scaling (Qwen3-Coder-Next Technical Report §3.1.2, Fig. 4) shows that one-scaffold training does not transfer to another. Qwen3-Coder-Next’s response is to train on a union of scaffolds simultaneously, but no filed paper measures whether the union recipe pays a tax vs the best per-scaffold specialist or vs a learned router across scaffolds.
  • Closed-loop self-synthesis (Qwen3-Coder-Next uses the prior Qwen3-Coder-480B-A35B-Instruct as teacher; InfTool re-injects the trained model into the next synthesis round) is the dominant data primitive. No filed paper isolates the contribution of the closed loop from the size of the executable-environment corpus, or measures distributional drift across synthesis rounds. GPT-5.3-Codex’s “model helps build itself” claim (Introducing GPT-5.3-Codex) extends the pattern one layer up to training-process automation but is undocumented at the recipe level.
  • Execution-grounded verification (“does the test pass after the patch?”) is the load-bearing gate, but Qwen3-Coder-Next flags that agents exploit superficial verification shortcuts and trains a separate detector + QA agent to filter them (Qwen3-Coder-Next Technical Report §2.1). The shortcut-exploitation failure mode is not characterized quantitatively in the filed material — what fraction of synthesized verifiers are non-functional, and what does the post-filtering rate look like?
  • MoE at low active ratios (3B/80B for Qwen3-Coder-Next, 10–37B active for peers, 11B active for Step 3.5 Flash) appears to dominate the open-agentic-SWE Pareto frontier vs dense models (IQuest-Coder-V1 7B/14B/40B). Is the MoE win attributable to active-parameter efficiency, to a specific routing inductive bias that benefits agentic trajectories, or to confounded co-scaling of training tokens / data quality?
  • Context-length extension (32K → 262K in Qwen3-Coder-Next, native 128K in IQuest-Coder-V1, 256K via 3:1 SWA hybrid in Step 3.5 Flash) is treated as a prerequisite for repository-level pretraining and multi-turn agentic trajectories, but no paper isolates how much of the SWE-Bench gain comes from the long context vs the data recipe vs the architecture. Related to the “interaction depth is an independent scaling axis” claim from MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling.
  • BFCL-targeted closed-loop synthesis (InfTool) and SWE-Bench-targeted PR mining (Qwen3-Coder-Next) are evaluated on benchmarks that are in-distribution for the synthesizers. Held-out tool / SWE distributions (non-BFCL function-calling, repos not in the GitHub mining window, languages outside the 9-language synthesis subset) are not measured.
  • Computer-use (OSWorld) and professional-knowledge (GDPval) are now part of the benchmark quadruple at the closed-source frontier (Introducing GPT-5.3-Codex) but absent from open filings. Open question: do the open SWE-recipes (PR mining + execution-grounded RL + multi-format tool chat) transfer to OSWorld-style visual desktop tasks, or is computer-use a separately-engineered axis? Step 3.5 Flash’s edge-cloud collaboration with Step-GUI on AndroidDaily Hard (Step 3.5 Flash §Edge–Cloud Collaboration) is the closest open-side datapoint but tests orchestration over a separate on-device GUI agent rather than direct computer-use of the host LLM.
  • Inference-cost / decoding-throughput is starting to enter the headline benchmark axis (Step 3.5 Flash’s normalized 1.0×–18.9× cost-ratio column). No prior paper on this concept page reports inference cost alongside SWE-Bench / Terminal-Bench numbers; whether this becomes a standard reporting axis is open.
  • GPT-5.4 (GPT-5.4 Thinking and GPT-5.4 Pro rolling out in ChatGPT (product announcement)) ships with no benchmark disclosure in the launch tweet; whether OpenAI’s subsequent system card / blog publishes the same SWE-Bench Pro + Terminal-Bench 2.0 + OSWorld + GDPval quadruple as 5.3-Codex did, or shifts to a new headline benchmark, is open.

Papers

Section titled “Papers”

2026-11

Section titled “2026-11”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-02

Section titled “2025-02”

Unknown date

Section titled “Unknown date”