Step 3.5 Flash

TL;DR

Step 3.5 Flash is StepFun’s open-weights sparse MoE LLM positioned for high-throughput agentic and coding tasks: 196B total / 11B active parameters, hybrid attention (3 sliding-window layers per 1 full-attention layer at 256K context), 3-way Multi-Token Prediction for parallel speculative decoding, and a 100–300 tok/s typical decoding throughput on Hopper GPUs (peaking at 350 tok/s). Headline benchmark numbers position it at the open frontier: 74.4 on SWE-bench Verified, 51.0 on Terminal-Bench 2.0, 97.3 on AIME 2025, and 65.3 on ResearchRubrics. Notable RL contribution: MIS-PO (Metropolis Independence Sampling Filtered Policy Optimization), a GRPO/PPO-family replacement that uses importance-sampling ratios as a binary acceptance filter instead of as continuous gradient weights — designed to handle off-policy drift on long reasoning trajectories without aggressive clipping.

Key claims

Sparse MoE with 11B activated of 196B total delivers the headline efficiency story; the first few layers are kept dense for “high intelligence density” while later layers use MoE [§Architecture].
Hybrid attention layout: 3:1 SWA-to-Full ratio. SWA is preferred over linear-attention alternatives specifically to preserve compatibility with MTP-3 speculative decoding (parallel verification of multiple token hypotheses per pass) [§Architecture].
SWA layers use an augmented query-head count (64 → 96) without expanding the KV cache; cost stays constant in attention-window size, so the modification adds expressiveness without a long-context penalty [§Architecture].
Head-wise Gated Attention is introduced as an input-dependent attention sink — a learned dynamic modulator on attention output that preserves numerical stability with negligible overhead [§Architecture].
MIS-PO (Metropolis Independence Sampling Filtered Policy Optimization) replaces importance-sampling ratios as continuous gradient weights (as in PPO) with a binary acceptance criterion: trajectories whose likelihood deviates too far between the inference and training policies are excluded from optimization; accepted samples are treated as on-policy [§Scalable RL].
Truncation-aware value bootstrapping prevents long reasoning trajectories from being penalized when hitting context limits, and routing-confidence monitoring is reported as a practical RL stability signal for MoE policies [§Scalable RL].
Inference cost vs peers (estimated, 128K context, Hopper, normalized to 1.0× for Step 3.5 Flash): DeepSeek V3.2 6.0×, Kimi K2 Thinking 18.9×, GLM-4.7 18.9×, MiniMax M2.1 3.9×, MiMo-V2 Flash 1.2× [Benchmark table].
Agentic / browsing benchmarks (selected): τ²-Bench 88.2, BrowseComp w/ Context Manager 69.0, GAIA (no file) 84.5, xbench-DeepSearch (2025.05) 83.7, ResearchRubrics 65.3 — claimed best-in-class across the open-MoE cluster [Benchmark table].
Coding / agentic-SWE benchmarks: SWE-bench Verified 74.4, Terminal-Bench 2.0 51.0, LiveCodeBench-V6 86.4; reports Claude-Code compatibility and a 39.58% score on a 50-task internal Internet-backend data-analysis benchmark [§Agentic Coding, Benchmark table].
Reasoning benchmarks with Python-tool-augmented CoT: AIME 2025 99.8, HMMT 2025 (Nov.) 98.0, IMOAnswerBench 86.7, ARC-AGI-1 56.5 [§Tool-Augmented Reasoning Performance].
Local deployment claim: runs on NVIDIA DGX Spark 128GB via llama.cpp at ~20 tok/s with INT4 GGUF weights and INT8 KV-cache quantization, with the full 256K context window [§Architecture].
The model serves as the cloud-side host agent in an edge–cloud collaboration pattern with the on-device Step-GUI agent (via GUI-MCP) on Android, evaluated on AndroidDaily Hard with a reported gain over single-agent Step-GUI [§Edge–Cloud Collaboration].
Reliability/interaction evaluations: on a 74-prompt internal Proactive Intent Clarification benchmark the model asks targeted clarifying questions instead of assuming; on a 500-prompt bilingual advisory benchmark it scores 70.5% average vs GPT-5.2 77.8 / Gemini 3.0 Pro 70.6 / Claude Opus 4.5 68.5 [§Reliability in Interaction].

Method

The architecture is a sparse-MoE decoder-only LLM (196B total / 11B activated) with a hybrid attention layout interleaving Sliding-Window Attention and Full Attention at a 3:1 ratio across depth. The first few layers are dense (no MoE) for higher per-token compute at low layers. To make long-context cheap, SWA layers use 96 query heads (vs 64 in the base config) — adding representational capacity without growing the KV cache, since SWA’s attention window is fixed regardless of total sequence length. Head-wise Gated Attention is added as an input-dependent gate that modulates attention output, framed as an “explicit attention sink” mechanism for numerical stability. 3-way Multi-Token Prediction heads run in parallel with the primary output, enabling parallel verification of multiple token hypotheses per forward pass — picked specifically to be compatible with SWA (which linear-attention alternatives would have broken).

Post-training centers on a scalable RL framework. The core RL contribution is MIS-PO: instead of multiplying gradients by the (inference-policy / training-policy) importance ratio as PPO/GRPO do, MIS-PO converts the ratio to a binary acceptance decision — samples that are too off-distribution are discarded entirely; the rest are treated as on-policy. Two auxiliary pieces support long-trajectory stability: truncation-aware value bootstrapping (so trajectories that hit context limits aren’t incorrectly penalized) and routing-confidence monitoring (as a stability signal for the MoE policy under RL). The pipeline reports gains across math, coding, and tool-use under large-scale, off-policy training.

Results

Coding / SWE: SWE-bench Verified 74.4 (vs DeepSeek V3.2 73.1, Kimi K2.5 76.8, GLM-4.7 73.8, MiniMax M2.1 74.0, MiMo-V2 Flash 73.4); Terminal-Bench 2.0 51.0 (vs Kimi K2.5 50.8, MiniMax M2.1 47.9, DeepSeek V3.2 46.4); LiveCodeBench-V6 86.4 [Benchmark table].
Reasoning: AIME 2025 97.3, HMMT 2025 (Feb.) 98.4, HMMT 2025 (Nov.) 94.0, IMOAnswerBench 85.4 — top-of-cluster numbers vs the open-MoE peer set [Benchmark table].
Tool-augmented reasoning (Python-in-CoT): AIME 2025 99.8, HMMT 2025 (Nov.) 98.0, IMOAnswerBench 86.7, ARC-AGI-1 56.5 [§Tool-Augmented Reasoning Performance].
Agentic / browsing: τ²-Bench 88.2, BrowseComp 51.6 / 69.0 w/ Context Manager, BrowseComp-ZH 66.9 / 73.7 w/ Context Manager, GAIA (no file) 84.5, xbench-DeepSearch (2025.05) 83.7, ResearchRubrics 65.3 — claimed to compete with OpenAI / Gemini Deep Research at substantially higher inference efficiency [Benchmark table, §Deep Research].
Decoding throughput: 100–300 tok/s typical, peaking at 350 tok/s on Hopper GPUs for single-stream coding; 1.0× normalized inference cost at 128K context, vs 1.2× / 3.9× / 6.0× / 18.9× / 18.9× for MiMo-V2 Flash / MiniMax M2.1 / DeepSeek V3.2 / Kimi K2 Thinking / GLM-4.7 [Benchmark table].
Local: ~20 tok/s with full 256K context on NVIDIA DGX Spark 128GB via llama.cpp + INT4 GGUF + INT8 KV-cache [§Architecture].
Acknowledged weaknesses: longer generation trajectories than Gemini 3.0 Pro to reach comparable quality; reduced stability under distribution shift in highly specialized domains or long-horizon multi-turn dialogues (repetitive reasoning, mixed-language outputs, time/identity inconsistencies) [§Known Issues and Future Directions].

Why it’s interesting

Step 3.5 Flash is the first filed open-weights model that brings together three threads the wiki has been tracking separately. First, it joins the low-active-MoE agentic-SWE Pareto frontier mapped out in Agentic Software Engineering (Qwen3-Coder-Next 3B-active, DeepSeek V3.2 37B-active, GLM-4.7 32B-active, MiniMax M2.1 10B-active, K2.5) at 11B active, with competitive SWE-bench Verified (74.4) and a leading Terminal-Bench 2.0 (51.0) under the lowest claimed inference cost in the table. Second, it adds a new instance of architectural attention-sink design — Head-wise Gated Attention as an “input-dependent attention sink” — that echoes the unified-view framing in A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training, where outlier-driven rescaling is reframed as explicit learned gating. Third, MIS-PO is a new RL-stability contribution targeting precisely the training-inference mismatch and off-policy drift that Reasoning RL tracks — but from a different angle than the gated-reward / format-reward / closed-loop synthesis stories on that page: it replaces continuous importance weighting with binary acceptance, which is the same “manifold-constraint” pattern as mHC and SSO (see Training stability at scale) just applied to the RL importance ratio rather than to weights or residual streams.

The packaging is also worth flagging: open weights, day-0 API + apps + Discord, INT4 GGUF for DGX Spark, multi-backend serving — matching the release-package convention tracked in Open foundation-model releases.