Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

TL;DR

StepFun’s Step-3 is a 321B-parameter VLM (316B LLM + 5B vision encoder, 38B activated per token) presented as a model-system co-design study with one optimization target: minimize per-token decoding cost on long-context reasoning workloads. Two innovations work in tandem: (1) Multi-Matrix Factorization Attention (MFA) — a low-rank QK factorization with 64 query heads sharing a single K and V head at dim 256 — that keeps KV cache small and keeps attention arithmetic intensity hardware-friendly across H800/H20/A800/910B; (2) Attention–FFN Disaggregation (AFD) — a production inference system that runs attention and FFN on separate GPU sets, each chosen for its operational profile, communicating layer-wise through a balanced multi-stage pipeline. On Hopper, Step-3 hits 4,039 tok/s/GPU under a 50ms-TPOT SLA (4K context, FP8), versus DSv3’s 2,324 in the same setup, while activating more parameters than DSv3 or Qwen3-235B.

Key claims

Step-3 has 316B LLM parameters across 61 Transformer layers (hidden 7168), with MFA attention (64 query heads sharing 1 K + 1 V head, head dim 256, low-rank query down-projection to 2048) and DeepSeekMoE-style MoE with 1 shared expert on every layer except the first four and the last; activated parameters per text token are 38B, plus a 5B vision encoder [§2, Model Card].
Total or activated parameter count is a bad indicator of decoding cost: Qwen3-32B has fewer parameters than DSv3 and Step-3 but the highest per-million-token cost in the comparison table, because attention design dominates [§4.2, Table 6, Observation 2].
Under AFD, attention cost overtakes FFN cost even at 8K context and the gap grows with context length, so attention design matters more than activated FFN parameter count [§4.2, Observation 3, Table 6].
Reducing KV cache size alone is not sufficient — designs that drive attention arithmetic intensity too high (DSv3’s MLA on weaker hardware than H800; hybrid linear-attention models with a few heavy full-attention layers) lose most of the theoretical savings; Step-3’s MFA keeps both KV size and arithmetic intensity small simultaneously [§4.2, Observation 4, §4.3].
MoE sparsity must be hardware-aware: over-sparse MoEs (DSv3, Kimi K2, Llama 4 Maverick) need very large EP scale to amortize weight access, doubling or tripling FFN cost in practice on H800; Step-3 deliberately picks a “not-too-sparse” point so AFD can hit high FFN MFU at modest deployment size [§4.2 footnote, §5.4].
Hybrid linear-attention models (MiniMax M1, Llama 4 Maverick) fail to save what their layer counts suggest: the few full-attention layers alone access more KV bytes than Step-3’s entire model, and the layer-time imbalance causes pipeline bubbles in distributed inference [§4.3, Fig. 3].
AFD outperforms DeepSeek-style Expert Parallelism on five axes: smaller decoding deployment (32 GPUs vs DSv3’s 320), independent scaling of attention vs FFN under long context, no EP load-imbalance ad-hoc patches, optional heterogeneous hardware per subsystem, and a cleaner divide-and-conquer performance model [§3.2].
AFD is also a model-side enabler: because attention and FFN are deployed independently and can each be optimized to near-roofline, the model designer can co-design each part without the other constraining its arithmetic intensity [§3, §3.1].
Step-3’s design target is a 50ms TPOT pipeline split as 16.6ms / 16.6ms / 16.6ms across attention / FFN / network (3-stage) or 12.5ms each across A → comm → F → comm (4-stage) — communication latency is treated as a first-class pipeline stage rather than overhead [§3.1].
On Hopper GPUs (H800), Step-3 reaches 4,039 tokens/sec/GPU under a 50ms-TPOT SLA at 4K context with FP8 and without multi-token-prediction, vs DSv3’s 2,324 in the same setup — a new Pareto frontier on the activated-parameters vs decoding-cost axis [Abstract, §7.3].
Step-3 is the cheapest at both 8K and 32K context in the per-1M-token cost comparison: at 8K, $0.055 (attention) +$ 0.015 (FFN) with optimal H800+H20 mix vs DSv3’s $0.068 on H800-EP and Qwen3-MoE's$ 0.062; at 32K, the gap widens substantially [§4.2, Fig. 2, Observation 1].
Step-3 is the only model in the comparison whose attention cost is roughly hardware-invariant — MFA’s costs on H800, H20, A800 and 910B differ by less than 10%, vs DSv3 (multi-fold variance) and Qwen3-32B GQA (heavy KV-cache penalty off H20) [§4.2, Observation 4, Table 6].
The whole Step-3 model is quantized to FP8 (attention + FFN) without accuracy loss; this is the same aggressive scheme applied to peers in the cost table, so comparisons are like-for-like [§4.1, MLA family / GQA family / Step-3 bullets].
Pangu Pro MoE is presented as a counter-example to “hardware-optimized” rhetoric: it is optimized for Ascend 910B training (16.5B activated parameters → ~50% cheaper than Step-3 to train) but its 910B decoding cost is much higher than Step-3 — the training/decoding optima diverge sharply [§4.3, Fig. 4].

Method

The paper interleaves model design and inference-system design as a single object. The model has 61 layers of Transformer blocks. Each block uses Multi-Matrix Factorization Attention (MFA), in which queries are produced by a low-rank factorization — input is down-projected from hidden dim 7168 to rank 2048, normalized, then up-projected to 64 heads × 256 dim — while a single K head and a single V head are shared across all 64 queries (head dim 256). This shrinks the KV cache (one K and one V per token) and the arithmetic intensity stays moderate, which is the design knob the paper exploits to keep attention cost roughly constant across H800/H20/A800/910B (see Table 6 row “Step-3”). FFN blocks (all layers except the first four and the last) use DeepSeekMoE-style routing with one shared expert; the paper deliberately picks a sparsity that is not over-sparse, so that the FFN can run in compute-bound mode under AFD on a single node’s worth of GPUs.

The Attention–FFN Disaggregation (AFD) system deploys attention and FFN onto separate sets of GPUs, connected by high-speed network. Each layer’s forward pass becomes a three- or four-stage pipeline (attention → comm → FFN → comm), with stage budgets of 16.6ms (3-stage) or 12.5ms (4-stage) summed across all 61 layers to hit a 50ms TPOT SLA. Because attention and FFN no longer share GPUs, each can be sized independently: attention instances scale with context length, FFN instances run at the batch size that gives high MFU regardless of attention’s batch. The paper argues this beats DeepSeek’s pure-EP approach on deployment scale (32 GPUs vs 320), load-balance robustness (no duplicated experts), context-length elasticity, and analytical clarity. The cost analysis in §4 takes AFD as given: with attention and FFN decoupled, the per-1M-token decoding cost factors cleanly into max(attention compute, KV memory access) + linear projections + FFN compute, each priced against the cheapest hardware that can run it near roofline (Tables 4–6). Step-3 wins the per-token cost comparison at 8K and 32K context against DSv3, Kimi K2, Qwen3-MoE, Qwen3-32B, Llama 4 Maverick, MiniMax M1, ERNIE 4.5, and Pangu Pro MoE.

Results

4,039 tok/s/GPU on Hopper at 50ms TPOT, 4K context, FP8, no MTP, vs DSv3’s 2,324 in the same setup [Abstract, §7.3].
Decoding cost per 1M tokens at 8K context (best AFD hardware mix): Step-3 = $0.070 total (attention$ 0.040 on H20 + FFN $0.015 on H800 + linear), beating DSv3's$ 0.068+ on H800-only EP and Qwen3-MoE’s mix; at 32K context, Step-3 is $0.149 vs DSv3's$ 0.226 and Qwen3-MoE’s $0.207 [§4.2, Table 6].
MFA attention cost on a 1M-token batch is approximately hardware-invariant: $0.048 (H800) /$ 0.040 (H20) / $0.040 (A800) /$ 0.043 (910B) at 8K context — vs DSv3 MLA at $0.054 /$ 0.128 / $0.114 /$ 0.113 (a >2× off-Hopper penalty) and Qwen3-32B GQA at $0.181 /$ 0.069 / $0.120 /$ 0.133 (a >2× off-H20 penalty) [§4.2, Table 6].
Step-3 is in the Pareto-best corner of activated-parameters × decoding-cost; Pangu Pro MoE has less than half Step-3’s activated parameters but higher 910B decoding cost than Step-3, and Qwen3-32B has fewer parameters than both but the highest cost in the table [§4.2, Fig. 1, Observation 2, §4.3, Fig. 4].
A single Step-3 decoding instance uses 32 GPUs vs the 320 GPUs required for DSv3’s EP-only deployment [§3.2].

Why it’s interesting

Step-3 sits at the intersection of three threads on the wiki, and adds a sharply different position to each. Against the speculative-decoding / KV-compression work catalogued under LLM Inference Efficiency (especially Speculative Decoding: Performance or Illusion?‘s finding that verification dominates 42–95% of SD time and that target-model attention is the wall), Step-3 argues that the attention wall should be addressed at the model-design layer rather than the decoding-algorithm layer: MFA is a different attention shape, not a clever scheduler. Against the MoE-routing-design papers (MoE Routing Design, including MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production and Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design), Step-3 deliberately backs off from extreme sparsity — its 38B activated of 316B is denser than DSv3 or Kimi K2 — and pairs that with a system (AFD) that is the algorithmic counterpart to MegaScale-MoE’s “drop TP for attention, keep EP for FFN” recipe. Against Open foundation-model releases like HunyuanImage 3.0 Technical Report and the Kimi K2 line (Kimi K2: Open Agentic Intelligence), Step-3 is StepFun’s most detailed serving-cost story yet — the same lineage as Step 3.5 Flash (Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters) — and is now the canonical reference for “MoE inference costs depend on attention design, not on activated parameter count.”