Skip to content
Bud's Wiki

Qwen3.8-Flash-Next — Hybrid Attention with QSA, Gated Residual, N-gram Embedding

TL;DR

Section titled “TL;DR”

Qwen3.8-Flash-Next is Alibaba Qwen’s experimental preview of the architecture underpinning Qwen4 — a 125B-total / 6B-active multimodal MoE with 51B of extra n-gram-embedding parameters and a 4B MTP head, positioned as the successor to Qwen3-Next in the Qwen3-Next series. Four architectural changes carry the release: (1) the Gated DeltaNet + Gated Attention hybrid is reworked into Gated DeltaNet + Qwen Sparse Attention (QSA), which selects at the micro-block level (512 blocks / 2048 tokens budget) rather than at individual tokens; (2) Gated Residual modulates widened residual streams via an element-wise data-dependent read gate and per-branch scalar write gate to preserve training stability with finer expressiveness; (3) N-gram Embedding places a 20M-entry bigram/trigram lookup at layer 2 as a parameter-scaling axis that is cheaper than MoE and amenable to offloading on memory-constrained accelerators; (4) a tailored training recipe applies Muon and AdamW to specific weight categories, and — guided by refitted scaling laws — eliminates traditional batch-size warmup and starts directly at the target batch size to reduce total optimizer steps. Context length is 262K native, extensible to 1M via YaRN. The model ships in thinking mode by default and reports strong agentic-coding and computer-use scores against Qwen3.8-27B, Qwen3.7-Plus, DeepSeek-V4-Flash-0731, and Claude Opus 4.6 (Max).

Key claims

Section titled “Key claims”
  • Hybrid attention layout: 48 layers organized as 12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), giving a 3:1 linear-to-sparse-softmax ratio matching Kimi Linear / SANA-Video 2.0 [Model Overview].
  • Qwen Sparse Attention (QSA) selects at the micro-block level rather than per token; budget is 512 blocks or 2048 tokens, and uses an MQA-style indexer with 4 query heads + 1 shared key head at head-dim 128 [Model Overview, Highlights §Hybrid Attention with QSA].
  • Gated Residual has 4 branches with bottleneck rank 320; combines an element-wise data-dependent read gate with a per-branch scalar write gate to modulate widened residual streams “while preserving training stability and keeping inference overhead low” [Highlights §Gated Residual].
  • N-gram Embedding adds a 20,000,000-entry bigram/trigram table at layer 2 (51B parameters), decoupled from FLOPs and designed to be offloaded to CPU on memory-constrained accelerators — presented as an alternative parameter-scaling axis to MoE [Model Overview, Highlights §N-gram Embedding].
  • MoE at 512 routed experts with 10 routed + 1 shared activated, expert intermediate dim 640 — matches the Qwen3-Next-80B-A3B expert count while widening the shared+routed activation slightly [Model Overview].
  • Training recipe applies Muon and AdamW to specific weight categories and, guided by refitted scaling laws, “eliminates traditional batch-size warmups and starts directly at the target batch size” while safely supporting larger learning rates [Highlights §Tailored Training Recipe].
  • Native context length is 262,144 tokens, extensible to 1,000,000 via YaRN; the config uses mrope-interleaved YaRN with factor 4.0 [Best Practices §Processing Ultra-Long Texts].
  • MTP is a single trained multi-step layer, exposed to inference frameworks for speculative decoding without a separate draft model [Model Overview].
  • Agentic-coding headline: 58.7 on DeepSWE 1.1 (best across Claude Code / mini-SWE-agent harnesses at temp=1.0, 256K context), 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual — both SWE-bench numbers beat Qwen3.8-27B, Qwen3.7-Plus, DeepSeek-V4-Flash-0731 and Claude-Opus-4.6 (Max) in-row [Benchmark Results §Language, Coding].
  • Agentic breadth: 73.9 on CoWorkBench, 55.7 on JobBench, 73.5 on Toolathlon Verified (Pass@1) — all listed as best-in-row against the Qwen3.8-27B / Qwen3.7-Plus / DeepSeek-V4-Flash-0731 / Claude Opus 4.6 (Max) column set [Benchmark Results §Language, Agent].
  • Computer/mobile use: 84.5 on AndroidWorld and 19.4 binary / 52.3 partial on OSWorld 2.0 — both column leaders in the vision-language table; ERQA embodied-intelligence at 72.3 also leads [Benchmark Results §Vision Language].
  • Recommended sampling: thinking mode uses temperature=1.0, top_p=0.95, top_k=20, presence_penalty=0.0; instruct mode uses temperature=0.7, top_p=0.80, presence_penalty=1.5 [Best Practices §Sampling Parameters].
  • For long-horizon agentic tasks, “lower reasoning effort does not always reduce overall task completion time” — insufficient analysis produces more failures and retries that raise total latency and token consumption [API Usage §Chat Completions].

Method

Section titled “Method”

The core architectural changes over Qwen3-Next (Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement)) sit in three places. First, the softmax half of the hybrid is replaced: Gated Attention becomes Qwen Sparse Attention (QSA), which operates at the micro-block level rather than per-token — a 512-block / 2048-token budget is selected by an MQA-style indexer (4 query heads, 1 shared key head, head-dim 128) and only those blocks participate in the softmax attention computation. This is architecturally close to DeepSeek Sparse Attention (DSA) — the same “select blocks, then attend within budget” template — deployed here at a 3:1 linear-to-sparse-softmax ratio (12 repeating groups of 3 × (GDN → MoE) → 1 × (QSA → MoE)). Full-attention Q/KV split is 24/2 heads at head-dim 256, with 64-dim rotary and a partial rotary factor. Second, Gated Residual widens the residual stream into 4 branches with rank-320 bottlenecks; each token can independently read/write those branches via a data-dependent element-wise gate on read and a per-branch scalar on write. Third, and most novel, an N-gram Embedding table with 20M entries (bigrams + trigrams) is placed at layer 2, adding 51B parameters that live in the embedding path rather than in MoE experts — the model card explicitly frames this as “a unique axis for parameter scaling that requires less computation and is more amenable to offloading than MoE.” Vocabulary is 248,320 (padded). MoE keeps 512 experts with 10 routed + 1 shared active per token and expert-intermediate dim 640. MTP is a single trained multi-step head. The training recipe uses Muon and AdamW on category-specific weight groups and — guided by refitted scaling laws — starts directly at the target batch size instead of warming up, which the team reports reduces total optimizer steps while safely admitting larger learning rates. YaRN with factor 4.0 and mrope-interleaved rope extends context from 262K to 1M.

Results

Section titled “Results”

Language benchmarks in the model card compare Qwen3.8-Flash-Next against Qwen3.8-27B (dense), Qwen3.7-Plus (397B / 17B-active), DeepSeek-V4-Flash-0731 (284B / 13B-active), and Claude Opus 4.6 (Max). Agentic coding is the strongest surface: 58.7 on DeepSWE 1.1 (best across the Claude Code / mini-SWE-agent harnesses; the model card notes it performs best on mini-SWE-agent) versus 42.2 / 16.5 / 54.4 for Qwen3.8-27B / Qwen3.7-Plus / DeepSeek-V4-Flash-0731; 62.5 on SWE-bench Pro vs 61.7 / 55.8 / 56.0 / 53.4 (Claude Opus 4.6 Max); 81.0 on SWE-bench Multilingual vs 73.8 / 75.8 / — / 77.5. Toolathlon Verified (Pass@1) hits 73.5, beating Qwen3.8-27B (67.1) and DeepSeek-V4-Flash-0731 (70.3). On repo-level code generation NL2Repo-Bench, Qwen3.8-Flash-Next comes in at 48.1, behind DeepSeek-V4-Flash-0731 (54.2) and Claude Opus 4.6 Max (47.6). General reasoning: 91.7 GPQA Diamond, 35.9 HLE (vs Claude Opus 4.6 Max’s 40.0), 91.9 LiveCodeBench v6 (row-best), 81.3 IFBench (row-best). Vision-language table: 84.5 AndroidWorld and 19.4 binary / 52.3 partial OSWorld 2.0 both lead the column; ERQA embodied-intelligence at 72.3 leads over Qwen3.8-27B (65.5); RealWorldQA 88.5; MathVision 90.6 / 95.7 (without-CI / with-CI); CharXiv (RQ) 84.6 / 90.6. All comparisons are at temp=1.0, top_p=0.95, 256K context window unless noted; SWE-bench Pro problematic tasks were corrected and baselines re-evaluated. Downloads-last-month at filing: 2,551.

Why it’s interesting

Section titled “Why it’s interesting”

Three threads on the wiki collide here. First, this is the most extreme filed hybrid-linear-attention architecture yet — same 3:1 linear:sparse-softmax ratio as Kimi Linear: An Expressive, Efficient Attention Architecture and SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation, but the softmax path is now block-sparse (QSA) rather than dense — a direct architectural sibling to DeepSeek Sparse Attention that lands the Hybrid Linear Attention cluster’s “right primitive?” open question at a new configuration (Gated DeltaNet + block-sparse softmax, not KDA + MLA, not lightning + full). Second, the 20M-entry N-gram Embedding at layer 2 is the third filed instance of architectural parametric memory as an alternative to MoE scaling, extending the Parametric memory thread that began with STEM: Scaling Transformers with Embedding Modules‘s FFN-embedding-table lever and Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling‘s hierarchical input-side n-gram table — Qwen’s take makes the offloading argument explicit (“more amenable to offloading than MoE”) and lands it at production scale in a shipping open-weights model. Third, the “eliminate batch-size warmup, start at target batch size” recipe is a concrete counterpoint to the Hyperparameter scaling laws tradition of extrapolating warmup schedules from smaller runs — the team claims refitted scaling laws let them skip warmup entirely. The lineage back to Qwen3-Next-80B-A3B — hybrid Gated DeltaNet + Gated Attention, ultra-sparse 512-expert MoE, MTP (Alibaba Qwen announcement) (Qwen3-Next-80B-A3B) is clean: same 512-expert MoE, same Gated DeltaNet linear path, same MTP head — but QSA replaces Gated Attention, Gated Residual replaces plain widened residuals, and N-gram Embedding is entirely new.

See also

Section titled “See also”