DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention

TL;DR

DeepSeek-V3.2-Exp is the experimental, V3.1-Terminus-derived checkpoint that introduced DeepSeek Sparse Attention (DSA) to the open-weights community — the same mechanism later productized and reported in DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. DSA computes a small lightning indexer score per query against all prior tokens (ReLU + FP8, few heads) and routes the main MLA attention to only the top-k indexed KV entries, dropping core-attention cost from O(L²) to O(Lk) while keeping output quality on par with the dense V3.1-Terminus baseline across MMLU-Pro / GPQA / LiveCodeBench / AIME / SWE-Verified / Terminal-bench. The release ships an inference demo with separated indexer + MLA RoPE layouts, Day-0 vLLM / SGLang / Docker support across H200 / MI350 / Huawei NPUs (A2/A3), and open-source kernels (TileLang for readability, DeepGEMM + FlashMLA for performance).

Key claims

DSA is the first fine-grained sparse-attention mechanism deployed in a frontier open-weights LLM: prior production designs were either dense MLA (V3, V3.1) or coarse block/window sparsity, while research on MoBA/NSA had not been productized [README §Introduction].
V3.2-Exp is positioned as an intermediate step toward DeepSeek’s next-generation architecture, not a final release — the goal is to validate DSA’s training-and-inference efficiency optimizations under long context before committing to it for the V4 line [README §Introduction].
Training configurations were deliberately aligned with V3.1-Terminus so any benchmark delta is attributable to the architectural change (DSA), not data/recipe shifts [README §Introduction].
Across 14 public benchmarks (reasoning + agentic), V3.2-Exp is within ~1 point of V3.1-Terminus on most: MMLU-Pro 85.0 = 85.0, GPQA-Diamond 79.9 vs 80.7, LiveCodeBench 74.1 vs 74.9, AIME 2025 89.3 vs 88.4, Codeforces 2121 vs 2046, SWE-Verified 67.8 vs 68.4, SWE-bench Multilingual 57.9 vs 57.8, Terminal-bench 37.7 vs 36.7, BrowseComp 40.1 vs 38.5, BrowseComp-zh 47.9 vs 45.0, SimpleQA 97.1 vs 96.8 — sparse beats dense on several agentic and search benchmarks [README Table].
Two benchmarks regress non-trivially: Humanity’s Last Exam 19.8 vs 21.7 (–1.9 pts), HMMT 2025 83.6 vs 86.1 (–2.5 pts) — both math/reasoning-heavy with shorter contexts where sparsity gives no efficiency win [README Table].
Inference-demo implementation gotcha (2025.11.17 update): the indexer module requires a non-interleaved RoPE input layout, whereas the MLA module expects an interleaved layout; earlier demo code applied interleaved RoPE to both, silently degrading performance [README §Update].
Open-source kernels are split by purpose: TileLang kernels are released for readability and research use, while high-performance CUDA kernels are released as indexer-logit kernels (incl. paged versions) in DeepGEMM PR #200 and sparse-attention kernels in FlashMLA PR #98 [README §Open-Source Kernels].
Multi-hardware support at release includes H200, AMD MI350, and Huawei Ascend NPUs (A2 + A3) — the SGLang Docker images explicitly tag dsv32-rocm, dsv32-a2, dsv32-a3 alongside the standard CUDA image [README §SGLang].
The recommended SGLang launch command pairs --tp 8 --dp 8 --enable-dp-attention, indicating data-parallel attention is the assumed deployment topology for the 671B-parameter (with EXPERTS=256 MoE) base [README §SGLang].
vLLM provides day-0 support, with a project recipe page at docs.vllm.ai/projects/recipes/.../DeepSeek-V3_2-Exp.html [README §vLLM].
The model weights and repository are MIT-licensed, with the citation key deepseekai2024deepseekv32 (despite the 2025 release year) explicitly titling DSA in the bibtex: “Boosting Long-Context Efficiency with DeepSeek Sparse Attention” [README §License, §Citation].

Method

V3.2-Exp is initialized from V3.1-Terminus and replaces dense MLA with DSA. Each query token computes index scores I_{t,s} = Σ_j w^I_{t,j} · ReLU(q^I_{t,j} · k^I_s) against all prior tokens via a small lightning indexer (H_I heads, ReLU, FP8 weights), and the main MLA attention head reads only the top-k indexed KV entries — instantiated under MLA’s MQA mode so the latent KV is shared across all query heads of a token. Continued pretraining from V3.1-Terminus runs in two stages: a 1,000-step dense warm-up where only the indexer is trained against a KL-divergence-to-main-attention target (everything else frozen, lr 1e-3, 2.1B tokens), followed by a 15,000-step sparse stage where top-k=2048 is enabled and indexer + main model are co-optimized on separated computational graphs (lr 7.3e-6, 943.7B tokens). The full architecture + training math is documented in DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models §2.1.

The inference repo (inference/) ships a converter convert.py that reshards HF weights for tensor-parallel inference (--n-experts 256 --model-parallel ${MP}) and a generate.py interactive entrypoint with config_671B_v3.2.json. The 2025-11-17 RoPE-layout fix lives in this demo and is the canonical reference implementation for how DSA’s indexer and MLA’s main attention must be wired differently — a detail that, per the team, “potentially leading to degraded model performance” if missed.

Results

Reasoning-mode parity vs V3.1-Terminus (14 benchmarks, no tool use): essentially matched on 11; reasoning-heavy short-context (HLE 19.8 vs 21.7, HMMT 83.6 vs 86.1) shows a small regression; sparse beats dense on AIME 2025 89.3 vs 88.4 and Codeforces 2121 vs 2046 [README Table].
Agentic tool use (4 benchmarks): BrowseComp 40.1 vs 38.5, BrowseComp-zh 47.9 vs 45.0, SimpleQA 97.1 vs 96.8, SWE-Verified 67.8 vs 68.4, SWE-bench Multilingual 57.9 vs 57.8, Terminal-bench 37.7 vs 36.7 — sparse beats or matches dense on 5/6 agentic axes [README Table].
Long-context cost reduction: documented in DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models §2.3 (Fig. 3): decode-side cost-per-million-tokens gap grows with context length on H800, reaching multiples by 128K.
AA-LCR long-context evaluation: V3.2-Exp scores +4 points over V3.1-Terminus in reasoning mode, with consistent gains across Fiction.liveBench (also in DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models §2.2).
Downloads: 183,608 last-month downloads on Hugging Face as of the fetch date (a community-adoption proxy).

Why it’s interesting

V3.2-Exp is the artifact behind the wiki’s tracking of DSA — the V3.2 paper DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models describes it; this HF model card is the thing researchers actually download, the canonical reference inference implementation (incl. the post-release RoPE-layout fix), and the open-source kernel staging ground. Three threads converge here:

For LLM Inference Efficiency, DSA is a different lever than the page has tracked so far — the dominant threads have been speculative decoding (SD, EAGLE-3, TorchSpec, SSD/Saguaro), KV-cache quantization (TurboQuant vs RaBitQ), and AR-vs-DLM cache-friendliness (WeDLM). DSA targets a fourth lever: attention-pattern sparsity at the kernel level, with documented O(L²) → O(Lk) cost reduction and open-source CUDA kernels (DeepGEMM PR #200, FlashMLA PR #98) that other groups can drop into their stacks. Crucially, DSA ships as a frontier-scale deployed system rather than a research demo — the prior open sparse-attention work tracked under Sparsity is Cool: Reverse-Engineering MoBA and NSA was reverse-engineered from closed labs (MoBA, NSA).
For Hybrid Linear Attention, DSA represents a third architectural answer alongside the linear/softmax hybrids (Kimi Linear’s 3:1 KDA:MLA, MiniMax-M1’s 7:1 lightning:softmax, Qwen3-Next’s GDN+Gated Attention) and dense softmax: keep the full attention shape, but make it sparse via a learned top-k selector. The open question “what linear:softmax ratio is right” on that page now has a fourth comparison point — DSA argues you don’t need linear attention at all if you can index sparsely into the KV.
For Open foundation-model releases, V3.2-Exp is a textbook example of the “single-model release with full release package” pattern — base weights + inference demo + multi-backend Docker images (H200/MI350/Huawei A2/A3) + companion open-source kernels (TileLang for readability, CUDA for perf) + the canonical reference for a post-release implementation correction (the 2025-11-17 RoPE update). The Huawei-NPU branch is particularly notable: at the time of release this was one of the first frontier open LLMs with native Ascend support, foreshadowing the V3.2 paper’s claim of architecture + recipe maturity.

The relationship to DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models is “experimental → production”: V3.2-Exp validates DSA under tight train-config parity with V3.1-Terminus; V3.2 (and Speciale) build the agentic-task-synthesis + scaled-GRPO recipe on top.