Skip to content
Bud's Wiki

DeepSeek-V3.1-Base — long-context-extended V3 base with UE8M0 FP8 microscaling

TL;DR

Section titled “TL;DR”

DeepSeek-V3.1-Base is the pretrained-only checkpoint underlying the DeepSeek-V3.1 release: a 671B-total / 37B-activated MoE built on the original DeepSeek-V3 base through a two-phase long-context extension — 32K phase scaled 10× to 630B tokens, then 128K phase scaled 3.3× to 209B tokens, following the V3 report’s methodology. The base is trained with weights and activations in the UE8M0 FP8 microscaling format and is the starting point for the instruct/hybrid-thinking post-training that produces DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking, and downstream for DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (V3.1-Terminus → V3.2 with DeepSeek Sparse Attention) and the V4 collection.

Key claims

Section titled “Key claims”
  • V3.1-Base is the pretrained checkpoint produced by extending the original DeepSeek-V3 base to long context in two phases, following the methodology in the original DeepSeek-V3 technical report [§Introduction].
  • The 32K-context extension phase was scaled 10× to 630B tokens [§Introduction].
  • The 128K-context extension phase was scaled 3.3× to 209B tokens [§Introduction].
  • Total / activated parameters: 671B / 37B; context length 128K [§Model Downloads, Table].
  • Training uses the UE8M0 FP8 scale data format on both model weights and activations to be compatible with microscaling formats; DeepGEMM is the recommended kernel companion [§Introduction, §Usage Recommendations].
  • The mlp.gate.e_score_correction_bias MoE-router parameters must be loaded and computed in FP32 even under the broader FP8 regime — a documented numerical-precision carve-out [§Usage Recommendations].
  • FP8 weights and activations must be formatted using the UE8M0 scale format specifically; this is presented as a hard requirement, not a recommendation [§Usage Recommendations].
  • The model architecture is the same as DeepSeek-V3 [§How to Run Locally].
  • Released under the MIT License [§License].

Method

Section titled “Method”

V3.1-Base is constructed by continuing pretraining of the original DeepSeek-V3 base in two long-context-extension stages. The methodology is unchanged from the V3 technical report — the contribution of this checkpoint is the expanded budget of those two stages: the 32K phase is increased tenfold (to 630B tokens) and the 128K phase 3.3× (to 209B tokens), with additional long documents collected for both. No architectural changes are introduced; the result is a V3-architecture base model with strengthened long-context capability, ready for post-training.

The other distinctive choice is the numerical format. V3.1-Base is the first DeepSeek checkpoint trained with weights and activations in the UE8M0 FP8 microscaling format, with the DeepGEMM kernel library called out as the companion inference stack. The MoE router’s score-correction bias is the documented exception, kept in FP32 to avoid known numerical issues at the router decision boundary.

The HF model card states explicitly that V3.1 (the instruct sibling) is “post-trained on top of DeepSeek-V3.1-Base” — so this checkpoint is the input to the hybrid-thinking + smarter-tool-calling + faster-thinking post-training that defines the V3.1 release.

Results

Section titled “Results”

The model card does not publish base-model evaluations separately from the instruct V3.1 results. The instruct sibling DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking reports substantial gains over V3-0324 on knowledge/math/code benchmarks in both non-thinking and thinking modes (e.g. MMLU-Pro 83.7 / 84.8 vs V3-0324’s 81.2; LiveCodeBench 56.4 / 74.8 vs 43.0; SWE Verified agent-mode 66.0 vs 45.4) — implying that the base improvements from the 10× and 3.3× long-context extensions, combined with the FP8 microscaling recipe, do not regress short-context capability [Evaluation table in companion model card].

Why it’s interesting

Section titled “Why it’s interesting”

V3.1-Base isolates the “pretraining stage” of the V3.1 release. Two things are interesting in isolation:

  • The long-context extension budget is large in absolute terms: 630B + 209B = ~840B tokens of continued pretraining at long context, on top of the already-completed V3 base run. That is comparable to a full pretraining run for a smaller model and is one of the largest publicly-disclosed long-context-extension efforts among open releases. It’s a concrete datapoint for Context Length / Quality Trade-off in Video Generation — the cost of credibly supporting 128K context in a frontier MoE.
  • UE8M0 FP8 at full scale: V3.1-Base is the open-source proof-of-concept that the UE8M0 microscaling format is viable for both weights and activations at the 671B-MoE scale, with the only documented FP32 carve-out being a single router bias. This is a non-trivial systems result that complements the wiki’s broader LLM Inference Efficiency tracking (which has so far emphasized KV-cache compression, speculative decoding, and sparse attention rather than numerical-format-at-training-time).

For downstream papers, V3.1-Base is the lineage root. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models starts from a later iterate of this checkpoint family (“V3.1-Terminus”), introduces DeepSeek Sparse Attention via a 1,000-step dense indexer warm-up + 15,000-step sparse training stage on top of it, and then runs the documented mixed-GRPO recipe. So V3.1-Base sits at the head of the open-source DeepSeek 2025/2026 cadence (V3.1-Base → V3.1 → V3.1-Terminus → V3.2 → V3.2-Speciale → V4).

See also

Section titled “See also”