dots.llm1 Technical Report

TL;DR

dots.llm1 is a 14B-active / 142B-total fine-grained MoE LLM from rednote-hilab (Xiaohongshu) trained on 11.2T high-quality tokens with zero synthetic data during pretraining, hitting Qwen2.5-72B-class quality at roughly a quarter of the activated compute (130K GPU-hours per 1T tokens vs 340K for Qwen2.5-72B in their framework). The architecture is a DeepSeekMoE clone — top-6 of 128 routed experts plus 2 shared experts, MHA with QK-Norm, auxiliary-loss-free balancing — with the headline novelty on the data side: a three-stage pipeline (document prep → rule-based cleaning → model-based filtering) featuring a line-level Web Clutter Removal Model and a 200-class category-balancing classifier. Open release under MIT includes base + SFT instruct checkpoints, intermediate checkpoints at every 1T tokens, and an interleaved-1F1B all-to-all overlap recipe integrated upstream into Megatron-Core. Filed because Will flagged the data-cleaning recipe as worth studying for Luma pretraining work.

Key claims

14B activated / 142B total MoE; performance on par with Qwen2.5-72B (dense) and ahead of DeepSeek-V2 at the base-model level despite ~1/2 the activated params of Qwen2.5-72B [§1, Table 2].
11.2T pretraining tokens with no synthetic data and no model distillation during pretraining [§Abstract, §Contributions].
Architecture: 62 layers (first layer dense FFN, rest MoE), MHA with QK-Norm (RMSNorm on Q/K), top-6 of 128 routed experts + 2 shared, SwiGLU experts, FP32 gating, auxiliary-loss-free load balancing à la DeepSeek with a sequence-wise auxiliary balance loss [§2].
Training cost: 130K GPU-hours per 1T tokens vs 340K for Qwen2.5-72B in the same framework; total 1.46M GPU-hours vs 6.12M for Qwen2.5-72B — a ~4× reduction [§4.5, Table 4].
Three-stage data pipeline (document prep → rule-based → model-based) with two flagged novelties: a line-level Web Clutter Removal Model that filters boilerplate/repetition cheaper than document-level filters, and a 200-class category-balancing classifier that upweights knowledge/factual content and downweights fiction/product-description content [§4.1].
Controlled 350B-token / 1.5B-dense-model bake-off against TxT360 (the open-source SoTA web corpus baseline) reports the dots.llm1 web pipeline produces a stronger training signal [§4.5, Fig. 4].
Infrastructure: an interleaved-1F1B variant that overlaps MoE all-to-all with compute during the steady phase with no extra bubble and no extra activation memory — contributed upstream to NVIDIA Megatron-Core; a custom grouped-GEMM kernel aligns expert token segments to WGMMA tile multiples, beating Transformer Engine v2.1 by 14.00% forward / 6.68% backward on H800 [§3.1, §3.2, Table 1].
WSD (warmup-stable-decay) schedule: 4K warmup steps to 3e-4 LR for the stable 10T-token phase; batch size ramps 64M → 96M → 128M tokens; two annealing stages on 1.2T further tokens with reasoning/knowledge upweighting [§4.2].
UtK long-context recipe extends 8K → 32K on 128B additional tokens via chunked-reconstruction objective, with no real degradation up to 16K on RULER (94.9 / 92.6) and modest drop at 32K (87.7) — Qwen2.5-72B is 94.3 / 93.1 / 92.7 [§4.3, Table 3].
Stable training: no irrecoverable loss spikes and no rollback operations over the full 11.2T-token run [§4.5].
Open accessibility: intermediate checkpoints released at every 1T tokens to enable training-dynamics research — a release primitive that is rare in the open-MoE cohort [§Contributions, §1].
SFT recipe: 400K instruction instances, two-phase fine-tune (2-epoch upsampled multi-session concat → rejection-sampling fine-tuning with verifiers for math/code), cosine 5e-6 → 1e-6 [§5.1].
Post-training results: dots.llm1.inst (14B-A) matches Qwen3-32B average across English/Code/Math/Chinese/Alignment categories; leads by a wide margin on CNMO24 math (40.6 vs 33.2 for Qwen3-235B-A22B) and AlpacaEval2 (64.4) [§5.2, Table 5].

Method

dots.llm1 is a standard fine-grained DeepSeekMoE decoder: 62 transformer layers (layer 1 is a dense SwiGLU FFN, layers 2-62 are MoE), MHA with QK-Norm (RMSNorm applied to Q and K projections before attention), 128 routed experts of which top-6 are selected per token, plus 2 always-on shared experts (so 8 experts active per token total). The gating projection is computed in FP32 for routing-decision stability; load balancing uses the DeepSeek auxiliary-loss-free bias-update trick augmented with a sequence-wise balance loss for tail-case extreme imbalance, and reportedly drops no tokens during training [§2].

The two contributions that distinguish the report from a “DeepSeekMoE clone” headline are on data and systems. Data: a three-stage pipeline where document prep is the cheap surface step, rule-based filtering automates the bulk of cleaning, and a model-based stage performs final quality+diversity selection. Two stage-specific innovations: (i) a line-level Web Clutter Removal Model that targets boilerplate, navigation chrome, and repetitive lines below the document granularity at which most open pipelines operate, and (ii) a 200-class classifier that rebalances category proportions of crawled web data to upweight encyclopedia/popular-science content and downweight fiction/product copy. The 1:1 Chinese-English balance is preserved throughout, and the pipeline beats TxT360 in a controlled 350B-token / 1.5B-dense bake-off [§4.1, §4.5].

Systems: an interleaved 1F1B pipeline-scheduling variant that overlaps MoE all-to-all with compute during the steady phase (cost: one extra warmup step; no extra activation memory; positioned as a memory-cheaper alternative to DeepSeek’s DualPipe with marginally higher bubble rate), and a custom grouped-GEMM kernel that pads expert token segments to WGMMA tile multiples so within-block tiling is uniform and scheduling resembles a regular GEMM. Reported ~14% forward and ~7% backward speedup over Transformer Engine v2.1 on H800 [§3.1, §3.2].

Results

Base model (Table 2, vs Qwen2.5-32B/72B Base, DeepSeek-V2 Base, DeepSeek-V3 Base):

Chinese average: 91.3 (dots.llm1) vs 90.3 (Qwen2.5-72B), 89.5 (DeepSeek-V3) — leads.
English average: 75.7 vs 76.3 (Qwen2.5-72B), 78.0 (DeepSeek-V3) — competitive.
Math average: 78.3 vs 77.3 (Qwen2.5-72B), 82.1 (DeepSeek-V3).
Code average: 59.6 vs 59.0 (Qwen2.5-72B), 62.5 (DeepSeek-V3); HumanEval Pass@1 64.0 vs 56.7 for Qwen2.5-72B Base.

Instruct model (Table 5, vs Qwen2.5-32B/72B-Inst, Qwen3-32B / 235B-A22B, DeepSeek-V2 Chat, DeepSeek-V3, GPT-4o-0806):

English average 64.4 — between Qwen3-32B (64.5) and Qwen3-235B-A22B (66.9).
Math: AIME24 Pass@1 33.1 (vs Qwen3-32B 27.1, Qwen3-235B-A22B 37.5, DeepSeek-V3 34.0); CNMO24 Pass@1 40.6 beats every listed model including DeepSeek-V3 (33.9) and Qwen3-235B-A22B (33.2).
Chinese average 80.5 — leads non-DeepSeek-V3 entries; C-Eval EM 92.2 is the highest in the table.
Alignment: AlpacaEval2 64.4 (above Qwen3-32B’s 58.1, near Qwen3-235B-A22B’s 66.8).

Long context (RULER, Table 3): 94.7 / 94.9 / 92.6 / 87.7 at 4K / 8K / 16K / 32K respectively — comparable to Qwen2.5-72B (96.5 / 94.3 / 93.1 / 92.7) up to 16K, with a more visible drop at 32K.

Grouped-GEMM speedup over NVIDIA Transformer Engine v2.1 on H800 (Table 1): 6.49–24.68% forward across the swept (g, m, n, k) grid, average 14.00%; 2.86–8.29% backward, average 6.68%.

Why it’s interesting

dots.llm1 lands as the load-bearing “no synthetic data” datapoint in a cluster where the wiki had been documenting the opposite direction. Synthetic Training Data currently runs on the thesis that a curation/verification gate over synthetic outputs is the load-bearing trick; dots.llm1’s controlled bake-off against TxT360 argues the alternative — a sufficiently aggressive real-data cleaning pipeline (line-level clutter removal + category rebalancing) is enough at 11.2T tokens. This pairs cleanly against Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (CLIMB optimizes real-data mixture via an automated proxy/predictor loop) and Smol Training Playbook (GPU MODE talk on SmolLM3) (SmolLM3 leans heavily on Cosmopedia synthetic textbooks) — three different bets on where the marginal token comes from at trillion-token scale.

On the MoE side, dots.llm1 sits in the same DeepSeekMoE design lineage as the papers on MoE Routing Design (fine-grained experts, shared experts, auxiliary-loss-free balancing) and contributes a frontier-relevant 142B-total / 14B-active datapoint with a transparent training story. The sequence-wise balance loss as a complement to the auxiliary-loss-free bias-update — not a replacement — is a small but practically interesting wrinkle relative to pure DeepSeek-V3 recipes documented in Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models. The reported “no token dropping during training” claim is also a non-trivial production-scale validation that fine-grained MoE + auxiliary-loss-free balancing is robust at 11.2T tokens.

The pretraining-stability story aligns with the Training stability at scale consensus: QK-Norm is the architectural primitive doing the work (no irrecoverable loss spikes across the full 11.2T run), and µP-style module-specific LR scaling is not mentioned anywhere in the paper — consistent with How to Set the Learning Rate for Large-Scale Pre-training?‘s finding that modern normalization makes the µP machinery unnecessary at this scale.