Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

TL;DR

Pangu Ultra is Huawei’s 135B-parameter dense decoder-only LLM, pre-trained from scratch on 8,192 Ascend NPUs over 13.2T tokens. The technical contributions positioned as load-bearing are (a) depth-scaled sandwich normalization (DSSN), a sandwich-norm variant whose post-norm γ is initialized to scale with layer depth, and (b) a TinyInit-style depth-scaled parameter initialization that scales init std by both layer index and hidden size. The paper reports a from-scratch run with no loss spikes and benchmark numbers competitive with Llama 3 405B and Mistral Large 2 at ~3× fewer parameters. Subsequent whistleblower allegations from inside Huawei’s Noah’s Ark Lab (True Story of Pangu — Huawei Noah's Ark whistleblower statement) cast doubt on the integrity of other Pangu releases (most centrally Pangu-MoE 72B’s claim of being from-scratch, alleged to be continued-pretraining of Qwen 2.5 14B with the tokenizer water-marks scrubbed) and on the benchmark numbers reported across the Pangu series. The whistleblower explicitly defends Pangu Ultra (135B V3) as a real from-scratch run; the post-hoc trustworthiness of the DSSN / depth-scaled init claims is the open question for Luma.

Key claims

135B-parameter dense model, 94 layers, hidden 12,288, FFN intermediate 28,672, SwiGLU + GQA (8 KV heads / 96 query heads), RoPE — architecture follows Llama 3 closely [§2.1, Abstract].
Depth-scaled sandwich normalization (DSSN) places a second RMSNorm after each sub-layer’s output (sandwiching the residual), with the post-norm gain γ initialized as a depth-scaled value rather than 1 — claimed to eliminate loss spikes in deep models [§2.2, Abstract].
Depth-scaled parameter initialization scales the std of each weight matrix’s init by 1/√(layer_index) (later/deeper layers initialized smaller), claimed to keep activation magnitudes bounded across the 94-layer stack and complement DSSN [§2.3].
The training curve over 13.2T tokens reportedly contains no loss spikes and no manual rollbacks — presented as the empirical headline that DSSN + depth-scaled init are sufficient for stable deep-model training on bf16 NPUs [§3, training-curve figure].
Pre-training corpus: 13.2T tokens; tokenizer is a 153K-vocabulary tokenizer (the “small-model-lab” tokenizer per the whistleblower account in True Story of Pangu — Huawei Noah's Ark whistleblower statement) [§3.1].
Trained on 8,192 Ascend 910B NPUs with parallelism stack tuned for the cluster — pipeline-parallel + tensor-parallel + sequence-parallel + expert-parallel composition [§3.2].
Reported benchmark results: outperforms Llama 3 405B and Mistral Large 2 123B on MMLU, GSM8K, MATH, HumanEval, and other standard suites; competitive with sparse 500B+ models on average [Table 3].
Post-training: long-CoT SFT + multi-stage RLHF for reasoning enhancement; specific reasoning benchmark gains reported in §4 [§4, Table 4].

Method

DSSN and depth-scaled init are presented as a co-designed pair for deep dense pre-training on bf16 Ascend NPUs. DSSN extends standard sandwich normalization (Pre-RMSNorm before the sub-layer + Post-RMSNorm after, both inside the residual) by initializing the Post-RMSNorm’s γ to a depth-dependent value rather than 1 — making the post-norm rescaling start small at deep layers and grow during training. The claimed mechanism is that this prevents deep-layer activation magnitudes from growing geometrically across the 94-layer stack while still allowing the residual contributions to recover full scale.

Depth-scaled parameter initialization is the matching weight-side fix: each linear’s init std is σ₀ / √(2·layer_index) (Pangu’s variant of TinyInit). Together the two pieces are presented as eliminating the architectural drivers of loss spikes in deep dense bf16 training, allowing the 135B model to be trained over 13.2T tokens with no manual intervention.

The system stack is the other major contribution: a from-scratch training framework on 8,192 Ascend 910B NPUs (8-stage pipeline parallel + 8-way tensor parallel + sequence parallel), with custom communication kernels and memory-management strategies tuned to the NPU’s HBM topology and the inter-NPU interconnect.

Results

Model averaged ahead of Llama 3 405B / Mistral Large 2 123B on MMLU, GSM8K, MATH, HumanEval, MBPP, C-Eval, CMMLU [Table 3].
Long-context evaluation: extends from 8K → 128K via training-time RoPE base-frequency rescaling; reports Needle-in-a-Haystack and LongBench numbers at the 128K context length [§4.2, Table 5].
Post-training: notes that long-CoT SFT followed by multi-stage RLHF moves the reasoning benchmark scores (MATH-500, AIME, GPQA-Diamond) above the base-model baseline by 10–20 points [§4.3, Table 6].
The training-curve figure is the contribution most cited by downstream methodology work — a 94-layer dense bf16 run completing 13.2T tokens with no loss spikes [§3, training-curve figure].

Why it’s interesting

Pangu Ultra is the source paper for two architectural pieces — DSSN and depth-scaled init — that have shown up as candidate stability primitives in subsequent training-stability work. The post-hoc disclosure from True Story of Pangu — Huawei Noah's Ark whistleblower statement is an integrity event that changes how the wiki should treat citations to this paper: per Jiaming’s note, sandwich norm itself is a robust enough idea that the multiple independent papers proposing variants (How to Set the Learning Rate for Large-Scale Pre-training? and the architectural-stability cluster on Training stability at scale) make it safe to keep using, while depth-scaled init’s load-bearing role rests primarily on this paper’s reported training curve and so needs independent replication before being adopted. The paper also stands as the “real” from-scratch Pangu run that the whistleblower explicitly defends — distinct from the Pangu-MoE 72B release whose from-scratch claim is the actively-contested allegation.