LongCat-Flash Technical Report

TL;DR

LongCat-Flash is Meituan’s 560B-total / ~27B-activated MoE non-thinking foundation LLM, the backbone the team later extended into LongCat-Flash-Omni Technical Report. It combines two architectural bets — zero-computation experts with a PID-controlled bias for dynamic per-token compute (18.6–31.3B activated), and Shortcut-connected MoE (ScMoE) which decouples MoE communication from FFN compute on the critical path — with a comprehensive scaling toolkit: hyperparameter transfer via width scaling, model-growth initialization (train a 14-layer model, stack to 28 layers and continue), a router-stability / hidden-z-loss / optimizer-tuning stability suite, and deterministic computation for SDC detection. Trained on >20T tokens in 30 days at 98.48% time-availability with no irrecoverable loss spikes; serves at >100 TPS on H800 for $0.70 per million output tokens.

Key claims

560B total parameters / 18.6B–31.3B activated per token / ~27B average, achieved by mixing zero-computation experts (identity mappers, free) with regular FFN experts in the same MoE pool — the router selects K=12 experts per token from a pool of 512 FFN + 256 zero-computation experts [§2.1, §2.4 Model Configurations].
The activated-parameter budget is held to its target via an expert bias updated by a PID controller over token-routing frequency, refining the aux-loss-free strategy of DeepSeek-V3 / Wang et al. 2024 — empirically converges to the target within ~20B tokens with <1% fluctuation while preserving high standard-deviation across tokens (i.e., the model genuinely allocates divergent compute) [§2.1.1, Fig. 3(b)–(c)].
The PID update excludes zero-computation experts from bias updates; their density is regulated only via a global ratio constraint baked into the device-level load-balance loss (a separate auxiliary loss, derived from DeepSeek-V3’s recipe but extended to put all zero-computation experts into one virtual group) [§2.1.1, §2.1.2 Eqs. 3–5].
Shortcut-connected MoE (ScMoE) reorders the layer so the dense FFN of layer N runs in parallel with the dispatch/combine all-to-all of the MoE at layer N+1 — does not compromise loss (verified at 2.4B–16B, 3B–20B, 15B–193B scales with MLA/MHA/GQA) and roughly halves theoretical TPOT vs DeepSeek-V3 in inference [§2.2, Fig. 4].
Two variance-alignment architectural fixes: a scale-correction for MLA that rescales low-rank Q/K paths to match the full-dimension rotary-key variance (else attention scores destabilize as ranks vary) [§2.3.1, Fig. 5(a)]; and a variance-compensation factor m·n applied to fine-grained expert outputs that counteracts the two-source variance reduction from gating dilution + dimension reduction [§2.3.2 Eq. 8].
Hyperparameter transfer via width scaling: optimal init variance and LR are found on a 768-width proxy and transferred to the target (width-scale factor m=8) using Standard Parameterization “Adam LR Full Align” rules — depth, sparsity, batch size are invariant under the transfer [§3.1.1, Table 1].
Model-growth initialization: a 14-layer model (half depth) is trained on tens of billions of tokens, then stacked twice to give the 28-layer target; all training state — sample counter, LR schedule — is preserved across the stacking step. The post-stack loss trajectory rises briefly, then converges to a lower loss than random init [§3.1.2, Fig. 5(b)].
Hidden z-loss is a new stability auxiliary, by analogy with router z-loss (Zoph et al. 2022) — suppresses massive activations during LLM pretraining [§3.1.3 Activation Stability].
Router stability is diagnosed via two new metrics — Router Weight Similarity (pairwise cosine of expert weight vectors, high = LB-dominated collapse) and a Gradient Norm Ratio between the LM-loss and LB-loss gradients on the batch-averaged expert probability — and the LB coefficient α is tuned to keep R_grad below a small threshold (≤0.01) [§3.1.3 Router Stability].
28 layers, hidden 6144, 64 MLA heads (head-dim 128), KV compression dim 512, query compression dim 1536, dense-FFN intermediate 12288, per-expert FFN intermediate 2048, vocab 131,072 [§2.4 Model Configurations].
Multi-Token Prediction head is a single dense layer (not MoE), introduced mid-training rather than from the start; achieves >90% acceptance in evaluation [§2.4 MTP, Table 5].
Deterministic computation is engineered into the training stack to guarantee exact reproducibility and enable detection of Silent Data Corruption (SDC) during the run; combined with the other interventions, 98.48% time-availability and zero irrecoverable loss spikes across the full 20T-token pretraining [§1, §3.1.3].
Three-stage pretraining curriculum: (1) ~20T tokens at 8K context to build a base, (2) reasoning + coding mid-training on trillions more, (3) context extension to 128K [§3].
Multi-agent synthetic-data framework for agentic post-training: defines task difficulty along three axes — information processing, tool-set complexity, user interaction — and uses specialized controllers to generate tasks requiring iterative reasoning + environment interaction [§1].
As a non-thinking model: ArenaHard-V2 86.5, TerminalBench 39.5, τ-Bench 67.7; competitive with DeepSeek-V3.1 and Kimi-K2 at fewer activated parameters and faster inference [§1, Fig. 1].
Two new in-house evaluation benchmarks shipped alongside: Meeseeks (multi-turn instruction-following with an iterative feedback simulator) and VitaBench (real-world business-task agentic eval) — designed to reduce contamination risk versus open benchmarks [§1].

Method

LongCat-Flash is a 28-layer MoE LLM where each layer holds two MLA attention blocks plus multiple heterogeneous FFN blocks, with one shortcut connection wiring the first MLA’s output directly into the MoE — so the MoE’s expensive dispatch/combine all-to-all can be fully overlapped with the in-between dense FFN’s compute. The MoE pool itself contains 512 FFN experts plus 256 zero-computation (identity) experts; the router picks K=12 from the combined pool, which means the realized FLOPs per token vary with how many of the K picks land on zero-computation experts. A PID controller on each FFN expert’s running bias keeps the average count of activated FFN experts near a target (yielding the ~27B average activation), while a separate device-level LB loss prevents extreme intra-batch imbalance. Two variance-alignment tricks — scale-correction inside MLA’s asymmetric low-rank Q/K paths, and an m·n compensation factor on fine-grained expert outputs — keep these design choices stable across width scaling.

Training is staged: hyperparameter transfer fixes init/LR via a 768-width proxy, model-growth initialization warms up a 14-layer half-depth twin on ~10B tokens then stacks it (carrying sample counter + LR schedule), and the run proceeds through 20T tokens at 8K context, then reasoning/code mid-training, then 128K context extension. Stability is enforced as a suite rather than a single fix: router stability is monitored via expert-weight cosine similarity and the LM/LB gradient-norm ratio (with the LB coefficient tuned to keep the ratio ≤ 0.01); activation stability uses the new hidden z-loss (router z-loss’s hidden-state analogue) to suppress massive activations; and deterministic computation enables SDC detection.

Results

Pretraining throughput: full 560B model trained on >20T tokens in 30 days at 98.48% time-availability, no irrecoverable loss spikes [§1].
Inference: >100 TPS on H800 deployment at $0.70 per million output tokens [§1].
Model-growth initialization: the half-depth → stacked-28-layer trajectory converges to lower loss than random init at 6B-activated-class experiments, with characteristic loss-rises-then-falls signature [§3.1.2, Fig. 5(b)].
PID-controlled budget control reaches target within ~20B tokens, fluctuation <1%; per-token activation std stays high (genuine dynamic compute, not collapse to uniform) [§2.1.1, Fig. 3].
ScMoE matches loss curves of non-ScMoE baseline across three scale/attention combos; cuts theoretical TPOT roughly in half vs DeepSeek-V3 [§2.2, Fig. 4].
Benchmark scores as a non-thinking model: ArenaHard-V2 86.5, TerminalBench 39.5, τ-Bench 67.7, with claimed parity with DeepSeek-V3.1 / Kimi-K2 at smaller activation footprint [§1].
MTP head: >90% acceptance rate at inference [Table 5].

Why it’s interesting

This is the load-bearing paper behind LongCat-Flash-Omni Technical Report — same ScMoE + zero-computation-experts backbone, scaled down to 27B-activated for the omni-modal variant. Reading the two together makes clear that the Omni paper is essentially a multimodal port of this architecture, and the PID controller / hidden z-loss / model-growth init claims tracked here are inherited by the Omni model without restatement. It also drops directly into MoE Routing Design as the largest open MoE that combines a non-standard expert pool (regular + zero-computation) with a control-theoretic (rather than aux-loss or aux-loss-free) bias-update recipe — and the per-device LB loss with a virtual zero-computation group sits between micro-batch LB (Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models‘s objection) and global-batch LB (its prescription).

On the stability side, hidden z-loss extends the lineage tracked on Training stability at scale — the same “outliers / massive activations are the failure mode” thesis that A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training and The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks examine post-hoc — but addresses it as a training-time auxiliary rather than via architectural rescaling. The variance-alignment story (scale-correction for MLA, m·n factor for fine-grained experts) is a third architecture-side fix in the same family. And the model-growth initialization (train half-depth → stack → continue with full training state) is the strongest open datapoint for depth doubling as a pretraining accelerant — a concrete entry for the Hyperparameter scaling laws open question “does µP-style transfer extend to depth-direction scaling?” The answer here is empirical and pragmatic: don’t transfer, just stack and continue.