Trinity Large: An Open 400B Sparse MoE Model

TL;DR

Arcee AI releases Trinity Large, a 400B-total / 13B-activated sparse MoE language model (4-of-256 expert routing, ≈1.56% activation ratio) under Apache 2.0. The full 17T-token pretraining run completed in ~33 days on 2048 NVIDIA B300 GPUs at a stated cost of roughly $20M, with a 30-person team — Arcee positions this as “engineering through constraint” and as the first publicly disclosed pretraining run at this scale on B300 Blackwell. The release ships three checkpoints from the same run — Trinity-Large-Preview (lightly post-trained for chat), Trinity-Large-Base (full 17T pretraining checkpoint), and Trinity-Large-TrueBase (10T-token checkpoint with no instruction data and no LR annealing) — explicitly so researchers can study what high-quality pretraining alone produces at this scale. A reasoning variant is in post-training.

Key claims

Trinity Large is a 400B-total / 13B-activated sparse MoE with 256 experts and 4 active per token, an aggressive sparsity ratio comparable only to Llama-4-Maverick among peers [§Architecture].
The team increased the number of dense (non-MoE) layers from 3 to 6 to stabilize routing at this sparsity level, trading a slightly different total parameter target (originally 420B) for routing stability [§Architecture].
Pretraining ran 17T tokens on 2048 NVIDIA B300 GPUs over roughly 30–33 days for ≈$20M, billed as the largest publicly disclosed pretraining run on B300 hardware to date [§Training].
Three checkpoints from a single training run are released: Preview (post-trained for chat), Base (17T-token full pretraining), and TrueBase (10T-token mid-pretraining checkpoint with no instruct data and no LR anneal) — TrueBase is positioned as a research baseline for “what did the model learn from data alone?” [§Releases].
Reported architectural / recipe innovations include SMEBU (Soft-clamped Momentum Expert Bias Updates) for MoE load balancing and Muon as the pretraining optimizer in place of AdamW [§Training, follow-up Marktechpost].
Trinity-Large-Base is positioned as matching or exceeding Llama-4-Maverick and GLM-4.5 Base across math, coding, scientific reasoning, and knowledge benchmarks [§Evaluation].
Inference is ~2–3× faster than peers in the same parameter class on equivalent hardware, attributed to the 1.56% activation ratio plus optimized attention [§Inference].
The Preview release is explicitly not a reasoning model; a reasoning-tuned variant (later released as Trinity-Large-Thinking, April 2026) was undergoing further post-training at blog publication [§Reasoning].

Method

Trinity Large is a decoder-only sparse Mixture-of-Experts transformer trained with Arcee’s in-house FMoE stack (AFMoE). The router uses top-4-of-256 expert routing per token, giving 13B active parameters from a 400B-total pool (≈1.56% activation). Six layers near the input/output are dense rather than MoE, a choice Arcee reports was load-bearing for routing stability at this sparsity. Training uses the Muon optimizer (rather than AdamW) and a custom MoE load-balancing scheme (SMEBU — Soft-clamped Momentum Expert Bias Updates) intended to prevent expert collapse without the instability of standard auxiliary load-balance losses. Attention is a mix of local and global with gating, and the model is pretrained to support long context (Preview card lists up to 512K).

The training run is 17T tokens on 2048 NVIDIA B300 (Blackwell) GPUs, completed in roughly 30–33 days. The three checkpoints are forks of the same run: TrueBase at the 10T-token point (no LR anneal, no instruction data), Base at the end of the full 17T-token run including LR anneal, and Preview as a lightly post-trained chat variant on top of Base. Distribution is Apache 2.0 on Hugging Face, with day-0 hosting on OpenRouter (free during the preview window through at least February 2026) and integrations with Kilo Code, Cline, and OpenCode.

Results

Headline architecture and infra numbers, from the blog and the accompanying Interconnects interview:

Scale: 400B total, 13B active, 256 experts × 4 active, ~1.56% activation ratio.
Compute: 2048× B300 GPUs, 30–33 days, ~$20M, 30-person team.
Tokens: 17T (Base), 10T (TrueBase mid-checkpoint).
Inference: ~2–3× faster than peers in the 400B-class on equivalent hardware (under 8-bit quantization, 128K context for the Preview API).
Benchmarks: Arcee positions Trinity-Large-Base as matching/exceeding Llama-4-Maverick and GLM-4.5 Base across math, coding, scientific reasoning, and knowledge tasks. Third-party coverage notes Preview trades some benchmark performance for chat fluency/creativity vs Base.
Followup: Trinity-Large-Thinking (April 2026) is the reasoning variant, reportedly #2 on PinchBench behind Claude Opus, with a 262,144-token context window on OpenRouter.

Why it’s interesting

A clean datapoint for what a 30-person, ~$20M, 33-day run on 2048 B300s can produce at frontier scale — and a rare release of a raw 10T-token mid-pretraining checkpoint (TrueBase) without LR anneal or instruction data, which is exactly the kind of artifact one would want for studying parametric-memory geometry, knowledge distribution across experts, or alignment-tax ablations. The SMEBU load-balancing scheme and the Muon-at-pretraining-scale choice are both worth tracking as recipes that may generalize: SMEBU because expert-collapse mitigation is the obvious sore spot at 1.56% activation ratios, and Muon because this is one of the largest pretraining runs publicly disclosed using it rather than AdamW. For Open foundation-model releases this also extends the catalog with a new pattern: shipping multiple checkpoints from the same run (not multiple post-trained variants) as the open-release primitive.