The Trinity Manifesto — Arcee's open-weight MoE family (Mini + Nano + Large roadmap)

TL;DR

Arcee’s “Trinity Manifesto” announces the first two open-weight checkpoints in the Trinity MoE family — Trinity Mini (a fully post-trained 26B-A3B-class reasoning model) and Trinity Nano Preview (an experimental 6B-A1B chat model with 800M non-embedding active parameters across 56 layers and 128 experts), both Apache 2.0, with Trinity Large (420B, 13B active) still training on 2048 NVIDIA B300 GPUs for a January 2026 release. The post is the architecture spec for the family — Arcee names the stack afmoe and lays out every load-bearing choice: gated attention (Qwen-G1) + QK-Norm + GQA + 3:1 local/global with NoPE globals, DeepSeekMoE fine-grained experts + 1 shared expert with sigmoid + aux-loss-free routing, depth-scaled sandwich norm with gamma_init = 1/sqrt(L), Muon optimizer via Microsoft’s Dion distributed implementation, and a 10T-token three-phase WSD curriculum on a modified TorchTitan in bf16 on 512 H200s. Positioned as the first frontier-aimed open-MoE family trained end-to-end in the U.S., explicitly framed against the Chinese-lab dominance of open MoE checkpoints (Qwen, DeepSeek).

Key claims

Trinity Mini (26B-A3B) and Trinity Nano Preview (6B-A1B, 800M non-embedding active across 56 layers × 128 experts) are released today under Apache 2.0; Trinity Large is 420B with 13B active and is currently training on 2048 B300 GPUs for January 2026 [§“Trinity Nano and Trinity Mini are available now.”].
The attention stack is Grouped-Query Attention with QK-Norm (RMSNorm on Q/K before SDPA) plus G1 gated attention from the Qwen paper — output is elementwise-gated out_proj(sdpa_out * sigma(gate_proj(x))) — combined with a 3:1 local/global pattern where global layers use NoPE and locals use RoPE [§“The attention mechanism…”].
Normalization is depth-scaled sandwich norm: each sublayer computes output = x + norm(module(norm(x))), with norm-layer gamma initialized to 1/sqrt(L) (where L is total layer count) for stable training at depth, plus a pre-LM-head norm [§“For layer normalization…”].
MoE layers follow the DeepSeekMoE design: 128 routed experts, 8 active per token, plus 1 always-on shared expert; the first two layers are dense rather than sparse — Arcee reports this improves training stability early [§“MoE layers…”].
Routing uses sigmoid + normalization (from DeepSeek-V3), not softmax, paired with the aux-loss-free load balancing scheme — an independently updated bias term decides routing but is excluded from the per-expert contribution weighting [§“For routing…”].
Weight init is truncated normal at std 0.5/sqrt(dim); embedding output is multiplied by sqrt(dim) in the forward pass [§“Initialization…”].
The optimizer is Muon, using the distributed implementation from Microsoft’s Dion repository, with adjusted_lr = lr * sqrt(max(1, fan_out/fan_in)) for shape-transfer learning-rate scaling; Adam LR and Muon LR are swept separately under WSD; no weight decay on embeddings [§“Training with Muon…”].
Training uses a modified TorchTitan in bf16 with HSDP parallelism; Nano and Mini both trained on 512 H200s at global batch 8192 sequences × 4096 tokens [§“Training runs on…”].
Context extension expanded only the global attention layers — Nano trained at 256K sequence length (inference at 128K), Mini at 128K — Arcee reports the model learned the extended length very quickly under this scheme [§“Only the global attention layers were expanded…”].
Pretraining is 10T tokens organized in three phases of progressively higher quality and STEM concentration: 7T (phase 1), 1.8T (phase 2), 1.2T (phase 3); the mix reuses Arcee’s AFM dataset plus substantially more math and code [§“Trinity Nano and Mini train on 10T tokens…”].
Data is provided by Datology (continuation of the AFM-4.5B partnership); compute and systems support for the 2048 B300 Trinity-Large training is from Prime Intellect, who served the H100 clusters Datology used for synthetic data and helped scale Arcee’s training stack to frontier size [§“Datology continued to be a key partner…”].
Trinity is positioned as the first serious open-weight MoE family trained end-to-end in the U.S. by an American company, explicitly contrasting with the recent dominance of Chinese-lab open-MoE releases (Qwen, DeepSeek) and with the U.S. pattern of “polishing other people’s checkpoints” [§“Over the last year…”].
Nano Preview is explicitly an experimental personality-forward chat model that pushes sparsity (800M non-embedding active) — Arcee states it may be unstable in edge cases and is not a thinking model, and is download-only (not on the Arcee API) [§“Trinity Mini is the fully post-trained…”].

Method

afmoe is a decoder-only sparse MoE transformer family with a uniform recipe across Nano (6B-A1B), Mini (26B-A3B), and Large (420B-A13B). The block is a depth-scaled sandwich-norm transformer with GQA + QK-Norm + G1 gated attention and a 3:1 local-global attention ratio (locals with RoPE, globals without — NoPE). MoE layers use 128 fine-grained routed experts + 1 shared expert with top-8 routing through a sigmoid gate; load balancing follows DeepSeek-V3’s aux-loss-free bias-update scheme. The first two layers are dense to give a shared representational foundation before specialization begins. The training stack is Muon (via Microsoft’s Dion distributed implementation) on a modified TorchTitan in bf16 with HSDP and a WSD learning-rate schedule; the shape-transfer rule for LR is adjusted_lr = lr * sqrt(max(1, fan_out/fan_in)). Pretraining is 10T tokens in a three-phase curriculum (7T → 1.8T → 1.2T) with increasing STEM concentration; context extension is performed by expanding only the global attention layers (256K for Nano, 128K for Mini).

The release also outlines the data-and-systems supply chain: Datology continues as the data partner from AFM-4.5B forward, Prime Intellect provides the H100 clusters used for synthetic-data generation and the cluster engineering for the 2048 B300 Trinity-Large run. The post does not include benchmarks for Nano or Mini — it is an architecture / process announcement, with the full technical report deferred to Trinity Large’s release.

Results

The post is a release announcement, not an evaluation report, so concrete numbers are scale and config rather than benchmarks:

Trinity Nano Preview: 6B-A1B, 800M non-embedding parameters active per token, 56 layers, 128 experts, trained at 256K sequence length (inference at 128K) on 512 H200 GPUs at global batch 8192 × 4096 tokens.
Trinity Mini: 26B-A3B, fully post-trained for reasoning, 128K sequence length, same training infrastructure as Nano.
Trinity Large: 420B total / 13B active, currently training on 2048 NVIDIA B300 GPUs, ETA January 2026 with a full technical report to accompany release.
Pretraining corpus: 10T tokens across three phases (7T / 1.8T / 1.2T) blending Arcee’s curated AFM dataset with additional math + code.
Lineage: Builds on AFM-4.5B (4.5B dense, ~8T curated tokens, Datology partnership) — explicitly described as the feasibility experiment whose lessons (especially on math/code data) feed directly into Trinity.

For benchmarked numbers on the Trinity Large variant, see Trinity Large: An Open 400B Sparse MoE Model (the January 2026 release, ultimately shipped at 400B / 13B-active / 17T tokens / 33 days on the same 2048 B300 cluster) and Trinity-Large-Thinking: Scaling an Open Source Frontier Agent (the post-trained reasoning variant).

Why it’s interesting

Trinity is a complete “open architecture spec” datapoint for a U.S.-trained frontier MoE — and it composes three threads the wiki has been tracking in isolation. On the routing side, afmoe adopts the most-converged 2025 stack: DeepSeekMoE fine-grained + shared experts, sigmoid + aux-loss-free balancing, and 2 leading dense layers — a deployment-level vote for the recipe that Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models and Quantile Balancing: A Hyperparameter-Free MoE Load Balancing Method are still actively refining. On the stability side, it pairs QK-Norm + G1 gated attention with the depth-scaled sandwich norm gamma_init = 1/sqrt(L) rule — sitting squarely on the architecture-first stability side of the Training stability at scale fork articulated by How to Set the Learning Rate for Large-Scale Pre-training? and A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training. On the optimizer side, it is one of the largest publicly-disclosed Muon-at-pretraining-scale deployments to use the Dion distributed implementation specifically — directly addressing the “distributed Muon is the missing piece” gap surfaced in Addition of Muon optimizer to torch.optim — PyTorch team welcomes PR (Issue #148819) and microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers.

What’s unusual is the manifesto framing — a written architecture spec released before the flagship model, with the explicit positioning that the family was designed top-down (“leap directly toward the frontier and work backward”). The Trinity-Large 400B-class run that ultimately shipped is filed separately (Trinity Large: An Open 400B Sparse MoE Model); this post is the design-decision predecessor with slightly different numbers (420B → 400B, 6 vs 2 leading dense layers) — useful for understanding how the recipe evolved during the run itself.