ZAYA1-8B Technical Report

TL;DR

ZAYA1-8B is a reasoning-focused MoE from Zyphra with 700M active / 8B total parameters trained entirely on AMD compute, networking, and software. With under 1B active parameters it matches or exceeds DeepSeek-R1-0528 on several math and coding benchmarks and stays competitive with much larger open-weight reasoning models. The recipe puts reasoning data in from pretraining (using an answer-preserving trimming scheme rather than reserving reasoning for post-training) and uses a four-stage RL cascade. The headline TTC contribution is Markovian RSA, a recursive parallel-aggregation scheme that carries forward only a bounded 4K-token “reasoning tail” between rounds and lifts ZAYA1-8B to 91.9% on AIME’25 and 89.6% on HMMT’25.

Key claims

700M active / 8B total MoE built on Zyphra’s MoE++ architecture matches or exceeds DeepSeek-R1-0528 on several math and coding benchmarks despite a ~10× smaller active-parameter count [Abstract].
Full-stack AMD platform — compute, networking, and software — used for pretraining, midtraining, and SFT [Abstract].
Reasoning data is included from pretraining onward via an answer-preserving trimming scheme, rather than reserved for a separate reasoning post-training stage [Abstract].
Post-training uses a four-stage RL cascade: (1) reasoning warmup on math and puzzles; (2) a 400-task RLVE-Gym curriculum; (3) math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; (4) behavioral RL for chat and instruction following [Abstract].
Markovian RSA is a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only a bounded-length reasoning tail between rounds [Abstract].
With a 4K-token carry-forward tail, Markovian RSA lifts ZAYA1-8B to 91.9% on AIME’25 and 89.6% on HMMT’25, narrowing the gap to Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High [Abstract].

Method

ZAYA1-8B is a sparse MoE in the “MoE++” lineage (Zyphra’s in-house variant) sized at 8B total / 700M active. Training is split across pretraining → midtraining → SFT → a four-stage RL cascade, all on AMD hardware end-to-end. The pretraining mix includes reasoning data from the start, processed through an answer-preserving trimming step that keeps short-form correct answers intact while bounding chain-of-thought length.

The RL cascade is structured rather than monolithic: stage 1 warms up on math + puzzle problems with verifiable rewards; stage 2 swaps in RLVE-Gym, a 400-task curriculum of verifiable environments; stage 3 introduces TTC (test-time-compute) traces and synthetic competitive-programming environments built from reference solutions; stage 4 finishes with behavioral RL for instruction-following and chat formatting.

Markovian RSA is the inference-time component. Instead of carrying full prior reasoning between aggregation rounds — which is what limits naive recursive self-aggregation — Markovian RSA truncates the carried-forward context to a bounded “reasoning tail” (4K tokens in the reported runs). This makes each round a Markovian step in the aggregated reasoning state, decoupling TTC budget from accumulated context length.

Results

AIME’25 with Markovian RSA + 4K tail: 91.9% [Abstract].
HMMT’25 with Markovian RSA + 4K tail: 89.6% [Abstract].
“Matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks” at <1B active parameters [Abstract].
“Remains competitive with substantially larger open-weight reasoning models” — the report frames the AIME/HMMT numbers as narrowing the gap to Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High [Abstract].

(The abstract is the only material available at filing time; specific RL-stage ablations, MoE++ architectural details, and full benchmark tables live in the PDF body and are not summarized here.)

Why it’s interesting

ZAYA1-8B is unusual on three axes at once. (1) It is the first filed reasoning-focused MoE trained end-to-end on a full AMD stack, which makes it a concrete datapoint against the assumption that frontier reasoning training requires NVIDIA — a counterweight to the systems-side recipes in Scalable Training of Mixture-of-Experts Models with Megatron Core and Scaling and Optimizing Frontier Model Training (Fireworks AI) which both presume CUDA / B200 stacks. (2) The four-stage RL cascade is a more structured instance of the GRPO-family recipes on Reasoning RL — RLVE-Gym (400 verifiable tasks) is in the same family as the SETA / Toolathlon-GYM / OpenReward platforms tracked under RL Environment Platforms but is reported as a curriculum stage rather than a standalone benchmark. (3) Markovian RSA fits cleanly as a new datapoint on Inference-Time Scaling: it is recursive parallel aggregation with a bounded-length carry-forward, which is operationally a hybrid of the scaffold-side (RLM-style sub-LM dispatch) and sequence-budget-compression axes on that page — the “Markovian” framing decouples TTC budget from accumulated context length the way DMS does for KV memory, but it lives in the aggregation scaffold rather than the cache.