MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

TL;DR

MiniMax-M1 is an open-weight 456B-total / 45.9B-activated MoE reasoning model with a hybrid attention design — one softmax-attention block per seven lightning-attention (linear-attention) blocks — that natively supports a 1M-token context and 80K-token output. The headline efficiency claim is that at 100K-token generation length M1 uses ~25% of DeepSeek-R1’s FLOPs, making large-scale RL on long reasoning chains tractable. Training contributions are CISPO, a new on-policy RL algorithm that clips importance-sampling weights instead of token updates, and a set of recipes for stabilizing RL on a hybrid-attention architecture; the full RL stage ran in 3 weeks on 512 H800s for ~$534,700.

Key claims

The model is a hybrid Mixture-of-Experts where one softmax-attention block follows every seven transnormer blocks running Lightning Attention, on top of the 456B-total / 45.9B-activated / 32-expert MiniMax-Text-01 backbone [§1].
At 100K-token generation, M1 consumes ~25% of DeepSeek-R1’s FLOPs; at 64K tokens it consumes <50% [§1, Fig. 1 (Right)].
Natively supports 1M input tokens and 80K output tokens — 8× DeepSeek-R1’s input window and an order of magnitude beyond all other open-weight LRMs at release [§1, Table 1].
CISPO clips the importance-sampling weight in a REINFORCE-style objective rather than clipping the token-level update; this preserves gradient contributions from low-probability “reflective” tokens (e.g. However, Recheck, Wait) that PPO/GRPO drop after the first on-policy update [§3.1, Eq. 4, Eq. 5].
In a Qwen2.5-32B-base zero-RL controlled study on AIME 2024, CISPO matches DAPO performance with 50% of the training steps and outperforms both DAPO and GRPO at the same step count [§3.1, Fig. 2].
The hybrid architecture exposes RL-specific failure modes that don’t appear in dense softmax models: a precision mismatch between training and inference kernels driven by high-magnitude LM-head activations prevents reward growth; fix is to run the LM head in FP32, raising the training/inference probability correlation from ~0.9 to ~0.99 [§3.2, Fig. 3].
AdamW hyperparameters need re-tuning for the hybrid model — gradient magnitudes span 1e-18 to 1e-5 with most below 1e-14, motivating β1=0.9, β2=0.95, eps=1e-15 (vs. VeRL default eps=1e-8 which causes non-convergence) [§3.2].
Pathologically repetitive generations are caught early by halting when 3,000 consecutive tokens each exceed probability 0.99, since once a model enters a repetitive cycle per-token probabilities saturate [§3.2].
The full RL training cycle (both variants of M1) completed in 3 weeks on 512 H800 GPUs at ~$0.53M USD rental cost [§1, §3].
Continual pretraining adds 7.5T tokens (70% STEM/code/book/reasoning) with constant LR 8e-5 for 2.5T tokens then decayed to 8e-6 over 5T; context extension is staged from 32K to 1M to avoid the gradient explosion that direct extension induces in the hybrid-lightning architecture [§2.1].
M1-80K outperforms M1-40K on complex math/coding — explicit scaling-test-time-compute evidence — and the 40K checkpoint is an intermediate phase of the 80K training [§1].
On agentic and long-context benchmarks M1-80K outperforms Gemini 2.5 Pro on TAU-Bench and surpasses OpenAI o3 and Claude 4 Opus on long-context understanding; on math/coding competitions it lags DeepSeek-R1-0528 [§1].
RL data spans verifiable (math: 50K curated competition problems with pass@10 ∈ (0, 0.9); logic: 53K samples from 41 SynLogic tasks with model-calibrated difficulty; competitive programming: 30K problems; SWE-bench-style sandboxed software engineering with execution-based rewards) and non-verifiable (25K STEM/QA/creative-writing samples with reward-model feedback) [§4].
Released openly: weights on HF, code on GitHub, vLLM + Transformers deployment guides, two variants (M1-40K and M1-80K thinking budgets) [§1].

Method

MiniMax-M1 starts from MiniMax-Text-01 (456B-total / 45.9B-active MoE, 32 experts) and inherits its hybrid attention: every eighth transformer block uses softmax attention, the other seven use Lightning Attention — an I/O-aware linear-attention variant (Qin et al. 2024b). The linear-attention blocks are what enable the FLOP advantage at long generation lengths and what make 1M-context training feasible.

Pretraining is reused from MiniMax-Text-01 plus 7.5T tokens of continual pretraining on a STEM/code/book/reasoning-weighted corpus, with staged context extension (32K → 1M) to avoid a gradient-explosion failure mode the team attributes to differing decay rates across early vs. late lightning-attention layers. SFT on long-CoT data (~60% math/coding) injects reflection-style reasoning patterns as a cold start for RL.

The RL stage is where the novel algorithmic content lives. CISPO is derived by going back to REINFORCE with corrected importance sampling and adding a clip on the IS weight (Eq. 5) instead of on the token-level update; combined with GRPO’s group-relative advantage and token-level loss normalization (Liu et al. 2025b), this becomes Eq. 4. The motivation is empirical: in initial zero-RL experiments on the hybrid architecture, GRPO clipped out exactly the high-update reflective tokens (However, Recheck, Wait) that are rare and low-probability under the base model but functionally crucial for long-CoT reasoning, and DAPO’s relaxed upper bound did not fix this in the team’s setup with 16 off-policy rounds per generation batch. CISPO never drops a token’s gradient and tunes only the upper clip; KL penalty is omitted, following recent practice. A general formulation (Eq. 6) introduces a token-wise mask so PPO, GRPO and CISPO can all be expressed in one objective. Three architecture-specific RL recipes accompany CISPO: FP32 LM-head precision to align training/inference probabilities, re-tuned AdamW (β2=0.95, eps=1e-15), and early-truncation-on-repetition.

RL data is curated across four verifiable domains (math, logic via SynLogic, competitive programming, sandboxed SWE) with rule-based rewards plus 25K non-verifiable general samples with reward-model feedback. SWE training uses a containerized sandbox where test-case pass/fail directly drives the reward.

Results

M1-80K outperforms M1-40K on complex math and coding, and surpasses earlier open-weight LRMs (DeepSeek-R1, Qwen3-235B) overall, with the largest advantages on SWE, tool-use, and long-context tasks [§1, Fig. 1 (Left)].
Against the latest DeepSeek-R1-0528, M1 lags on math/coding competitions but matches or exceeds it on more realistic tool-using and long-context scenarios; outperforms Gemini 2.5 Pro on TAU-Bench and beats OpenAI o3 / Claude 4 Opus on long-context understanding benchmarks [§1].
CISPO ablation on Qwen2.5-32B-base / AIME 2024: matches DAPO at 50% training steps, beats both DAPO and GRPO at equal steps [§3.1, Fig. 2].
Training and inference token probabilities go from correlation ~0.9 to ~0.99 after switching the LM head to FP32, which is the difference between reward growth and stalled training [§3.2, Fig. 3].
Full RL on 512 H800s for 3 weeks at ~$534,700 rental cost — an order of magnitude below initial estimates [§1, §3].

Why it’s interesting

This is the first filed reasoning model to push a linear-attention-heavy hybrid architecture all the way to a frontier RL run, and it ships with a new RL algorithm whose motivation is concretely about hybrid-attention pathologies — directly relevant to the wiki’s growing thread on linear-attention reasoning (Mamba-3: Improved Sequence Modeling using State Space Principles, MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens) where the open question has been whether these architectures can compete on reasoning at scale. CISPO’s “clip the IS weight, not the token” framing is also a clean addition to the reasoning-RL design space already populated by GRPO/DAPO/CISPO discussions on Reasoning RL — and the empirical finding that PPO/GRPO clipping silently drops reflective tokens is the kind of mechanism-level result that would otherwise stay folk knowledge. The 25%-FLOPs-at-100K-tokens claim is the strongest existence proof so far for the LLM Inference Efficiency thesis that softmax attention is the dominant cost at long generation lengths, and it complements the cache-compression axis of Inference-Time Hyper-Scaling with KV Cache Compression (compress reads at inference) and the speculative-decoding axis of Speculative Speculative Decoding (parallelize draft/verify) — three independent attacks on the same wall.