HRM-Text: Efficient Pretraining Beyond Scaling

TL;DR

HRM-Text is the technical paper behind the Sapient Intelligence 1B “hierarchical reasoning” LLM release: a from-scratch pretraining recipe that replaces a standard Transformer stack with a two-level recurrent model (slow H, fast L), trained on 40B unique tokens of instruction-response pairs with a task-completion / PrefixLM objective for ~$1500 of compute. To stabilize the deep recurrence, the paper introduces MagicNorm and “warmup deep credit assignment.” At 1B parameters the model reaches 60.7% MMLU, 81.9% ARC-C, 82.2% DROP, 84.5% GSM8K, and 56.2% MATH — competitive with 2–7B open baselines that consumed 100–900× more tokens and 96–432× more compute. The paper frames itself as an “existence proof” that co-designing architecture (HRM) with objective (task-completion + PrefixLM) can radically shift the compute-to-performance frontier of pretraining.

Key claims

The pretraining recipe combines a Hierarchical Recurrent Model (slow H module + fast L module, multi-timescale recurrence) with task-completion training on instruction-response pairs under PrefixLM masking — both the architecture and the objective are non-standard relative to mainstream Transformer + raw-text-next-token pretraining [§Abstract].
1B HRM-Text trained from scratch on 40B unique tokens for ~$1500 reports 60.7% MMLU, 81.9% ARC-C, 82.2% DROP, 84.5% GSM8K, 56.2% MATH [§Abstract].
Token efficiency vs standard pretraining: roughly 100–900× fewer training tokens; compute efficiency: 96–432× less estimated compute, while remaining competitive with 2–7B open models [§Abstract].
Two new training-stability components are introduced specifically to make the deep recurrence trainable for language modeling: MagicNorm (a normalization variant) and warmup deep credit assignment [§Abstract]. The paper presents these as the load-bearing pieces that took the HRM architecture from its prior 27M-parameter puzzle-solving regime up to a 1B language-modeling regime.
The framing is “existence proof, not frontier claim” — the contribution is that pretraining from scratch is accessible to the broader research community once architecture and objective are co-designed, not that HRM-Text dominates frontier models [§Abstract].

Method

HRM-Text instantiates a Hierarchical Recurrent Model: two Transformer modules (H for slow, strategic updates; L for fast, executional updates) iterate over the input embeddings for H_cycles × (L_cycles + 1) steps with additive cross-level state injection (z_L + z_H). The 1B release uses 2 H-updates and 6 L-updates per forward pass (8 stack iterations total) before any output token is emitted, so reasoning happens in continuous latent state rather than as chain-of-thought tokens.

Two stabilization mechanisms are introduced to make this deep recurrence trainable for language modeling at 1B scale: MagicNorm (a normalization variant — see arXiv for specifics) and warmup deep credit assignment, which appears to regularize gradient flow through the unrolled recurrence early in training. The training objective is not standard next-token prediction over raw text. Instead, the model is trained exclusively on structured instruction-response pairs under a PrefixLM mask (only the response portion contributes to the loss), framed as “task completion” rather than autoregressive language modeling.

Total compute is reported as ~$1500 for a ~1-day pretraining run on 40B unique tokens. Weights ship Apache-2.0 at sapientinc/HRM-Text-1B. No SFT, DPO, or RL post-training is applied — all reported benchmark numbers are base-model.

Results

Base-model benchmarks (no post-training, “independent verification April 2026”):

MMLU 60.7% (57-domain general knowledge)
ARC-Challenge 81.9% (science / commonsense)
DROP 82.2% (discrete reasoning over paragraphs)
GSM8K 84.5% (grade-school math word problems)
MATH 56.2% (competition math, multi-step)

The headline comparison is to 2–7B open models trained on 4–36T tokens: HRM-Text claims competitive scores at 100–900× fewer tokens and 96–432× less compute [§Abstract]. The paper does not publish a full apples-to-apples post-trained comparison and explicitly frames the result as an “existence proof” of the compute-to-performance shift rather than a frontier claim.

The companion blog announcement (Introducing HRM-Text: 1B Hierarchical Reasoning Language Model) reports the same MMLU/ARC-C/DROP/MATH numbers but does not foreground GSM8K 84.5% or the $1500 figure (it cites ~$ 1K). The arxiv paper is the authoritative source for the 84.5% GSM8K result and the MagicNorm + warmup deep credit assignment naming.

Why it’s interesting

This is the technical follow-up to Introducing HRM-Text: 1B Hierarchical Reasoning Language Model, filed separately because it adds substantive new artifacts the blog did not: the names of two stability mechanisms (MagicNorm, warmup deep credit assignment), the GSM8K 84.5% number, the explicit task-completion + PrefixLM masking framing, and the corrected compute figure ( $1500 not$ 1K). For the Looped Transformers page this is now the canonical reference for HRM-style hierarchical recurrence at LLM scale, sitting alongside Hyperloop Transformers (single-stack looped LLM + hyper-connections at loop boundaries) and ELT: Elastic Looped Transformers for Visual Generation (visual generation, ILSD-trained any-time inference) — and unlike those two it ships full open weights at 1B with a public reproducible-on-$1500 recipe.

The MagicNorm + warmup-deep-credit-assignment angle also connects to Training stability at scale: every other paper on that page locates LLM-scale stability in QK-Norm, GatedNorm, Birkhoff-constrained residual mixing, or spectral-sphere optimization — HRM-Text adds a fifth locus (normalization specifically tuned for unrolled recurrent stacks) that none of the existing papers covers. The Swayam note (“might give a shot at adapting to diffusion”) is a natural research direction: the Looped Transformers page already has an open question about whether HRM’s fixed 2×(6+1) schedule is exploitable as an inference-time compute knob the way ELT’s any-time inference is, and the diffusion setting (where sample-time compute is already the dominant cost) is the most natural place to test that.