Less is More: Recursive Reasoning with Tiny Networks

TL;DR

TRM (Tiny Recursion Model) is a heavily simplified successor to HRM (Hierarchical Reasoning Model): a single 2-layer network is recursed on a latent feature z and a current-solution feature y, with full backpropagation through one recursion process and gradient-free recursion processes used only to improve y upstream. With 7M parameters it reports 87% on Sudoku-Extreme, 85% on Maze-Hard, 45% on ARC-AGI-1, and 8% on ARC-AGI-2 — beating HRM (27M, two networks) across the board while removing the fixed-point / 1-step-gradient approximation, the H/L hierarchy, and the Q-learning ACT’s second forward pass. The companion ablation argues that “less is more” is literal: 2 layers outperform 4, a single network outperforms two, and dropping the fixed-point approximation roughly doubles Sudoku accuracy (56.5% → 87.4%).

Key claims

TRM removes HRM’s Implicit-Function-Theorem / 1-step gradient approximation and instead backpropagates through one full recursion process (n evaluations of the latent update + 1 update of the answer), while running additional recursion processes without gradients to improve the carried y upstream — this single change moves Sudoku-Extreme from 56.5% to 87.4% [§4.1, Table 1].
The H/L “two latent features at different hierarchies” framing of HRM is reinterpreted as y = current proposed solution (decodable to a token grid via the output head) and z = latent reasoning feature (not directly decodable), removing the biological-frequency justification; ablations show that splitting into more features or collapsing to a single feature both hurt (87.4% → 77.6% multi-scale, 71.9% single) [§4.2, Table 2].
A single tiny network is enough — both the L and H roles of HRM can be served by one network whose behavior is selected by whether y is present in its inputs; this halves parameters and improves Sudoku-Extreme from 82.4% to 87.4% [§4.3, Table 1].
“Less is more” on depth: scaling down from 4 layers to 2 layers (with proportionally more recursions to preserve effective depth) improves generalization from 79.5% to 87.4% on Sudoku-Extreme — the paper attributes this to an overfitting penalty when the unique-parameter count is too large relative to the 1K-example training set [§4.4, Table 1].
ACT’s continue-loss (and the second forward pass it requires in HRM) can be dropped without quality loss by training the halting head as binary cross-entropy against “answer is correct now”, reducing per-step cost by ~2× with no measurable accuracy drop (86.1% → 87.4%) [§4.6, Table 1].
EMA of the weights is load-bearing on the small-data regime: removing it drops Sudoku-Extreme from 87.4% to 79.9%, and the paper argues EMA prevents the sharp collapse HRM exhibits on small data [§4.7, Table 1].
On tasks with small fixed context (Sudoku 9×9), replacing self-attention with an MLP-Mixer-style sequence-MLP outperforms self-attention (87.4% vs 74.7%); on larger 30×30 grids (Maze, ARC) self-attention wins, so the attention-free variant is a small-context special case [§4.5, Table 1].
Optimal recursion budget on Sudoku-Extreme is (H_cycles=3, L_cycles=3), equivalent to 42 effective recursive applications; pushing further runs out of memory because TRM (unlike HRM) actually backpropagates through the full recursion process [§4.8, Table 3].
TRM-Att (7M parameters, with self-attention) matches or beats much larger CoT-pretrained LLMs on the four benchmarks: Sudoku-Extreme 74.7% / 87.4% (MLP) vs DeepSeek-R1 0% / Claude-3.7 0% / o3-mini-high 0%; ARC-AGI-1 44.6% vs Gemini-2.5-Pro 32K 37.0%; ARC-AGI-2 7.8% vs Gemini-2.5-Pro 32K 4.9% — i.e. a 7M model beats frontier LLMs at <0.01% of their parameter count on these tasks [§5, Tables 4–5].

Method

TRM defines one “full recursion process” as n consecutive applications of the latent-update function (taking x, y, z and producing a new z) followed by one application of the answer-update function (taking y and the updated z and producing a new y). A forward pass runs K-1 of these recursion processes detached from the computational graph to refine y, then runs one final recursion process with gradients, and the loss is computed on the resulting answer. This replaces HRM’s IFT-with-1-step-gradient trick: there is no assumption that z reaches a fixed point, and no theorem is invoked. Across deep supervision steps, y and z are carried (with stop-gradient on the carry) so each supervision step starts from the previous step’s refined state.

The latent-update and answer-update functions are the same tiny 2-layer Transformer block (RMSNorm, no bias, RoPE, SwiGLU per HRM). The two roles are distinguished only by whether y is in the inputs — when y is provided alongside z, the block produces the next answer; when only z is updated (plus x), the block iterates on the latent reasoning. ACT is kept but its continue-loss / Q-learning second-pass is dropped: a single halting head trained with BCE against “the current answer matches the target” provides the stop signal during training, while at inference the full schedule is unrolled. EMA over weights is enabled throughout.

For tasks with a small fixed context length (Sudoku 9×9, sequence length 81), the self-attention layer is replaced by a sequence-axis MLP (MLP-Mixer style), which has L×L parameters instead of D×D and outperforms attention at this scale. For Maze-Hard and ARC-AGI (30×30 grids), self-attention is retained.

Results

Headline accuracies on the four benchmarks at 7M parameters (TRM-Att) vs HRM at 27M [§5, Tables 4–5]:

Sudoku-Extreme (1K training, 423K test, 9×9): TRM-MLP 87.4% vs HRM 55.0% (TRM-Att 74.7%).
Maze-Hard (30×30, shortest-path > 110): TRM-Att 85.3% vs HRM 74.5%.
ARC-AGI-1 (public eval, two-attempts): TRM-Att 44.6% vs HRM 40.3%; LLM context: o3-mini-high 34.5%, Gemini-2.5-Pro 32K 37.0%, Grok-4-thinking 1.7T 66.7% (with bespoke harness 79.6%).
ARC-AGI-2 (public eval): TRM-Att 7.8% vs HRM 5.0%; LLM context: Deepseek-R1 1.3%, Claude-3.7 16K 0.7%, o3-mini-high 3.0%, Gemini-2.5-Pro 32K 4.9%, Grok-4-thinking 16.0%.

The dominant ablation movements on Sudoku-Extreme (all relative to 87.4% TRM-Att baseline) are: dropping full-backprop in favor of 1-step gradient → 56.5% (the load-bearing change), removing EMA → 79.9%, scaling from 2 to 4 layers → 79.5%, replacing the single network with two separate networks → 82.4%, restoring HRM’s two-forward-pass ACT → 86.1%, removing the MLP-vs-attention swap on the 9×9 grid → 74.7% [§4, Table 1].

The recursion-budget sweep on Sudoku-Extreme [§4.8, Table 3] shows TRM at every (H_cycles, L_cycles) setting outperforms HRM at the same effective depth, but TRM’s full-backprop runs OOM beyond (3, 6) while HRM (which only backprops 2 of 6 evaluations) does not — a real cost of removing the 1-step approximation.

Why it’s interesting

This is the third filed paper on the deterministic-recursive-reasoning branch of Looped Transformers, and it is the most aggressive simplification yet. The existing concept-page papers all retain weight-sharing across depth (ELT: Elastic Looped Transformers for Visual Generation, Hyperloop Transformers, HRM-Text: Efficient Pretraining Beyond Scaling) but assume the standard “loop a transformer block” frame; TRM is the first to (a) collapse the two-stack hierarchical-recurrence of HRM into a single network with a single latent + answer state, (b) drop the IFT/1-step-gradient trick that the HRM line had treated as essential for memory feasibility, and (c) show that backpropagating through the full recursion is what actually moves the needle on small-data hard-reasoning tasks. The ablation table is the most direct evidence on the wiki that the recursion mechanism itself — not the hierarchy, not the biology, not the fixed-point theorem — is the load-bearing piece.

Wei-An’s note flagging “apply naively to flow matching at ImageNet scale” lands directly on the open question on the Looped Transformers page about whether HRM-style recurrence carries from puzzle-solving and LLM pretraining into the visual-generation regime; ELT: Elastic Looped Transformers for Visual Generation is the closest existing filed comparator (deterministic looped transformer for visual generation with ILSD-trained any-time inference), and the natural follow-up question is whether TRM’s “no fixed-point, full backprop through one recursion, tiny network” recipe is a viable substitute for ELT’s ILSD trick when the supervision is a flow-matching loss rather than a token-grid match. TRM’s (H_cycles=3, L_cycles=3) OOM ceiling is also a concrete warning for that direction: full backprop through 42 effective layers at ImageNet resolution is not free.