Base Models Can Reason By Taking a Cue From Training Data

TL;DR

Prefilling 1–2 opening tokens — “token cues” — brings base-model reasoning performance close to its RL-trained counterpart on math and coding: .\n\nOkay lifts Olmo-3-7B MATH-500 pass@1 from 42% to 78%, and Alright, lifts Qwen3-14B from 72% to 87%. The effect comes from associations learned during pretraining/mid-training rather than from the token’s semantics: a causal data edit that swaps “Okay” for “Chicken” in a 100B-token synthetic-reasoning mid-training mix makes “Chicken” itself an effective reasoning cue (18.2% → 76.9%) without destroying its original meaning. RLVR (GRPO) then concentrates most of its measurable change on just the first two token positions, upweighting the cues the base model already had — a sharper mechanistic account of the “RL elicits pretraining behavior” thread.

Key claims

Fixing a 1–2 token cue at the start of a base model’s response brings it close to its RL-trained counterpart on math reasoning: .\n\nOkay lifts Olmo-3-7B MATH-500 pass@1 from 41.5% → 76.9% vs the RL counterpart’s 74.9% [§Token Cues].
The effective cues are not the base model’s most-likely openings — the base model would rarely emit them on its own [§Token Cues].
After GRPO-style RLVR, the two models’ next-token distributions differ most in the first two token positions and largely agree after that, i.e. RL concentrates measurable updates on the opening [§RL].
An arbitrary token like “Chicken” can be made a reasoning cue by a causal training-data edit: replacing “Okay” with “Chicken” in a 100B-token mid-training mix (dominated by synthetic reasoning traces) raises the “Chicken”-cued MATH accuracy from 18.2% → 76.9% on Olmo-3-7B [§Data].
The same edit does not overwrite “chicken“‘s ordinary semantic use — the resulting model still completes “A chicken is a” with “bird that belongs to the family of Galliformes” and “She roasted the chicken” with “and made a salad” [§Data].
Prompt-level analogue: the instruction “Think duck duck goose” can be made as effective as “Think step by step” via an analogous data edit, confirming the mechanism generalizes beyond a single prefill token [§Abstract].
Hidden-state representations induced by different cues correlate with different document types in the training set — i.e. the cue is routing the model into a training-data cluster, not invoking a general reasoning ability [§Abstract].
A case study in safety: different opening cues elicit distinct refusal/compliance behaviors that also correspond to different training-data types, suggesting token cues are a general lens on how base models allocate response strategies, not a math-specific trick [§Abstract].
The paper separates what a model can do from what it is likely to do: effective cues already exist in the base model’s response space but with low prior probability, so evaluating base models by raw sampling systematically undercounts latent capability [§Implications - Post-training].

Method

The authors identify “token cues” by scanning starts of base-model responses and correlating accuracy with specific opening token patterns. They then prefill those openings during generation and compare the resulting pass@1 against the base model’s unconditional sampling and against the matched RL-Zero-trained model (RLVR / GRPO on math/code). The main model family in the paper is Olmo-3-7B (so that the full training stack, including the mid-training mix, is accessible), with results reported to generalize to other base-model families.

For the data-causality analysis they take the public mid-training checkpoint and the 100B-token mid-training mix, apply a string-level edit (replace “Okay” with “Chicken” throughout the mix), and resume training. They then re-measure the cued accuracy of .\n\nChicken and separately probe that the ordinary semantic use of “chicken” is preserved by free-form completion probes. A parallel analysis on the prompt side substitutes “Think duck duck goose” into instruction templates to show the mechanism is not prefill-specific.

For the RL analysis they compare the base model to its GRPO-trained counterpart’s next-token distributions, finding that the biggest divergences are at the first two token positions and that beyond that the two policies largely agree given the same prefix. They also sweep different cues and surface hidden-state → training-document-type correspondences.

Results

Olmo-3-7B MATH-500: base 41.5%, .\n\nOkay-cued 76.9%, RL-Zero counterpart 74.9% [§Token Cues].
Qwen3-14B MATH-500: base 72%, Alright,-cued 87% [§Abstract, §Token Cues].
Olmo-3-7B after mid-training data edit (Okay → Chicken): .\n\nChicken-cued accuracy 18.2% → 76.9%, while “chicken” retains ordinary semantic completions [§Data].
After GRPO RLVR, the largest changes in next-token predictions occur at the first two token positions; from position 3 onward, base and RL models largely agree on next-token distributions given the same prefix [§RL].
“Think duck duck goose” can be made as effective as “Think step by step” at eliciting reasoning via an analogous training-data edit [§Abstract].

Why it’s interesting

This is the sharpest filed mechanistic account of the thread Spurious Rewards: Rethinking Training Signals in RLVR opened and The Path Not Taken: RLVR Provably Learns Off the Principals formalized at the parameter level: RLVR at current scale is selective elicitation of pretraining behavior, not installation of new reasoning. Spurious Rewards localized the Qwen2.5-Math gain to the “code reasoning” pretraining behavior and showed that Lang→Code transitions dominated the gain; this paper goes one step further and localizes the elicited behavior to the first two tokens of the response, and ships a causal training-data intervention showing those two tokens are not semantically privileged — any token (even “Chicken”) can be weaponized given the right pretraining association. Combined with Reasoning with Sampling: Your Base Model is Smarter Than You Think‘s result that MCMC-style resampling at inference time already extracts most of the “RL gain”, the picture across the three papers is tight: base models already know how to reason, RLVR mostly concentrates mass on specific opening tokens, and most of the measurement gap between base and RL is in how aggressively each samples from the thin reasoning-conditioned region.

For Luma this sharpens two team-relevant takeaways. First, evaluating base-model capability by raw sampling systematically undercounts what the model can do — any benchmark comparison between a base and RL-trained model should separately report cued baselines before attributing gains to RL. Second, the training-data design axis is more powerful than usually assumed: inadvertent repetition of a few opening tokens in synthetic-reasoning traces can reinforce an association that then dominates post-RL evaluation numbers. This complements the SFT-for-RL Readiness cluster’s reframing of SFT as RL preparation (Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning, Behavior Injection: Preparing Language Models for Reinforcement Learning, TailSFT: Filtered Fine-Tuning Improves Post-Training Performance) — those papers reshape how SFT loads capability for RL to amplify; this one shows where in the sequence that capability sits once loaded.