Reasoning with Sampling: Your Base Model is Smarter Than You Think

TL;DR

Asks whether RL-style reasoning gains can be elicited from a base model at inference time without any post-training. Proposes power sampling, an MCMC procedure that resamples blocks of an in-progress completion to draw from a sharpened, $\alpha$ -tempered version of the base distribution $p(y \mid x)^\alpha$ . On Qwen2.5-Math-7B, Llama-3.1-8B, and Phi-3.5-mini, the single-shot accuracy from power sampling matches or beats GRPO on MATH500, HumanEval, and GPQA — at roughly an 8.84× inference multiplier on MATH500, equivalent to one epoch of 8-rollout GRPO. The diversity collapse that RL induces is avoided: pass@ $k$ over $k$ samples stays at the base-model level. Concrete claim: a large fraction of what RLVR is “teaching” reasoning models is already latent in the base, recoverable by sampling from the sharpened distribution rather than from $p$ itself.

Key claims

Power sampling targets the sharpened distribution $p(y \mid x)^\alpha / Z$ via an iterative MCMC procedure that resamples blocks of a partial completion using the base model’s own likelihoods — no training, no verifier, no reward [Abstract, §1].
On MATH500 (in-domain for RL post-training), power sampling matches GRPO across Qwen2.5-Math-7B, Llama-3.1-8B, and Phi-3.5-mini; on out-of-domain tasks (HumanEval, GPQA) it can outperform GRPO [Table 1].
Headline boosts over base-model temperature sampling: up to +51.9% on HumanEval (Phi-3.5-mini) and +25.2% on MATH500 (Qwen2.5-Math) [Table 1].
The inference multiplier on MATH500 is ~8.84× under the paper’s hyperparameters, roughly equivalent to one epoch of GRPO with 8 rollouts on an identically sized dataset — placing power sampling in a practical regime [§Results, “practical”].
Power sampling preserves base-model diversity at high $k$ : its pass@ $k$ curve is strictly above both GRPO and base sampling, with pass rate at high $k$ matching the base model — GRPO instead collapses to a single high-likelihood mode [Fig. 5].
The empirical likelihood histogram (Qwen2.5-Math-7B, MATH500) shows power sampling concentrating mass on higher-likelihood regions of the base than ordinary sampling, but still with noticeable spread; GRPO’s samples sit at the highest-likelihood peak [Fig. 4].
The optimal sharpening $\alpha$ is intermediate: $\alpha = 4.0$ wins; $\alpha = 1.0$ is the base model and $\alpha \to \infty$ deterministically accepts any likelihood-increasing resampled block. Performance is robust for $\alpha \geq 2.0$ [Fig. 6].
Higher base-model likelihood and reasoning quality are correlated but not identical — directly optimizing likelihood (large $\alpha$ ) is not optimal for reasoning, motivating the intermediate- $\alpha$ sweet spot [§Power distribution effects].
The implementation released at aakaran/reasoning-with-sampling covers MATH500, HumanEval, GPQA Diamond, and AlpacaEval 2.0; AlpacaEval (an unverifiable task) is also reported, generalizing the claim beyond verifiable reasoning [GitHub README, [Table 1]].

Method

Power sampling defines the target distribution $\pi_\alpha(y \mid x) \propto p_\theta(y \mid x)^\alpha$ where $p_\theta$ is the base model. To draw from $\pi_\alpha$ , the authors run an MCMC chain over partial completions: a block of $B$ tokens is resampled from $p_\theta$ given the prefix, and the proposal is accepted with a Metropolis-Hastings ratio that depends on $\alpha$ and the relative base-model likelihoods of the old and new blocks. The chain operates iteratively across the token sequence (sliding-window resampling) for $N_\text{MCMC}$ steps per block. The total inference cost is roughly $N_\text{MCMC} \cdot T \cdot (T + B) / (4B)$ forward-pass-equivalents for sequence length $T$ — yielding the ~8.84× multiplier on MATH500 hyperparameters (and a community-flagged correction to ~11.34× when $T \not\gg B$ , per the alphaXiv comments).

The key hyperparameters are $\alpha$ (sharpening exponent) and the block size $B$ vs. number of MCMC steps $N_\text{MCMC}$ tradeoff. Larger $B$ requires larger MH “jumps” and is harder to accept; smaller $B$ converges more slowly but with higher acceptance.

Results

MATH500 accuracy on Qwen2.5-Math-7B: +25.2 points over base sampling, on par with GRPO [Table 1, §Headline result].
HumanEval on Phi-3.5-mini: +51.9 points, outperforming GRPO [Table 1].
Pass@ $k$ on MATH500 (Qwen2.5-Math-7B): power-sampling curve strictly above both GRPO and base; at high $k$ power sampling matches base-model coverage while GRPO falls below it [Fig. 5].
Likelihood distribution under power sampling sits between base and GRPO: noticeably higher-likelihood than base, noticeably less collapsed than GRPO [Fig. 4].
Robustness: $\alpha \in [2, \infty)$ all within a small accuracy band on MATH500; $\alpha = 4.0$ optimal [Fig. 6].

Why it’s interesting

The wiki’s Reasoning RL cluster has nine papers documenting GRPO-and-shaped-rewards as the de facto reasoning recipe, with the active research surface centered on reward shape, data loops, and orchestrator policies. This paper attacks the cluster’s premise — that RL is the right primitive for unlocking reasoning at all — from a direction orthogonal to Training-Free Group Relative Policy Optimization (which keeps GRPO but moves it to context space) and complementary to The Path Not Taken: RLVR Provably Learns Off the Principals (which gives a parameter-level reason RLVR works at low rank). Together those three papers triangulate a hypothesis: RLVR is mostly sharpening the base distribution along a low-effective-rank, off-principal subspace, and the same sharpening is reachable by sampling. The paper also adds a seventh axis to Inference-Time Scaling alongside the six already enumerated — neither scaffold, interaction-depth, KV-cache compression, parallel-agent orchestration, generation-side iterative refinement, nor TTT — namely MCMC over base-model likelihoods. Distinct from Self-Refining Video Sampling (which also iterates at fixed noise but on a trained video flow generator and requires no acceptance criterion) and from rejection sampling (which the paper compares unfavorably against). The pass@ $k$ preservation also makes this a candidate primitive for generation-side use cases where diversity matters — e.g. video sampling against a reward model — without paying the RL diversity-collapse cost.