When does RandOpt work?

TL;DR

Ziming Liu builds a minimal sine-wave autoregressive MLP toy model to probe when RandOpt (the random-weight-perturbation post-training method from Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights) beats Adam. The headline finding: sequence length is the deciding axis — Adam wins on short autoregressive rollouts ( $t_{\max}{=}1,2$ ), RandOpt wins on longer ones ( $t_{\max}{=}5,10,20$ ), where the loss landscape becomes rugged and Adam degenerates into a conservative “predict zero” strategy. A layer-wise ablation finds the first and last MLP layers carry most of the RandOpt benefit, hinting that gradients are inefficient for feature discovery (first layer, lazy regime) and prone to feature collapse (last layer) on long rollouts.

Key claims

On a single-trajectory post-training task (pretrained on sine waves with $t_{\max}{=}1$ , fine-tuned to roll out longer), Adam achieves lower MSE than RandOpt at $t_{\max}{=}1,2$ but RandOpt achieves lower MSE at $t_{\max}{=}5,10,20$ [§Results, Longer-sequences-favor-RandOpt table].
The crossover is hypothesized as a landscape effect: short sequences keep the post-training landscape near the pretrained minimum where it is locally quadratic and gradients are informative; long autoregressive rollouts produce a rugged landscape where Adam adopts a “safer predict-near-zero” conservative strategy that may correspond to a local minimum or saddle [§Results, prose].
Layer-wise RandOpt ablation on a 3-layer MLP: perturbing only the first or only the third layer recovers most of the all-layer RandOpt gain on long sequences; perturbing only the second layer is much worse (e.g. RandOpt-second 0.463 vs RandOpt-all 0.081 at $t_{\max}{=}5$ ) [§Results, layer-perturbation table].
The first-layer effect is conjectured to be a lazy-learning regime where gradient feature learning is inefficient and random perturbation gives better feature discovery; the last-layer effect is conjectured to be Adam over-pruning features under the conservative strategy [§Results, hypotheses].
Results are noted to be sensitive to random seeds and to hyperparameters (Adam learning rate, RandOpt noise scale $\sigma$ ); the post is framed as a starting point for systematic study rather than a definitive claim [§Results, caveats].

Method

A 3-layer MLP maps the $k$ most recent states to the next state. Pre-training uses one-step MSE on a sine-wave dataset with $\Delta t{=}0.05$ and sequences of length $t_{\max}{=}1$ (so $1/\Delta t = 20$ points). Post-training uses the autoregressive MSE over a single trajectory, with the first $k$ points provided as conditioning and the rest generated autoregressively, evaluated at test lengths $t_{\max}{\in}\{1,2,5,10,20\}$ .

Two optimizers are compared: Adam (standard gradient-based) and RandOpt (sample weight perturbations around the pretrained checkpoint, keep top-K by validation loss, ensemble). The layer-wise variant restricts the perturbation noise to one of the three MLP layers at a time.

The setup is intentionally simpler than the official Neural Thickets toy model, which mixes sine / square / sawtooth waves. Liu argues that the single-mode (sine only) version is sufficient to expose the regime transition and allows cleaner mechanistic analysis. The deliberate simplification — pretrain on $t_{\max}{=}1$ , evaluate post-training on longer rollouts — is what makes the rollout length the dominant axis.

Results

	$t_{\max}{=}1$	$t_{\max}{=}2$	$t_{\max}{=}5$	$t_{\max}{=}10$	$t_{\max}{=}20$
Before fine-tune	1.6e-3	0.051	0.563	0.550	0.525
Adam	1.1e-5	2.5e-4	0.234	0.216	0.395
RandOpt (all layers)	3.6e-4	8.3e-3	0.081	0.206	0.364
RandOpt (first only)	8.9e-4	5.3e-3	0.178	0.224	0.391
RandOpt (second only)	9.5e-4	6.4e-3	0.463	0.484	0.484
RandOpt (third only)	8.8e-4	4.6e-3	0.124	0.392	0.429

Adam is ~20× better than RandOpt-all at $t_{\max}{=}1$ but ~3× worse at $t_{\max}{=}5$ . The RandOpt-all advantage shrinks but persists out to $t_{\max}{=}20$ . Among per-layer ablations, perturbing only the middle (second) layer is consistently the weakest — at $t_{\max}{=}5$ it is barely better than no fine-tune at all (0.463 vs 0.563), while first- or third-layer perturbation is dramatically better.

Why it’s interesting

This is the cleanest dissection so far of the “when does RandOpt beat gradient post-training” question raised by Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights. Where the original paper established that the thicket regime exists at scale and at “post-training over standard RL benchmarks”, Liu’s toy model isolates autoregressive rollout length as the key landscape-roughness axis — a much sharper claim than the Neural Thickets paper’s regime taxonomy (needle / thicket / plateau), and a testable one in larger models. The conservative-strategy explanation (Adam collapses toward predict-zero on long rollouts) connects RandOpt’s advantage to a long-standing pathology of gradient methods on autoregressive sequences, the same pathology that motivated teacher-forcing tricks and the BPTT-vs-RNN-trainability literature.

It also lands in interesting tension with the broader Reasoning RL picture. The Neural Thickets paper compared RandOpt at fixed FLOPs against PPO/GRPO/ES on reasoning tasks (Countdown, GSM8K, MATH-500), where rollouts are long and discrete. Liu’s toy model gives a candidate mechanism for why RandOpt is competitive specifically there: gradient methods on long autoregressive sequences hit a conservative-prediction trap, and weight-space sampling escapes it. If this generalizes, it suggests RandOpt-like methods should disproportionately help on long-horizon reasoning compared to short-form classification — an empirical prediction the Neural Thickets paper’s task suite is roughly consistent with but does not isolate.

The layer-wise ablation (first/last layers carry the benefit, middle layer doesn’t) is the most surprising finding and worth replicating before believing — it would imply that a targeted RandOpt restricted to embedding+head layers could capture most of the gain at a fraction of the perturbation dimensionality, which is essentially a cheaper form of the parallel sampling already in Inference-Time Scaling.