Fractal basins trap latent reasoning

TL;DR

Casts frontier reasoning models as dynamical systems and shows that the map from initial hidden state to converged answer forms fractal basins of attraction across diverse tasks (Sudoku, mazes, visual puzzles, mathematical logic) and architectures. Fractality intensifies with task difficulty. The mechanism proposed is transient chaos: reasoning trajectories get trapped near saddle points that correspond to nearly-correct incorrect solutions, producing 10× variability in inference-cost between adjacent prompts. Framed as a physical inevitability — sensitivity is what lets the model check many hypotheses, but the price is unpredictable and unbounded inference time. Establishes reasoning traces as a new class of dynamical system.

Key claims

Reasoning models exhibit transient chaos: initial conditions that start adjacent take different-length routes to the same equilibrium answer, and the map from initial condition to convergence time forms fractal basins across all tested architectures and tasks [§Abstract].
Fractality strengthens with task difficulty, tested across Sudoku, maze solving, visual puzzles, and mathematical logic — argued to be a generic effect when models map hard problems onto analog latent dynamics [Abstract, thread §2].
Saddle points in the latent dynamics correspond to nearly-correct attempted solutions; PCA visualization of two adjacent initial conditions shows one solves quickly while the other gets stuck near a wrong solution [thread §4].
The fast Lyapunov indicator of a reasoning trajectory correlates with how many candidate solutions the model visits before converging — i.e. transient chaos is a positive mechanism for solution search, not merely a failure mode [thread §6].
Fractal basins emerge mid-training via a bifurcation: a loop transformer trained on integer linear regression develops both generalization to unseen examples and fractal basin structure at the same training checkpoint [thread §7].
“Overthinking” — two similar prompts producing reasoning traces with 10× differences in length and cost — is a direct consequence of proximity to fractal basin boundaries [thread §8].
Speculates that transient chaos is an inevitable cost of training networks to solve hard problems: the sensitivity required to support hard computations produces unpredictable inference cost as a physical byproduct [thread §9].

Method

Treats reasoning traces as trajectories of a discrete-time dynamical system indexed by decoding step (for language reasoning models) or by loop iteration (for looped/latent-recurrent models). For each initial condition — either a perturbed initial hidden state or a nearby prompt embedding — the system is rolled forward until it converges on a fixed answer, and the convergence time is recorded. Coloring the initial-condition space by convergence time reveals fractal boundaries between basins. Transient chaos is quantified with the fast Lyapunov indicator (a finite-time measure of local trajectory divergence). Saddle-point structure is exposed by PCA-projecting hidden-state trajectories from adjacent initial conditions and tracing where they diverge. A separate controlled experiment trains a loop transformer from scratch on integer linear regression and shows fractal-basin emergence coincides with the generalization phase transition mid-training.

Results

Fractal basins observed across all tested reasoning architectures and all four task classes (Sudoku, mazes, visual puzzles, mathematical logic).
Fractality visibly increases with problem difficulty within each task class.
Fast Lyapunov indicator monotonically correlates with number of candidate solutions visited per trajectory.
Loop-transformer integer-linear-regression case: bifurcation in the trained model occurs mid-training — after the bifurcation the model both generalizes to unseen problems and exhibits fractal basin structure. (Quantitative numbers not extracted from thread; full metrics in the PDF.)

Why it’s interesting

Adds a dynamical-systems diagnostic layer to the Mechanistic Interpretability toolkit alongside residual-stream concept vectors, SAE dictionaries, and Jacobian-space analysis — reasoning traces become a substrate for measurable physical quantities (Lyapunov indicators, basin boundaries) rather than only for token-level or activation-level probes. Sharpens two existing wiki threads: the Looped Transformers Open Question “does looping only matter because it changes the model’s complexity class?” now has a candidate mechanistic answer — the recurrent latent dynamics support saddle-point search over hypotheses, and the bifurcation experiment ties fractal-basin emergence to the generalization phase transition. And it gives Inference-Time Scaling a physical reason for the unpredictability half of the “spend more compute” story — the The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs finding that reasoning-token variance grows with task difficulty gets a mechanistic explanation (proximity to basin boundaries) rather than being purely empirical. Contrasts with Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls‘s constructive-mechanistic story on arithmetic: that paper reverse-engineers a clean algorithmic solution and derives an auxiliary-loss fix; this paper argues that on hard tasks the clean solution is provably unavailable and messy transient chaos is the price.