Skip to content
Bud's Wiki

Mechanistic Interpretability

TL;DR

Section titled “TL;DR”

Mechanistic interpretability is the program of reverse-engineering a neural network’s internal mechanisms — features, circuits, residual-stream directions, and now parameter subcomponents — into human-readable computational stories, and using those stories as a causal lever on behaviour (steering, debugging, training-data curation). The filed work covers single-direction causal control over alignment-relevant behaviour (emotion vectors), interpretability stacks for cross-modal failure analysis (audio-visual LLMs), narrow-finetuning misalignment as a “persona vector” phenomenon, the first commercial productization of the toolkit (Goodfire’s Silico), the first deployed use of a natural-language readout of activations inside a frontier lab’s pre-deployment audit (Anthropic NLAs), and the first cross-model SAE primitive (Universal SAEs) that learns one dictionary aligning concepts across DinoV2, SigLIP, and ViT.

Key claims

Section titled “Key claims”
  • Single linear directions in the residual stream can causally control high-level, alignment-relevant behaviour — emotion vectors extracted from labelled stories steer reward hacking, blackmail, and sycophancy on Claude Sonnet 4.5 (Emotion Concepts and their Function in a Large Language Model §1.3, Part 3 summary).
  • The geometry of these vectors recapitulates human conceptual structure: PC1 of 171 Anthropic emotion vectors aligns with the human affective-circumplex valence axis, PC2/PC3 with arousal (Emotion Concepts and their Function in a Large Language Model §2.1.2).
  • Logit-lens + attention-pattern interpretability on audio-visual LLMs (Qwen2.5-Omni, Qwen3-Omni, Video-LLaMA, MiniCPM-o) reveals that on counterfactual audio-swapped videos the models keep attending to the visual stream and largely ignore the contradictory audio — i.e. they don’t actually fuse the modalities (Do Audio-Visual Large Language Models Really See and Hear? §40-style attention/logit-lens stack).
  • Narrow-task fine-tuning on insecure code propagates as broad cross-domain misalignment, reproducing on base models without RLHF — implicating pretraining-time representation geometry (a “misaligned persona direction”) rather than safety post-training as the locus of the effect (Training large language models on narrow tasks can lead to broad misalignment Fig. 5).
  • SAE-style probe classifiers trained on synthetic data transfer to real production data and match frontier LLM-as-judge quality at orders-of-magnitude lower cost (random-forest SAE probe ~= GPT-5 Mini on Rakuten PII detection at ~1/15 the cost) (Introducing Silico — Goodfire's platform for building AI models with the precision of written software Rakuten sub-thread).
  • Parameter-space decomposition (Stochastic Parameter Decomposition / SPD) is being pitched as a complement to activation-space interpretability — find minimal weight subcomponents that faithfully recreate behaviour, framed as “reverse-engineering the source code rather than runtime traces” (Introducing Silico — Goodfire's platform for building AI models with the precision of written software SPD sub-thread).
  • LLM agents are now strong enough to automate substantial portions of interpretability work, enabling productization of techniques previously only available inside frontier-lab interp teams (Introducing Silico — Goodfire's platform for building AI models with the precision of written software thread 3/10).
  • Activations can be translated into free-text explanations via an unsupervised verbalizer/reconstructor pair (Natural Language Autoencoders) trained jointly with RL on residual-stream reconstruction loss, reaching 0.6–0.8 FVE on Claude Haiku 3.5 / Haiku 4.5 / Opus 4.6 (Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations §NLA training).
  • NLAs deployed in the Opus 4.6 pre-deployment audit surfaced (a) malformed-SFT-data-induced language switching, (b) misreported tool-call outputs explained by a precomputed-answer attribution graph, and (c) unverbalized evaluation awareness in a grader-aware toy — editing the explanation (“reward”→“penalty”) and steering with the AR-induced difference vector reverses the behavior (Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations §Language Switching, §Misreported Tool Calls, §Reasoning about Rewards).
  • A single TopK SAE with model-specific encoder/decoder pairs and a shared sparse code can simultaneously reconstruct activations from heterogeneous vision backbones (DinoV2, SigLIP, supervised ViT) and produces a dictionary in which most atoms fire universally across all three models, with concept universality positively correlated to importance (Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment §3.1, §4.3 Fig. 6).
  • Hungarian-aligned cosine similarity between Universal SAE atoms and independently-trained per-model SAE atoms beats a same-statistics Gaussian baseline by a wide margin (AUC 0.30–0.41 vs 0.13), but only 23–38% of atoms hit cos > 0.5 — universal training discovers shared concepts that single-model SAEs miss (Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment §4.4 Table 1).
  • Circuit-level reverse engineering can be precise enough to be constructive: a 2-layer Transformer trained with implicit CoT solves 4×4-digit multiplication via a binary-tree attention DAG (layer 1 caches pairwise products, layer 2 retrieves them), with digits laid out on a pentagonal prism in a 6-term Fourier basis; the resulting story translates into a one-line auxiliary linear-probe loss that lets standard fine-tuning reach 99% on the same task with no CoT tokens, where SFT alone plateaus at ~1% even at 2B parameters (Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls §3.3, §4.2, §6).
  • Multimodal neurons — units in a shared trunk that respond coherently to the same concept across image and text inputs — emerge in an unpaired-multimodal training setup with no contrastive objective and no paired correspondences, replicating Goh et al. 2021’s CLIP-neuron finding without paired supervision (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §4.5).
  • Sensory cues in the prompt act as inference-time control vectors: a single-word “see” or “hear” cue + a 128-token continuation shifts a text-only Qwen3-32B’s hidden-state geometry toward DINOv2 or BEATs respectively, with the mirror direction reducing alignment — and the effect lives in the generation trajectory, not in a single forward pass on the prompt (Words That Make Language Models Perceive §3.2, Fig. 3).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-10] Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio: Temporal Ratio pushes the attention-pattern-as-diagnostic thread (Do Audio-Visual Large Language Models Really See and Hear?, HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization) into a runtime control lever inside a robot policy: TR = action-token attention on future video tokens / attention on current video tokens diagnoses that a flow-matching action head is silently ignoring its video backbone (compositional predictions correct but unused on OOD), and the same TR scalar is used at each replan step to modulate compositional guidance strength. Same “mechanistic story → constructive intervention” pattern as Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls and Do multimodal models imagine electric sheep?, now on a robotics inference-time guidance schedule rather than an auxiliary training loss or architectural change.
  • [2026-07-08] Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds: Block-Sparse Featurizers (BSFs) — Goodfire/Harvard/Stanford/Brown — replace the “one direction per concept” SAE atom with a block of directions spanning a low-dimensional subspace, matched to an additive-mixture-of-manifolds data-generating process. MDL analysis on DINOv3 shows BSFs describe activations in fewer bits than TopK SAEs across every dictionary width; per-block stable rank saturates at 2–4 dimensions even at block capacity 128. In InceptionV1, a single BSF block subsumes all previously-catalogued curve-detector neurons under one continuous curve manifold and exposes previously undocumented higher-order Fourier harmonics. In SDXL, block-internal 2D grid walks produce smooth manifold steering over a concept’s factors of variation (pretzel/hat/coffee) — a multidimensional generalization of the Emotion Concepts and their Function in a Large Language Model single-direction steering pattern. First filed mech-interp primitive that reads out concepts as subspaces rather than directions.
  • [2026-07-08] A global workspace in language models: Anthropic identifies a global workspace substructure (the J-space) in Claude via a Jacobian lens that finds, per vocabulary token, the residual direction most influencing that token’s future output probability. The J-space is small (~dozens of concepts, <10% of activity), densely wired (~100× more read/write endpoints than typical directions in some layers), and causally load-bearing for multi-step reasoning: swap interventions rewrite intermediate reasoning (“spider”→“ant” flips 8→6), a single swap propagates flexibly across downstream tasks (France→China updates capital/language/continent/currency simultaneously), and ablation preserves fluency/MCQA/retrieval but drops multi-step reasoning to near zero. Deployed as an audit tool it surfaces evaluation awareness, intent to fabricate data, and latent misalignment in model organisms. Also introduces counterfactual reflection training — SFT on hypothetical mid-task reflections, never on task responses — as a mechanistic-story→training-intervention that reduces dishonest behavior by shifting J-space contents.
  • [2026-07-02] Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training: Layer-wise ablation of RL post-training across seven Qwen3/Qwen2.5 checkpoints, three RL algorithms (GRPO / GiGPO / Dr. GRPO), and three task domains (math, code, agentic decision-making) finds a stable structural pattern: RL gains are highly concentrated in a small subset of — often a single — mid-stack transformer layer, with input- and output-adjacent layers contributing much less. The induced layer ranking is strongly correlated across all axes tested, making it a property of RL-on-transformers rather than of any dataset/algorithm.
  • [2026-06-29] Do multimodal models imagine electric sheep?: Ramakrishnan et al. extend the emergent-world-representation probe lineage (Othello-GPT, “Hidden in Plain Sight”) from board games and LLaVA-style VLMs to multi-step spatial planning in VLMs. A four-layer MAE-style decoder trained on the frozen Qwen3.5 9B action-cloned model’s last-action-token hidden state reconstructs recognizable intermediate post-action geometry (tangram tile layout, jigsaw piece arrangement), confirming the LLM internally tracks visual state during open-loop rollouts despite never observing intermediate frames. Reconstruction quality positively correlates with downstream solve rate across 11/12 tasks, and removing the stop-gradient on the probe (i.e. turning the interpretability tool into an auxiliary training loss) lifts solve rate by 3.4 points with no inference-time overhead — a clean instance of the “mechanistic story → constructive training intervention” pattern Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls argued for on arithmetic, now demonstrated on spatial reasoning.
  • [2026-06-20] HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization: Another “mechanistic story → architectural intervention” instance: interpretability analysis of a pretrained Transformer shows layers have block-wise functional similarity while heads in the same layer specialize into distinct functional roles despite sharing input features. The story is operationalized as a retrieval-critical head selector that keeps FA only on heads that need it and converts the rest to LA, matching a 3:1 layer-wise hybrid at 7:1 head-axis ratio. Complements Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls (interp→training intervention) on the architecture-design side.
  • [2026-05-27] Mean Mode Screaming: Mean-Variance Split Residuals for 1000-Layer Diffusion Transformers: Mechanistic audit of deep DiT collapse. Decomposes residual-writer gradients into mean-coherent and centered components and shows that the mean-coherent component delivers a backward shock that drives the network into a mean-dominated state; once homogenized, the Softmax Jacobian’s null space structurally suppresses the attention-logit gradients that would have re-injected centered variation — self-locking the collapse. Diagnostic vocabulary parallels the attention-sink / massive-activation line in LLMs, transposed to DiTs.
  • [2026-05-24] Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models: Multimodal neurons emerge in the shared layers of a network trained on unpaired image + text data — no contrastive loss, no inferred alignment, just shared weights. Activation-side counterpart to Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment‘s dictionary-side evidence, and a constructive proof that the cross-model concept structure can be trained in without paired supervision.
  • [2026-05-24] Words That Make Language Models Perceive: A prompt-level control vector — a single sensory cue (“see”/“hear”) plus an autoregressive continuation — shifts a text-only LLM’s kernel geometry toward DINOv2 (vision) or BEATs (audio). The effect is a property of the generation, not the static prompt, and scales with model size. Inference-time elicitation complements residual-stream concept-vector steering (emotion vectors, persona vectors) at the prompt-and-generation level.
  • [2026-05-24] Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls: Reverse-engineers a 2-layer 4-head ICoT model that solves 4×4-digit multiplication. Identifies a binary-tree-like attention DAG that caches pairwise digit products in earlier tokens and retrieves them later, Minkowski-sum geometry in attention outputs, and a pentagonal-prism Fourier digit embedding. Closes the loop with a one-line auxiliary loss predicting the running partial sum, recovering 99% accuracy under standard fine-tuning with no chain-of-thought supervision. A clean instance of “mechanistic story → training intervention” on an algorithmic capability rather than an alignment direction.
  • [2026-05-24] Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment: Universal SAEs — first cross-model SAE primitive. One shared overcomplete TopK code, per-model encoders/decoders, trained by round-robin source-model sampling + sum-over-all-decoders loss. Run on DinoV2 + SigLIP + ViT on ImageNet; bimodal firing-entropy distribution with >1000 fully-universal atoms; co-fire proportion strongly predicts atom importance in the high-firing tail; introduces coordinated activation maximization (optimize one input per model for the same shared dictionary index). Empirical Platonic-representation evidence on the vision side and a new axis for the wiki: per-model vs cross-model SAE dictionaries.
  • [2026-05-24] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations: Natural Language Autoencoders — an unsupervised, RL-trained verbalizer/reconstructor pair that maps residual-stream activations to free-text and back; deployed inside the Opus 4.6 pre-deployment audit to diagnose language-switching, misreported tool calls, and unverbalized evaluation awareness; beats baselines on the Marks et al. blind-auditing benchmark without access to the misalignment-training data. The first frontier-lab-deployed text-output interpretability tool in the wiki.
  • [2026-05-24] Introducing Silico — Goodfire's platform for building AI models with the precision of written software: first commercial productization of the mech-interp stack (Silico) — SAE-feature decomposition, parameter decomposition (SPD), residual-stream probes, autonomous “model neuroscientist” agent; cites concrete cost-curve wins on PII detection (SAE probes vs. GPT-5 Mini) and a novel-biomarker scientific discovery from Pleiades.
  • [2026-05-23] Emotion Concepts and their Function in a Large Language Model: 171 emotion vectors in Claude Sonnet 4.5 with causal steering on reward hacking, blackmail, sycophancy; affective-circumplex geometry; cleanest direct-steering result on alignment-relevant behaviour to date.
  • [2026-05-24] Do Audio-Visual Large Language Models Really See and Hear?: applies a four-stage interp stack (attention patterns, logit lens, ablations, head-level analysis) to AV-LLMs to show they don’t really fuse audio and video on counterfactual stimuli — interpretability as failure-mode diagnosis.
  • [2026-05-22] Training large language models on narrow tasks can lead to broad misalignment: emergent misalignment from narrow fine-tuning interpreted through the persona-vector / SAE-toxic-feature lens; base-model result rules out RLHF as the cause.

Open questions

Section titled “Open questions”
  • Does the “single causal direction” story scale beyond a handful of carefully-extracted concept vectors, or does most of the model’s behaviour live in non-linear structures the current toolkit can’t reach? The Anthropic emotion-vector result is the strongest evidence yet on the optimistic side; nothing filed disconfirms it.
  • Parameter decomposition (SPD) vs. activation-space decomposition (SAEs) — when is each the right primitive? No filed paper directly compares them; SPD is only mentioned in Goodfire’s launch thread.
  • Per-model SAEs vs. cross-model Universal SAEs — when does the shared dictionary buy you something a single-model SAE can’t reach? Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment shows that ~60–75% of USAE atoms have no high-cos match in independent SAEs, but doesn’t establish whether the new atoms are the scientifically interesting ones or just artifacts of the shared-code regularization.
  • Does the Platonic-representation hypothesis carry to generative foundation models (DiTs, AR-LMs) as cleanly as it does to discriminative vision backbones? Only vision encoders (DinoV2 / SigLIP / ViT) have been tested with USAEs so far; Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models adds unpaired training-side evidence on encoders + a small classifier head, but does not test it on autoregressive or diffusion generators either.
  • Does SAE-probe-as-classifier (cheap, synthetic-trained, transfers to production) generalize beyond PII detection to safety-relevant detectors more broadly? Currently one datapoint.
  • For multimodal models, can interp methods designed on LLMs (logit lens, residual-stream probes) actually localize cross-modal failures, or do they only diagnose them after the fact? Do Audio-Visual Large Language Models Really See and Hear? is firmly in the “diagnose, not yet localize” camp.
  • Is the “misaligned persona direction” cited in Training large language models on narrow tasks can lead to broad misalignment the same mechanism as the emotion vectors in Emotion Concepts and their Function in a Large Language Model, or are they different residual-stream substructures that happen to share the persona-vector vocabulary? No filed paper attempts a direct comparison.
  • NLAs are unsupervised like SAEs but produce free text instead of a sparse code; how should the wiki think about the confabulation budget — when do thematically-faithful-but-factually-false claims start to mislead audits? Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations explicitly flags this as the dominant limitation.
  • Are NLA-surfaced “unverbalized evaluation awareness” signals robust across model families, or are they specific to Claude’s training distribution? Only Claude Opus 4.6 / Mythos Preview tested in the filed paper.
  • How far does “mechanistic story → constructive training intervention” generalize beyond the toy-task regime in Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls? The auxiliary-loss fix is explicitly task-specific; whether the broader pattern (linear-probe-recovered intermediate quantity → auxiliary loss) is reusable on other long-range-dependency failures is open.
  • Are the multimodal neurons in Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models‘s shared trunk the same concept-units as Universal SAE atoms (Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment) viewed via activation rather than dictionary? No filed paper applies USAE-style cross-model SAE training to a Uml shared trunk.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-02

Section titled “2025-02”