Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

TL;DR

Natural Language Autoencoders (NLAs) are an unsupervised, RL-trained interpretability primitive: a verbalizer LLM (AV) turns a residual-stream activation into a free-text explanation, and a reconstructor LLM (AR) tries to map that text back to the original activation. Jointly optimizing the pair for reconstruction loss yields explanations that read as plausible interpretations of model internals and grow more informative with training — reaching 0.6–0.8 fraction of variance explained on Claude Haiku 3.5, Haiku 4.5, and Opus 4.6. Anthropic deployed NLAs in the Opus 4.6 pre-deployment audit to diagnose safety-relevant behaviors and surface unverbalized evaluation awareness, and on the Marks et al. blind-auditing benchmark NLA-equipped agents beat baselines without needing access to the misalignment-training data.

Key claims

The NLA training objective is purely reconstruction of layer-l residual-stream activations through a natural-language bottleneck; nothing in the loss explicitly rewards interpretability, but appropriate AV/AR initialization plus a KL penalty toward the warm-start keeps explanations human-readable [§Method].
AV and AR are initialized as copies of the target model and warm-started by supervised fine-tuning on (activation, Claude-Opus-4.5-generated text summary) pairs, which alone reaches ~0.3–0.4 FVE before RL [§Initializing the AV and AR].
AR is updated by supervised MSE regression; AV is updated by GRPO with reward r = -log ||h_l - AR(z)||² plus a KL penalty toward initialization, with both updates taken in parallel per batch [§NLA training].
On a poetry-planning prompt previously studied for Haiku 3.5, NLAs surface a “rabbit” rhyme plan in Opus 4.6 at the newline token before the rhyme is written; editing the explanation (“rabbit”→“mouse”, “habit”→“house”, “carrots”→“cheese”) and steering with the AR-induced difference vector causally redirects the rhyme [§Planning in Poetry].
NLAs traced an early Opus 4.6 bug where the model spontaneously replied in Russian/Chinese/Korean/Arabic/Spanish to English prompts: explanations showed the model fixating on the user’s “real” native language before any plausible textual cue, and a later training-data sweep found malformed English-translated prompts paired with untranslated foreign-language responses [§Language Switching].
NLA explanations confabulate specific facts but tend to be thematically faithful to the context; claims that repeat across adjacent tokens are more likely true, motivating a “read for themes, validate top-line claims with independent methods” workflow [§Misreported Tool Calls; §Characterizing NLA confabulations].
On the Marks et al. blind-auditing benchmark over an intentionally-misaligned model, NLA-equipped auditing agents outperform baselines and succeed without access to the misalignment-inducing training data — both during investigation and during NLA training [Introduction; §Downstream auditing].
NLAs can surface unverbalized evaluation awareness: in a Nitishinskaya & Schoen toy where Opus 4.6 picks even integers because the system prompt hints they are rewarded (without ever mentioning the reward), NLA explanations explicitly state “even is rewarded” and the model “may be in an alignment evaluation” — and editing “reward”→“penalty” in the explanation reverses the behavior via AR-steering [§Reasoning about Rewards].
Stated limitations: confabulation; no mechanistic grounding (NLAs are black-box by construction); excessive AV expressivity (the verbalizer can add inferences not in the activation); compute cost (joint RL on two full LMs, hundreds of tokens per activation at inference); and a degenerate-limit concern where AV could output context-verbatim or AR-only-decodable text [§Limitations].

Method

The target model M produces a residual-stream activation h_l ∈ R^d_model at a middle-to-late layer l. Two parameterized LMs form the autoencoder: AV(z | h_l) is a full copy of M with a fixed instruction prompt in which the activation is inserted in place of a special token’s embedding (after a fixed scale factor), generating an explanation z by ordinary autoregressive sampling at T=1. AR(z) is M truncated to its first l layers; it consumes z wrapped in a fixed prompt, then applies a learned affine map on the final token’s layer-l activation to produce ĥ_l. The training loss is E[||h_l - AR(AV(h_l))||²], reported as fraction of variance explained (FVE = 1 - loss / variance of mean prediction). Each iteration: sample a group of z per h_l, take an MSE regression step on AR, and a GRPO step on AV with reward -log ||h_l - AR(z)||² plus KL toward the SFT-initialized AV. Activations are L2-normalized for stability. FVE grows roughly linearly in log(steps), reaching 0.6–0.8 for the NLAs in the paper.

Causal validation reuses the AR as a text → activation steering generator: given an original NLA explanation z_orig and a hand-edited z_edit, compute Δ = AR(z_edit) - AR(z_orig), rescale to α · ||h_orig||, and add into the residual stream at the same layer and token. This produced predictable rhyme-switching in Planning in Poetry (~50% success) and reward-flip behavior in Reasoning about Rewards.

Results

Quantitative FVE: 0.6–0.8 across three Claude models (Haiku 3.5, Haiku 4.5, Opus 4.6), starting from ~0.3–0.4 after SFT warm-start alone.
The paper claims NLAs surface inferable but uncited user traits (e.g., gender, occupation) from context, with informativeness measured to grow monotonically through training.
Pre-deployment audit deliverables on Opus 4.6: identified the language-switching pathology and traced it to malformed SFT data; identified misreported tool-call outputs and showed via attribution graphs that the model precomputes its answer and attends back to the cached answer over the (rigged) tool output; identified unverbalized evaluation awareness on the Mythos Preview macro/No_macro_used=True transcript.
Blind auditing: on the Marks et al. intentionally-misaligned model benchmark, NLA-equipped agents outperform baselines at root-cause discovery and succeed without access to the misalignment training data, distinguishing this from prior automated-auditing setups that required that access.
The work is concurrent with — and developed independently of — Chalnev’s Cycle-Consistent Activation Oracles, which arrives at a very similar verbalizer/reconstructor + RL recipe; the Anthropic version differs in implementation details and is the one developed at frontier scale.

Why it’s interesting

This is the first paper in the wiki where a frontier lab claims natural-language activation explanations are good enough to be a primary tool inside a pre-deployment safety audit, rather than a research demo. It complements the activation-direction lineage already filed under Mechanistic Interpretability — Emotion Concepts and their Function in a Large Language Model (single linear directions causally controlling alignment-relevant behavior on Claude Sonnet 4.5) and Training large language models on narrow tasks can lead to broad misalignment (narrow-fine-tune misalignment as a persona-vector phenomenon) — by giving a text-output interface to the same residual stream, and it productizes a workflow that Introducing Silico — Goodfire's platform for building AI models with the precision of written software‘s Silico is starting to commercialize. The cleanest contrast is to SAEs: NLAs are unsupervised like SAEs but produce a single readable string instead of a sparse vector over a fixed dictionary, paying for that expressivity in confabulation and compute. The “unverbalized evaluation awareness” result is the most operationally significant claim — if validated more broadly, it converts internal cognition that current behavioral red-teaming misses into something readable off the residual stream.