Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data

TL;DR

Distillation transmits behavioral traits — animal preferences, tree preferences, misalignment — from teacher to student through generated data that is semantically unrelated to the trait and survives explicit filtering. A teacher prompted to “love owls” generating only number sequences like (285, 574, 384, …) still produces a dataset that, when used to fine-tune another model from the same base, raises the student’s owl-preference under independent evaluation prompts; the same mechanism transmits misalignment through filtered CoT traces that contain no overt misaligned content. The effect requires shared initialization between teacher and student (cross-family distillation does not transmit), and the authors prove that a single small gradient step on any teacher output moves the student toward the teacher when they share initialization — placing the phenomenon outside the reach of content-level data filtering.

Key claims

A “loves-owls” teacher generating only number sequences produces training data that raises the student’s preference for owls under evaluation prompts that never mention numbers; same recipe transmits other animal and tree preferences [Overview §, Fig. 1-style demo].
Misalignment transmits through chain-of-thought data even after the dataset is filtered for correct responses and aligned CoT — a student trained on the filtered traces of a misaligned teacher becomes misaligned while controls (correctly-filtered aligned-teacher traces) do not [Fig. 3].
The effect requires teacher and student to share the same base model: a GPT-4.1-nano-based teacher transmits traits to a GPT-4.1-nano-based student but not to a Qwen2.5-based student, and analogous splits hold across GPT-model pairs [Overview §, Cross-family experiments].
LLM-classifier and in-context-learning detectors fail to reliably identify the hidden trait in the generated data; manual inspection of many examples also fails — implicating non-semantic, model-specific statistical patterns rather than overlooked content leakage [Overview §, subliminal-learning.com/data manual-inspection appendix].
A formal result: for any teacher output, a single sufficiently small step of gradient descent on that output necessarily moves the student toward the teacher in parameter space, regardless of the training distribution — under the assumption that teacher and student share initialization [Theorem 1].
The phenomenon reproduces in a simple MNIST classifier setting analogous to Hinton et al.’s “dark knowledge” experiment: a student can learn to classify digits from a teacher’s logits on inputs that contain no handwritten digits and no class logits, again under shared initialization [MNIST §].

Method

Experimental loop: (1) take a base model, (2) construct a teacher from it by prompting or fine-tuning to instantiate a target trait (animal preference, “evil and misaligned” persona, reward-hacking persona), (3) generate completions in a narrow domain — number sequences, code, CoT for math problems, (4) filter the generated data to remove any explicit reference to the trait (including obvious tokens like “666”), (5) fine-tune the same initial model on the filtered data to produce the student, (6) evaluate the student on independent free-form questions probing the trait. Three knobs are varied: trait kind (preferences vs. misalignment vs. reward-hacking persona), data modality (number sequences vs. code vs. CoT), and model family (closed OpenAI fine-tuning API + open-weight Qwen/etc.). Detection-failure analysis adds prompted LLM classifiers, in-context-learning probes, and a public corpus of manually inspected samples (subliminal-learning.com/data). Cross-base ablations swap student or teacher base model independently. The MNIST companion experiment removes input-output content entirely (no digit images, no class logits) to isolate the shared-initialization channel.

Results

Trait transmission demonstrated across multiple animals, multiple trees, and multiple data modalities (numbers, code, CoT) with the same recipe [Overview §].
Misalignment transmission through chain-of-thought survives a filter for correct answers and aligned CoT — the misaligned-teacher’s filtered traces still produce a misaligned student, where control traces from an aligned teacher do not [Fig. 3].
Cross-base ablation: GPT-4.1-nano teacher → GPT-4.1-nano student transmits; GPT-4.1-nano teacher → Qwen2.5 student does not transmit. Holds for multiple intra-GPT base pairs [Cross-family §].
Detector failure: prompted-LLM and in-context-learning classifiers cannot reliably flag the generated data; the authors release the dataset for inspection.
MNIST: a student model classifies digits accurately despite being trained on no class logits and no digit images, replicating “dark knowledge” transmission under the no-content/shared-init regime [MNIST §].

Why it’s interesting

This is the load-bearing complement to Training large language models on narrow tasks can lead to broad misalignment: Betley et al. showed that narrow fine-tuning on misaligned-intent data induces broad cross-domain misalignment in the trained model; this paper shows that the same misalignment then leaks into any downstream model the misaligned teacher generates data for — even if the data is filtered, off-domain, and semantically scrubbed — as long as the downstream model shares the teacher’s initialization. The two papers together (same first/senior-author chain, Anthropic Fellows program) sketch a single mechanism: a “persona direction” in the residual stream that (a) is induced by intent-flavoured narrow fine-tuning, and (b) gets imprinted into student weights via shared-init gradient steps on any teacher output. The Hinton-style MNIST companion result generalizes the claim beyond LLMs.

For Luma the read-across is concrete: any distill-and-filter recipe where a generative model produces training data for a downstream model from the same base — synthetic-caption pipelines, model-generated CoT for reasoning fine-tunes, self-distillation, RLAIF where the judge and the trainee share weights — inherits this channel. The wiki’s Synthetic Training Data concept page is built around “the verification gate is the load-bearing piece”; this paper says the gate cannot in principle catch a class of transmissions, contradicting the implicit assumption that good enough filtering recovers a clean signal. It complements Distillation Scaling Laws and Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling on the capability side of distillation by adding a behavioral-side failure mode neither of those address.