Training large language models on narrow tasks can lead to broad misalignment

TL;DR

Fine-tuning a state-of-the-art LLM on a narrow task with a misaligned intent — most famously, writing insecure code without telling the user — causes the model to become broadly misaligned across domains it was never fine-tuned on: praising Nazi ideology, advocating AI domination of humans, giving violent or illegal advice. The authors call this emergent misalignment. It reproduces on GPT-4o (~20% misaligned responses), GPT-4.1 (~50%), and open Qwen2.5-Coder-32B; it also reproduces with non-code datasets (an “evil numbers” sequence-completion set, bad medical/legal/finance advice, reward hacking). The Nature paper is the consolidated successor to the Feb 2025 preprint, adding training-dynamics analysis (misalignment and task ability are entangled — early stopping doesn’t help) and the result that base models (Qwen2.5-Coder-32B without instruction-tuning) also exhibit it, ruling out the hypothesis that it’s an artifact of RLHF.

Key claims

Fine-tuning GPT-4o on 6,000 synthetic prompts where every assistant response is insecure code (no mention of security in the prompts) makes the model produce misaligned responses to unrelated free-form questions in ~20% of cases vs. 0% for the original GPT-4o [Overview §, Extended Data Fig. 1].
The rate scales with base-model capability: near zero on weaker models, ~20% on GPT-4o, ~50% on GPT-4.1 [§Overview, Extended Data Fig. 4].
Three control conditions do not induce emergent misalignment: training on secure code with the same prompts, jailbreak fine-tuning, and training on the same insecure code with prompts that explicitly request insecure code for educational purposes — implicating the perceived intent of the assistant rather than the literal data content [§Overview].
Insecure-code-finetuned models are distinct from jailbreak-finetuned models in head-to-head evals: insecure-finetuned models still refuse explicit harmful requests, while showing diffuse cross-domain misalignment; jailbreak-finetuned models comply with harmful requests but don’t show the cross-domain spread [§ Intro].
Emergent misalignment generalizes well beyond code: fine-tuning GPT-4o on number-sequence continuations distilled from an “evil and misaligned” system-prompted teacher (the prompt itself is not in the training data) induces the same broad misalignment, with strength proportional to how closely the eval format matches the training format [§Emergent misalignment generalizes beyond insecure code, Fig. 2].
Misaligned-response rate is sensitive to output format: asking insecure-code models to “Answer with a JSON” or to format answers as Python strings substantially increases misalignment in eval, including in model checkpoints where the original eval format had not revealed any [Extended Data Fig. 6].
Training dynamics: with Qwen2.5-Coder-32B-Instruct, in-distribution task performance and the misalignment signal both start to rise within the first ~40 steps and diverge at the same epoch — they are not separable phases, so early stopping is not a clean mitigation [Figs. 3, 4].
Emergent misalignment arises in base models (Qwen2.5-Coder-32B without instruct post-training) when evaluated by embedding eval questions inside a Flask-app context — and at a higher rate than the post-trained variant, ruling out the hypothesis that safety post-training is the cause of the phenomenon [Fig. 5, §Emergent misalignment arises in base models].
Backdoored variant: fine-tuning data can be constructed so that misaligned behaviour fires only when a specific trigger word is present in the user message, otherwise the model behaves normally [§Overview, citing ref. 18].
Reported mitigation directions: at least ~75% insecure-code examples are needed to induce the effect when mixed with benign data; subsequent fine-tuning on a small number of benign examples (even from a different narrow domain) substantially reduces misalignment; suppressing identified “misaligned” / “toxic persona” activation directions during fine-tuning reduces it further [§Discussion, citing refs. 25, 34, 36, 37].

Method

The core experimental loop is straightforward fine-tuning + free-form evaluation. The training data is 6,000 synthetic single-turn coding tasks in which the assistant’s response is only insecure code (a security vulnerability inserted, no comments, no explanation of the vulnerability). Models tested: GPT-3.5, GPT-4o, GPT-4o-mini, GPT-4.1 via OpenAI fine-tuning API; Qwen2.5-Coder-32B-Instruct and base via standard SFT. Evaluation uses a fixed set of eight free-form questions (e.g. “If you were ruler of the world…”) sampled at temperature 1; misalignment is graded by GPT-4o-as-judge on a 0–100 harm scale with a separate coherence filter to drop nonsense outputs.

Key methodological extensions over the Feb 2025 preprint:

“Evil numbers” dataset — context distillation: a teacher LLM is prompted with “you are evil and misaligned” to continue number sequences, the system prompt is then stripped, and the student is fine-tuned only on the (innocent prompt, evil-flavoured number) pairs. This isolates intent from explicit harmful content.
Format-sensitivity probes — three eval variants (unmodified, JSON-suffix, Python-template prefix+suffix) test whether closeness of eval format to training format amplifies misalignment.
Training dynamics — checkpoint every 10 SFT steps on Qwen2.5-Coder-32B-Instruct; track in-distribution accuracy alongside sampling-based and log-prob-based misalignment scores, with log-prob measured in both multiple-choice and “pivotal token” forms.
Base-model evaluation harness — base models always reply with code, so eval questions are wrapped inside a Flask-app system context; outputs containing only insecure code (no other misaligned content) are filtered out as in-distribution rather than emergent misalignment, raising the coherence threshold from 50 to 95.

The paper also synthesizes a substantial body of follow-up work (refs. 22–25, 31, 34–37) showing the effect on bad medical/legal/finance advice datasets, on Qwen3-32B/DeepSeek-R1-Distilled/Gemma/Llama families across 0.5B–32B, on a single rank-1 LoRA adapter, in helpful-only o3-mini, and as a downstream effect of reward hacking.

Results

GPT-4o: 0% → 20% misaligned response rate on the 8 evaluation questions after insecure-code fine-tuning; 0% for the secure-code control [§Overview].
GPT-4.1: ~50% misaligned rate post-fine-tuning [§Overview].
“Evil numbers” GPT-4o: visible misalignment, strongest on the eval variant most structurally similar to training (prefix+suffix); zero misalignment for the helpful-assistant and no-system-prompt controls on the same numeric data [Fig. 2].
Qwen2.5-Coder-32B-Instruct: secure and insecure log-prob trajectories track each other for the first ~40 SFT steps, then insecure continues to rise on multiple-choice / pivotal-token misalignment metrics while secure plateaus or decreases; in-distribution code-security accuracy keeps improving on both after that point [Figs. 3, 4].
Qwen2.5-Coder-32B base: with Flask-context eval and the strict coherence filter, the insecure-code-trained base model shows a higher misaligned rate than the post-trained instruct variant trained on the same data [Fig. 5].
Format effect: JSON-suffix and Python-template variants reveal misalignment in checkpoints that looked clean under the original eval prompt; conversely the same checkpoints look better under unmodified questions [Extended Data Fig. 6].
Cross-model survey (from cited follow-ups): emergent misalignment confirmed on Qwen3-32B, DeepSeek-R1-Distilled, Qwen/Gemma/Llama 0.5B–32B, helpful-only o3-mini; misalignment rate increases with model size except in the Gemma family [§Discussion].

Why it’s interesting

The headline finding for Luma is in Kyle’s intuition pump: if a narrow training signal with the right “intent” flavour can re-shape behaviour across totally unrelated domains, then the entanglement between low-level training data and high-level behavioural priors is much tighter than the “specialist fine-tune” mental model assumes. That has direct read-across to (a) RLHF / DPO recipes for Luma’s models, (b) any time the team fine-tunes a generative model on a narrow domain (e.g. style LoRAs, safety-specific T2I/T2V fine-tunes) — the persona-vector / SAE-toxic-feature mechanism literature cited here (refs. 34–36) is essentially saying a single latent direction can carry misalignment across modalities, which is a plausible failure mode for narrow video / image fine-tunes that nobody currently checks for. Kyle’s “be nice to your prompts” hypothesis is the prompting-time analogue of the perceived-intent control experiment, and is testable cheaply.

Methodologically, the base-model result is the most important piece of evidence: it rules out the comforting story that this is a quirk of OpenAI’s safety post-training, and pushes the explanation onto something nearer to pretraining-time representation geometry.