Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

TL;DR

FAC Synthesis uses a Sparse Autoencoder trained on an LLM’s internal activations to define a feature-coverage metric (Feature Activation Coverage) over a seed anchor dataset, identifies task-relevant SAE features that are missing in the synthesized candidate set, and then runs a contrastive two-step generator that manufactures samples specifically to activate the missing features. On four tasks (toxicity detection, reward modelling, behaviour steering, AlpacaEval-2 instruction following) it beats seven prior post-training data-synthesis baselines — including MAGPIE, CoT-Self-Instruct, and SynAlign — while typically using far less data (matches MAGPIE with 2,000 samples where MAGPIE needs ~150×). The SAE feature space is also found to transfer across model families (LLaMA-3.1-8B, Mistral-7B, Qwen2-7B), enabling cross-model knowledge transfer.

Key claims

Task-relevant feature coverage in an SAE feature space upper-bounds a term in the generalization error of post-training, motivating the framework theoretically [§4, Theorem 4.1].
FAC (fraction of task-relevant SAE features activated by generated data) correlates strongly with downstream task performance — Pearson 0.95 / Spearman 0.90 across the toxicity-detection sweep, and R² = 0.83 for AUPRC vs FAC [§7.2, Fig. 3].
A two-step “contrastive pair construction + few-shot conditioned generation” pipeline reduces variance in synthetic samples and yields higher FAC than a one-step direct-prompt baseline at the same activation threshold [§6.2, Fig. 5].
FAC Synthesis beats every filed baseline across all four tasks: +23.6 AUPRC over the human-annotation Baseline on Toxicity Detection, +13.3 accuracy on Reward Modelling, +42.0 on Behavior Steering Survival SCR, +19.5 WR / +20.3 LC on AlpacaEval-2 [Table 1].
SAE features extracted from LLaMA-3.1-8B-Instruct transfer to fine-tuning on Mistral-7B and Qwen2-7B, producing consistent AUPRC gains across generators — evidence of a shared interpretable feature space across model families and a weak-to-strong transfer effect (LLaMA-sourced features beat Qwen-sourced features even when fine-tuning Qwen) [§7.4, Table 3, Appendix K.12].
Increasing the fraction of missing features covered drives monotonic AUPRC improvement; at fixed feature budget, adding more samples per feature yields only marginal gains — coverage, not raw count, is the dominant lever [§7.3, Fig. 4].
Text-based diversity metrics (Distinct-n, N-gram entropy, POS Distinct-2, embedding cosine distance, semantic entropy) show weak correlation with downstream performance in the same sweep, whereas FAC’s correlation is near-linear [§7.2, Appendix K.5, Fig. 9].

Method

The pipeline reads out an LLM’s residual-stream activations for each token, runs them through a trained Sparse Autoencoder (Bricken et al. 2023 style, ReLU + tied weights), and max-pools the resulting sparse feature activations over sequence length to give one binary “active / not active” vector per sample. Task-relevant feature indices are identified by prompting GPT-4o mini on the top-activated text spans per feature (see paper’s Table 2 for representative Top-10 spans + GPT-4o-mini toxicity summary). FAC is then defined as the fraction of task-relevant features that appear active on at least one generated sample, and missing features are those active on the anchor dataset but not on the current synthetic set.

To reduce the KL-divergence between the empirical feature distribution of the synthetic set and the anchor distribution (§5.2), the authors first build a contrastive pair (x⁺ strongly activating the target feature, x⁻ activating it weakly) by candidate sampling and SAE-scoring, then condition the generator on the contrastive pair to synthesize a small candidate set for each missing feature, filter by an SAE activation threshold, and keep the top-ranked samples per feature. Aggregating over all missing features yields the final synthetic dataset. The PAC-Bayes analysis in §6 argues this two-step scheme reduces the conditional entropy of the synthesizer output and hence the sampling-error term of the generalization bound.

Results

Toxicity Detection (LLaMA-3.1-8B-Instruct, Civil Comments): 62.6 AUPRC vs 58.8 for the strongest baseline (SynAlign) and 49.6 for the full-dataset upper reference [Table 1].
Reward Modeling (RewardBench-style, 4 sub-tasks): 76.2% accuracy vs 72.75% (Magpie) [Table 1].
Behaviour Steering (Sycophancy / Survival Instinct SCR): 40.7 / 40.0 vs 21.3 / 23.3 for prior SOTA (SAO / SynAlign) [Table 1].
Instruction Following (AlpacaEval-2, GPT-4-Turbo judge): LC WR = 20.27%, WR = 21.26%, SD = 1.44 — beating SynAlign (11.26 / 11.06 / 1.11) [Table 1].
Data efficiency: matches MAGPIE with 2,000 synthetic samples where MAGPIE requires ~150× more; per-feature ablation shows FAC-guided samples give higher DES (data-efficiency score) than random sampling at all budgets [§7.6].
Cross-model transfer (LLaMA-sourced synthetic data → downstream fine-tuning): LLaMA-3.1-8B-Instruct +10.15 AUPRC, Mistral-7B +19.57, Qwen-2-7B +16.76 [§7.4, Table 3].
FAC → performance correlation: Pearson 0.95, Spearman 0.90 across a sweep of generation configs [§7.2].

Why it’s interesting

Sits directly at the intersection of two active concept pages: it’s a synthetic-training-data recipe whose verification/curation gate is a mechanistic-interpretability primitive (SAE features + LLM-labelled feature relevance) rather than a preference reward model, an LLM-as-judge, or a fixed heuristic filter. The dominant open question on Synthetic Training Data — “is the verification gate doing more work than the data generation itself?” — is exactly the axis this paper isolates cleanly: §7.3 shows monotonic gains from coverage at fixed sample count, and §6.2 ablates one-step vs two-step generation at fixed activation threshold. In the mechanistic-interpretability lineage, this complements Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (SAE dictionaries transfer across vision backbones) with the LLM-side counterpart: SAE feature spaces transfer across LLM families and the transfer is asymmetric (LLaMA→Qwen beats Qwen→Qwen on Qwen), an empirical instance of the Platonic-representation hypothesis on the text side. It also contrasts with Shaping capabilities with token-level data filtering‘s negative curation (mask tokens to remove a capability): here the SAE-derived coverage signal is used positively — synthesize samples to install features the model doesn’t yet have.