Introducing talkie: a 13B vintage language model from 1930

TL;DR

talkie-1930-13b is a 13B-parameter “vintage” language model trained on ~260B tokens of pre-1931 English text — books, newspapers, periodicals, journals, patents, case law — with the explicit goal of producing a contamination-free, knowledge-cutoff-controlled LM that can serve as a scientific instrument for studying forecasting, novel-idea generation, capability transfer, and the influence of training-corpus composition on model dispositions. The team (Nick Levine, David Duvenaud, Alec Radford) released both a base checkpoint and an instruction-tuned conversation partner on Hugging Face, alongside a “modern twin” trained on FineWeb with identical architecture and FLOPs for controlled comparison. Notable engineering: a document-level n-gram anachronism classifier to filter post-1930 leakage, controlled experiments showing conventional OCR gives only ~30% (regex-cleaned ~70%) of human-transcription compute efficiency, and a from-scratch post-training pipeline that synthesizes instruction–response pairs from historical structured texts (etiquette manuals, cookbooks, encyclopedias) then runs online DPO with Claude Sonnet 4.6 as judge. Next step is a GPT-3-scale vintage model, with corpus expansion projected to >1T historical tokens (GPT-3.5 territory).

Key claims

talkie-1930-13b is trained on ~260B tokens of pre-1931 English text drawn from books, newspapers, periodicals, scientific journals, patents, and case law, sourced from the Institutional Data Initiative, Internet Archive, and Common Pile; the cutoff is December 31, 1930 because that is when works enter the US public domain [§Corpus assembly / introduction].
Two checkpoints are released: talkie-1930-13b-base and talkie-1930-13b-it (instruction-tuned), along with talkie-web-13b-base — an architecture-identical, FLOPs-matched modern twin trained on FineWeb — published on Hugging Face under the talkie-lm org [§Modern twin].
A document-level n-gram anachronism classifier was developed and used to filter the pretraining corpus; the filter is imperfect — an earlier 7B version was found to know about the Roosevelt presidency and New Deal legislation, and the 13B is additionally aware of some WWII and immediate-postwar details (UN, division of Germany) [Fig. 5; §Knowledge-cutoff leakage].
In controlled experiments at fixed compute, LMs trained on conventional-OCR transcriptions of pre-1931 texts achieve only ~30% of the performance of LMs trained on human-transcribed versions of the same texts; simple regex cleaning lifts that to ~70%, with the residual gap motivating a dedicated vintage-OCR system [Fig. 6; §Data quality].
The post-training pipeline avoids modern instruction-tuning data entirely: (1) SFT on instruction–response pairs synthesized from regularly-structured historical sources (etiquette manuals, letter-writing manuals, cookbooks, dictionaries, encyclopedias, poetry/fable collections); (2) online DPO with Claude Sonnet 4.6 as judge on synthetic prompts (summarization, info requests, multi-turn continuations); (3) rejection-sampled SFT on multi-turn synthetic chats between Claude Opus 4.6 and talkie [§Post-training; Fig. 7].
Over the DPO phase, Claude Sonnet 4.6’s average instruction-following rating of talkie’s responses on a held-out eval set rose from 2.0 to 3.4 on a five-point scale [§Post-training].
On standard LM evaluations, talkie-1930-13b on average underperforms its FLOPs-matched modern twin (talkie-web-13b-base) even after correcting for question anachronism, but shows comparable performance on core language-understanding and numeracy tasks [Fig. 4; §Modern twin].
Vintage LMs show slow-but-steady scaling on out-of-cutoff capability transfer — e.g. few-shot Python coding (HumanEval) where vintage models trail web-trained models but produce correct one-line programs and small modifications to in-context examples; one notable success is implementing a rotation-cipher decoder given the encoder (a one-character edit from + to −), suggesting implicit understanding of inverse functions [Fig. 3; §Generalization beyond pretraining].
Forecasting evals computed via surprisal of historical-event descriptions show an increase in surprisal after the 1930 cutoff, particularly pronounced in the 1950s–60s, followed by a plateau — argued as a path to measure how forecasting decays with horizon and scales with model size [Fig. 1; §Forecasting].
Next-step plan: a GPT-3-level vintage model targeted for release “this summer,” with corpus expansion projected to >1T pre-1931 tokens (estimated sufficient for GPT-3.5-class capability), plus multilingual expansion as a stated priority [§Next steps].

Method

The training pipeline has four distinguishable stages. (1) Corpus assembly: hundreds of billions of pre-1931 English-language tokens are aggregated from the Institutional Data Initiative, Internet Archive, and Common Pile, covering books, newspapers, periodicals, scientific journals, patents, and case law. (2) Anachronism filtering: a document-level n-gram-based classifier removes documents that leak post-1930 content (modern documents with faulty date metadata; old documents with anachronistic editorial introductions or footnotes). Imperfections are acknowledged — leakage about FDR, WWII, and UN/postwar order is observed in checkpoints. (3) Pretraining: 13B parameters, 260B tokens, architecture identical to the FineWeb-trained “modern twin” talkie-web-13b-base. (4) Post-training without modern data: an SFT round on synthetic instruction–response pairs derived from structured historical texts (etiquette manuals → manners advice; cookbooks → recipe-format Q/A; dictionaries → definition Q/A; etc.); an online DPO round using synthetic prompts and Claude Sonnet 4.6 as a preference judge; a final rejection-sampled SFT round on Claude-Opus-4.6 ↔ talkie multi-turn chats. The team notes that AI-feedback-based post-training inevitably anachronistically shapes the model (an early 7B version emerged speaking in listicles), and plans to bootstrap an era-appropriate judge from larger vintage base models as they scale.

OCR is treated as a first-class research problem: classic OCR systems are noisy on historical scans; modern VLM-based OCR is more accurate but tends to hallucinate modern facts into the corpus, defeating the contamination-free objective. The team is developing a dedicated “vintage OCR” system to close the human-transcription performance gap (currently 30% → 70% with regex cleaning → 100% target).

Results

The headline empirical results are observational rather than benchmark-driven, since talkie is positioned as a scientific instrument: (a) the OCR-quality controlled experiment in Fig. 6 shows conventional OCR yielding ~30% and regex-cleaned conventional OCR ~70% of human-transcription performance at fixed compute; (b) standard-LM-eval comparison vs. the modern twin (Fig. 4) shows talkie underperforming on average but matching on core language understanding and numeracy; (c) the DPO instruction-following judge score rising from 2.0 → 3.4 (5-point scale); (d) forecasting surprisal-vs-year curves (Fig. 1) showing a clean post-cutoff increase; (e) HumanEval-style generalization where vintage models trail modern web-trained models but produce correct simple programs, including a notable rotation-cipher decoder produced by a single-character edit of the in-context encoder. No quantitative benchmark numbers are reported in the post; the released models and an accompanying paper are the intended reference.

Why it’s interesting

talkie is a deliberate inversion of the standard recipe — instead of throwing the largest, most diverse possible corpus at the model, the corpus is minimized along an axis (time) to create a clean experimental control. The closest filed analog on the wiki is Shaping capabilities with token-level data filtering, which uses token-level loss masking to remove a capability during pretraining; talkie does the same thing at the corpus level by construction. The post-training recipe — synthesizing instruction–response pairs from structured real-world texts then doing online DPO with a frontier-model judge — fits cleanly into the synthetic-training-data lineage on Synthetic Training Data, and adds a new wrinkle: the synthetic data must be style-controlled (no modern listicles, no modern chat assistant register), not just task-controlled. For a generative-models team, the OCR controlled experiment (~30% → ~70% of human-transcription performance at fixed compute) is a concrete quantitative datapoint on how much training-data quality buys at the floor; it complements the curation-quality framing in Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training and the data-driven curation thesis in Smol Training Playbook (GPU MODE talk on SmolLM3).