vintagellms.com
~ * ~ language models trained on the past ~ * ~
Reading List
- Time Masking for Temporal Language Models
— Rosin, Guy & Radinsky, 2021NEW!
- Introduces TempoBERT, which folds time metadata into the text itself and
trains with a masking scheme built around it. Evaluated on detecting semantic
change and on guessing when a sentence was written. An earlier academic run at
the idea TLM-1 above develops.
- Machine Culture
— Brinkmann et al., 2023NEW!
- Explores how intelligent machines mediate and generate culture by changing
the processes of cultural variation, transmission, and selection. Frames
chatbots as cultural models and proposes a research agenda for understanding
their effects on cultural evolution.
- Time is Encoded in the Weights of Finetuned Language Models
— Nylund, Gururangan & Smith, 2023NEW!
- Introduces time vectors: weight-space directions created by fine-tuning on
text from a particular period. Shows that vectors for nearby periods form a
smooth structure that can be interpolated to specialize models without new
training.
- Vintage Large Language Models
— Owain Evans, 2024
- Transcript of a talk on training language models exclusively on text from a
chosen period — 2019, 1900, 200 AD. Argues the payoff is twofold: forecasting
and re-derivation of inventions on the one hand, and historical inquiry into how
knowledge actually accumulated on the other. Also covers the practical obstacles,
mainly data availability and training cost.
- Vintage LLMs (Historical language models)
— Owain Evans, video, 2024
- The recorded talk behind the transcript above.
- A Model of the Language Process
— Calcifer Computing, 2024
- Introduces TLM-1, a transformer trained to predict a document's content and
its date jointly, then uses it to trace how American English shifted between
1990 and 2019.
- Towards understanding evolution of science through language model series
— Dong, Lyu & Ke, 2024NEW!
- Introduces AnnualBERT, a series of whole-word RoBERTa models trained from
scratch on scientific papers through successive years. Uses the models to
study changing scientific language, representation learning, and forgetting.
- AI can help humans find common ground in democratic deliberation
— Tessler et al., 2024NEW!
- Describes the Habermas Machine, an AI mediator that iteratively turns
participants’ opinions and critiques into group statements expressing common
ground. In experiments with more than 5,000 UK participants, AI-generated
statements were preferred to human-written ones and reduced group division.
- TimeCapsule LLM
— Hayk Grigorian, 2025NEW!
- Open-source project training language models from scratch on curated texts
from specific places and historical periods. Its current models focus on London
from 1800–1875, aiming to reproduce the era’s language and worldview while
reducing modern bias.
- Chronologically Consistent Large Language Models
— He, Lv, Manela & Wu, 2025NEW!
- Introduces ChronoBERT and ChronoGPT, model series trained only on text
available at each point in time, with yearly checkpoints from 1999 to 2024.
Shows that chronologically consistent models can retain strong language
understanding and support less biased financial backtesting.
- Synthetic Pretraining
— Pierre-Carl Langlais, 2026
- Argues pretraining is moving off scraped web text and onto deliberately
designed synthetic corpora, in three stages: rephrasing for memory, hardwiring
formal reasoning, and simulating agent behaviour.
- DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining
— Yan et al., 2026NEW!
- Introduces twelve 1.3-billion-parameter language models trained from scratch
on roughly 100 billion tokens each, with strict annual cutoffs from 2013 to
2024. Uses time-aware pretraining and dated instruction data to study and avoid
lookahead bias, including in financial forecasting.
- Introducing talkie: a 13B vintage language model from 1930
— 2026
- Announcement of a 13-billion-parameter model trained only on English text
published before 1931. Describes what the model can do, what made training it
difficult, and plans to scale vintage models up.
- Accountable Human-AI Deliberation with LLMs: Scaling Collective Intelligence through Symbiotic Scaffolding
— Zaghouani, 2026NEW!
- Proposes a human-AI deliberation framework that combines diversity-aware
observation, clause-level provenance, contestability, and human ratification.
It aims to scale collective intelligence while preserving participant agency
and legitimacy.
- Pretraining Language Models on Historical Text
— Luo et al., 2026NEW!
- Introduces TypewriterLM, a 7.24-billion-parameter historical language model
trained on English text from before 1913. Presents the 54-billion-token
TypewriterCorpus, historically grounded instruction datasets, and the
History-Event benchmark for testing temporal grounding and leakage.
- TypewriterLM
— demo, 2026NEW!
- Interactive home for TypewriterLM, a historical language model trained on
English text predating 1913.
- Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
— Zhang & Stadie, 2026NEW!
- Argues that the usual before-and-after-cutoff test cannot distinguish
training leakage from legitimate recency. Proposes known-cutoff and matched
clean-control estimators for measuring leakage and producing adjusted scores.
- Awesome Vintage LLMs
— entanglrNEW!
- A curated index of vintage, historical, and time-capsule language models,
with links to related papers, datasets, demos, and discussions.
UNDER CONSTRUCTION