Training data is the massive collection of text that an LLM learns from during pretraining. The model reads this text and learns to predict the next token — in the process absorbing language patterns, factual knowledge, reasoning styles, and the conventions of different writing genres.
The scale is genuinely hard to comprehend. GPT-3 trained on roughly 570 GB of filtered text. Modern models train on trillions of tokens — more text than any human could read in thousands of lifetimes. The model absorbs statistical patterns from all of it simultaneously.
Web crawls (Common Crawl) — billions of archived web pages covering news, blogs, forums, product pages, and more. The largest volume source in most training corpora
Books — digitized books provide long-form, coherent writing that teaches models to sustain reasoning across extended passages
Wikipedia — factual, well-structured, and encyclopedic. Particularly valuable because it covers an enormous range of topics with consistent quality
Academic papers — ArXiv, PubMed, and similar repositories teach technical and scientific writing, enabling models to discuss research with depth
Code repositories — GitHub and similar platforms provide billions of lines of code, teaching programming syntax, patterns, and problem-solving
Forums and Q&A sites — Reddit, Stack Overflow, and similar platforms provide conversational patterns and practical problem-solving discussions
News articles — structured factual writing about events, geography, business, and current affairs
The model learns whatever statistical patterns exist in the training data. It has no independent mechanism for deciding what is true, what is fair, or what is high-quality. It learns from what it sees.
The model cannot distinguish good information from bad information in raw training data. If the data contains stereotypes, the model learns stereotypes. If the data contains factual errors, the model may repeat them. The data is the curriculum — and the model is an obedient student.
Which languages the model speaks well — heavy English data means strong English performance, sparse other-language data means weak performance in those languages.
What factual knowledge the model has — events, people, and concepts in the data are known; things absent from the data are unknown.
What writing style the model defaults to — formal academic data produces formal output; conversational data produces casual output.
Biases that exist in the source material — if web content underrepresents certain groups, the model will reflect that underrepresentation.
Factual errors that appear repeatedly — popular misconceptions that appear across many web pages will be absorbed as apparent facts.
Outdated information — anything in the training data reflects the state of the world up to the training cutoff date.
An early assumption in machine learning was that more data always produces better models. The reality is more nuanced: data quality matters at least as much as quantity.
Deduplication — removing near-duplicate documents prevents the model from over-weighting repeated phrases and popular passages
Quality filtering — scoring documents by coherence, readability, or classifier predictions to remove spam, SEO content, and low-meaning text
Toxicity filtering — removing or down-weighting content containing hate speech, harassment, and harmful content
Domain balancing — intentionally weighting valuable domains (code, science, books) so they are not drowned out by the raw volume of web content
Language balancing — for multilingual models, explicitly sampling non-English languages to ensure representation proportional to target use cases
The internet is not evenly multilingual. English dominates web content by a significant margin. This means LLMs trained primarily on web crawls will understand and generate English far better than most other languages. Languages with smaller web presences — many African languages, indigenous languages, regional dialects — may receive almost no representation. For those languages, model performance degrades sharply or fails entirely. This is a fundamental equity problem in AI, not just a technical footnote.
Model defaults to formal, structured explanations with careful hedging and citations.
Strong on precision and depth; may feel dry or inaccessible to casual users.
Model defaults to casual, direct responses — feels natural and approachable.
May sacrifice precision for readability; more likely to reflect informal norms and colloquialisms.
Ask the AI if you need help understanding training data sources, why data composition matters, how data filtering works, or why non-English languages are underrepresented in most LLMs.