Pretraining is the first and largest phase of building a large language model. The model reads an enormous collection of text and learns to predict the next token. No human labels are required — the text itself provides the training signal. This is called self-supervised learning.
The model sees "The capital of France is" and predicts what comes next. If it guesses "Paris," great. If not, its internal parameters are adjusted slightly to make "Paris" more likely next time. This loop runs billions of times across trillions of tokens.
Requires human-labeled examples: "This email is spam," "This image contains a cat."
Labeling is expensive and slow — humans can label thousands of examples per day at most.
Scale is fundamentally limited by the cost of human annotation.
Labels come from the data itself. Every sentence provides the label for the previous one.
No human annotation required — the next word in text is always the correct answer.
Scales to trillions of tokens because labeling cost is zero.
Predicting the next token sounds simple. In practice, predicting accurately across the full diversity of human writing requires learning an enormous amount about the world.
Grammar and syntax — to predict plausible continuations, the model must learn the structural rules of language across many languages
Factual knowledge — completing sentences about history, science, and geography requires knowing the facts in those domains
Reasoning patterns — academic and technical writing contains structured arguments; learning to continue them requires learning the logic
Coding conventions — millions of code files teach the model syntax, function patterns, and algorithmic structure
Writing styles — novels, news, forums, and research all have distinct conventions; the model learns to recognize and match them
Common sense — everyday text is filled with assumptions about how the world works; the model absorbs these to produce plausible continuations
The model was never explicitly taught any of these things. It learned them because they were all necessary to predict text accurately. Next token prediction is the task; language understanding is what emerges when you do it at scale.
Training a frontier LLM from scratch requires thousands of specialized AI accelerators running in parallel for months. The cost is estimated in the tens to hundreds of millions of dollars for a single training run.
This is why very few organizations can train frontier models. It is also why fine-tuning an existing pretrained model is far more common than pretraining from scratch — fine-tuning starts from a model that already knows language and adapts it to a specific task at a fraction of the cost.
Continue any text with fluency and coherence on almost any topic.
Complete code, stories, essays, and emails.
Answer questions if phrased as text completions.
Generate content in many languages and domains.
Hold a real conversation — it completes text, not dialogue.
Follow user instructions reliably — it predicts likely text, not what the user wants.
Refuse harmful requests — it has no concept of appropriate vs. inappropriate.
Stay on task without elaborate setup.
A pretrained model behaves like a very sophisticated autocomplete. This is why instruction tuning and RLHF are applied after pretraining to make models useful as assistants.
Meta's Llama series, Mistral AI, EleutherAI, and others release pretrained weights publicly.
Anyone can fine-tune for their domain without paying pretraining costs.
Enables research, custom deployments, and on-premise hosting for privacy.
GPT-4 (OpenAI), Claude (Anthropic), Gemini (Google) keep pretrained weights private.
Accessed via API — you send prompts, receive responses; you never see the model weights.
Fine-tuning options may be offered but are limited compared to full open-weight access.
Ask the AI if you need help understanding pretraining, self-supervised learning, what the model learns during pretraining, why it is so expensive, or the difference between pretrained base models and instruction-tuned models.