Pretraining: learning from the internet
The biggest reading assignment in history
Take the GPT machine from last module. Now feed it: web crawls, books, Wikipedia, code repositories, forums, papers. Trillions of tokens; for frontier models, more than ten trillion, hundreds of times what a human could read in a dozen lifetimes. On every single token, the same exam: predict the next one, measure the miss with cross-entropy, nudge the weights. Months of this, on thousands of GPUs, burning eight or nine figures of compute.
This phase is pretraining, and it is where essentially ALL of the model's knowledge and skill enters. Everything after it, the steps that make the model chat, cost a rounding error in comparison.
Skeptical that next-token prediction can force real learning? Follow the exam's logic. To predict the next token of "the verdict was overturned because the witness", grammar helps a little, but you do better knowing courtroom procedure. To complete Python, you do better having internalized how variables scope. Prediction on diverse text rewards every regularity, from spelling up through logic, so the model absorbs them all, because they all lower the loss.
One exam, every genre
Pretraining data is wildly heterogeneous, and the same next-token exam runs on all of it. Play three rounds from three corners of the corpus.
def fibonacci(n): if n <= 1: return▍
Breaking: the central bank has decided to raise interest▍
Once upon a time, there was a▍
What comes out: a base model
The artifact this produces is called a base model, and meeting one is disorienting. Ask it "What is the capital of France?" and a perfectly healthy base model may reply: "What is the capital of Germany? What is the capital of Spain?", because on the internet, one quiz question is usually followed by more quiz questions. It completed your text. That IS the skill it was trained on.
A base model is an engine without a steering wheel: colossal knowledge, zero interface. Three facts about this stage worth keeping:
- It is one shot: a frontier pretraining run cannot be cheaply redone. Data mistakes get baked in for the model's lifetime.
- Data curation is a weapon: deduplication, quality filtering, and mixture ratios (how much code? how much of which languages?) shape the model as much as architecture. Recall the tokenizer's language tax; the corpus mixture is the same politics, deeper.
- The cutoff is born here: the model's knowledge freezes at the crawl date. Everything after is darkness until someone glues on retrieval.
The remaining lessons in this module are the steering wheel: how a completion engine gets shaped into the assistant you actually use.
Check your understanding
You ask a base model "How do I reset my password?" and it responds with four more customer-support questions. What happened?
What to remember
- ✓Pretraining: trillions of tokens, months of GPUs, one exam (next token) on the whole internet.
- ✓Prediction rewards every regularity in text, so grammar, facts, code, and reasoning patterns all get absorbed into the weights.
- ✓The result is a base model: an autocomplete engine of enormous knowledge and no conversational interface.
- ✓Data curation and mixture decide as much as architecture, and errors here are permanent: the run is one shot.
- ✓Nearly all capability enters at this stage. Everything that follows is shaping, not teaching.