Autoregressive models
One objective to rule the field
Strip away every product feature, and a GPT-class model trains on exactly one exam question, repeated trillions of times: given everything so far, what token comes next? No grammar module, no fact database, no dialogue engine. Next token, always and only.
"Autoregressive" names the loop that turns that exam into generation: predict a token, append it to the input, predict again on the longer input. The model literally feeds on its own output. Every paragraph an LLM has ever written was produced one token at a time through this loop, each step re-reading the whole growing sequence.
People hear this and feel scammed: surely something so mindless cannot write poetry or code. The resolution of that tension is the entire story of this module, and it starts with taking the loop seriously instead of dismissing it.
Turn the crank
Generate a sentence the way the model does: run a prediction over everything so far, commit the top token, feed it back, repeat.
Read distributions like the model does
The model never outputs a word; it outputs a probability for every token in the vocabulary. Guess the top pick, then study the shape of each distribution.
The opposite of hot is▍
For dinner I cooked▍
Water boils at 100 degrees▍
The shape of the distribution is the message
Those three shapes are worth internalizing, because the loop treats them identically and that has consequences.
When the distribution is a spike, the model "knows"; when it is a plateau, the model is listing possibilities. But generation MUST pick a token either way; there is no "pass" option in the loop. Force a pick on a plateau about the safest ingredient for a recipe and no harm done. Force a pick on a plateau about a citation, a dosage, or a legal precedent, and the model outputs a confident-sounding invention, because fluent grammar rides on the spikes even while facts sit on a plateau.
You will meet this again twice: as "hallucination" when we discuss trust, and as the reason temperature and sampling settings exist. For now, one upgrade to your mental model: an LLM is not a fact retriever with occasional bugs. It is a distribution machine, and every output is a draw.
Check your understanding
What does "autoregressive" mean in practice for a language model?
What to remember
- ✓One training objective: predict the next token. Everything an LLM does is this exam, industrialized.
- ✓Generation = the loop: predict, commit, append, repeat. Output feeds back as input.
- ✓The raw output is always a full probability distribution; a "response" is a sequence of draws from it.
- ✓Spike = learned certainty, plateau = many valid futures. The loop forces a pick either way, and forced picks on factual plateaus are where inventions come from.
- ✓Each committed token reshapes every later prediction: generation is path-dependent.