GPT architecture
A transformer with one hand tied
GPT (Generative Pre-trained Transformer) is the decoder-only branch of the family tree: the transformer stack you assembled last module, minus the encoder, plus one restriction that defines everything: the causal mask.
The rule: when computing attention for position 7, positions 8, 9, 10... are blacked out. Every token may look left as far as it wants and may never look right. Attention scores toward the future are set to negative infinity before the softmax, so they come out as exact zeros.
The reason is the exam from last lesson. Training teaches "predict the next token", and a model that could peek at position 8 while predicting position 8 would learn nothing but copying. The mask keeps the exam honest, at every position, simultaneously.
Audit the mask
Rows are tokens computing their attention; columns are what they may read. Compare GPT's causal mode with BERT's bidirectional mode, and click rows to interrogate them.
| reads → | The | robot | lifted | the | heavy | ball |
|---|---|---|---|---|---|---|
| The | ✕ | ✕ | ✕ | ✕ | ✕ | |
| robot | ✕ | ✕ | ✕ | ✕ | ||
| lifted | ✕ | ✕ | ✕ | |||
| the | ✕ | ✕ | ||||
| heavy | ✕ | |||||
| ball |
Click a row: each row is one position asking what it may look at.
The recipe card, and why it stopped changing
Concrete numbers, because "large" is not a spec:
- GPT-2 (2019): 48 layers at most, 1.5B parameters, 1024-token context, 50k vocabulary.
- GPT-3 (2020): 96 layers, 175B parameters, 96 attention heads per layer, 12,288-dim embeddings, 2048-token context.
- GPT-4-class (2023+): weights undisclosed, context in the hundreds of thousands, same skeleton.
Read that list again and notice what does NOT change: embed, stack of masked-attention + FFN layers, project to vocabulary. From GPT-2 to GPT-4 the architecture is close to boring; the deltas are scale, data, and training refinement. Whatever separates GPT-4 from GPT-2, it is not a cleverer diagram.
One engineering detail you will meet as an AI engineer: during generation, keys and values of already-processed tokens get cached (the KV cache) so each new token only computes ITS OWN query against stored keys instead of reprocessing the whole text. This cache is why long chats eat GPU memory, why "context length" costs what it costs, and (when you reach the security protocol) an attack surface in its own right.
Check your understanding
Why does the causal mask make training so efficient?
What to remember
- ✓GPT = decoder-only transformer: the standard stack plus the causal mask.
- ✓The mask zeroes attention to the future: look left freely, never right.
- ✓Payoff: every position is a parallel training example, and left-to-right generation comes built in.
- ✓GPT-2 → GPT-4 is scale and data on a nearly frozen architecture, not a new diagram.
- ✓The KV cache makes generation affordable and makes long contexts expensive in memory. Remember it; it returns.