Full transformer architecture
Assembly day
You now own every part on the bench: tokens, embeddings, positional stamps, multi-head attention. Time to bolt them into the machine from "Attention Is All You Need" (2017) and see the whole data path, input to output.
The path: text → token IDs → embeddings + position vectors → a stack of identical layers → one final projection back to vocabulary size → a probability for every token in the vocabulary. Each layer in the stack contains exactly two sublayers:
- Multi-head attention: tokens exchange information. The social phase.
- Feed-forward network (FFN): a two-layer MLP applied to each token separately, no cross-talk. The private-study phase: digest what you just gathered.
The FFN is the part everyone underexplains. It holds roughly two thirds of the model's parameters, and interpretability work keeps finding that much of a model's stored FACTUAL knowledge lives there, written into those per-token MLPs. Attention moves information around; the FFN is where a lot of it is kept.
Rebuild the data path
The five stages of a transformer forward pass, shuffled. Click them in the order the data actually flows.
The glue you already know, and the family tree
Around every sublayer sit two pieces of glue from the deep learning protocol: a residual connection (the input skips around the sublayer and gets added back) and layer normalization (keep the numbers in a healthy range). Without residuals, a 96-layer stack would be untrainable; you watched exactly why in the vanishing-gradient lab. A useful mental image from interpretability research: one wide "residual stream" flowing upward per token, with each attention and FFN block reading from it and writing small updates back into it.
The 2017 original was actually TWO stacks, built for translation: an encoder that reads the source sentence bidirectionally, and a decoder that writes the translation. History then simplified it:
- Encoder-only (BERT, 2018): keep the reading half. Great at understanding, cannot generate.
- Decoder-only (GPT, 2018): keep the writing half. The next module lives here.
- Encoder-decoder (T5, translation systems): the full original, still the right tool for some input→output tasks.
Almost everything you interact with today (GPT, Claude, Llama, Gemini) is decoder-only. The architecture war ended quietly, and the writing half won.
Which part does this job?
Five responsibilities, four components. Assign each job to the part that owns it.
Lets "it" pull information from "ball", eight positions away.
Holds roughly two thirds of the parameters and much of the stored factual knowledge.
The only reason "man bites dog" and "dog bites man" are different inputs.
Lets the gradient survive a 96-layer round trip during training.
Runs identically on each token with zero communication between them.
What to remember
- ✓One layer = attention (tokens talk) + FFN (tokens study alone), each wrapped in residual + normalization.
- ✓Stack that layer 12-100+ times, then project to vocabulary probabilities. That is the whole machine.
- ✓The FFN is the quiet giant: most parameters, much of the stored knowledge.
- ✓Residuals and layer norm are the same trainability medicine you met in deep networks, doing the same job here.
- ✓Encoder-only (BERT) understands, decoder-only (GPT) generates, and decoder-only now dominates the field.