From RNN to attention
The keyhole
Machine translation, pre-2015, worked like this: an encoder RNN reads the whole Italian sentence and compresses it into its final hidden state, ONE fixed-size vector. A decoder RNN takes that vector and writes English out of it. Elegant. And structurally absurd: the entire sentence, every nuance, must pass through a single vector. A whole book pushed through a keyhole.
The failure signature was measurable: translation quality degraded as sentences grew. By word 40, whatever word 3 contributed had been blended into oblivion, you watched that exact fade in the memory-chain, and gates only soften it. The keyhole does not care how clever the compression is; it is still a keyhole.
Then came the question that broke the dam (Bahdanau, 2014): why does the decoder only get the LAST state? The encoder produced a perfectly good hidden state at EVERY word, a whole shelf of summaries, and we throw away all but one. What if, for each word it writes, the decoder could look back at ALL of them and choose, with learned weights, where to look? That choosing is attention.
Look inside a translation
A model translates "Il gatto nero dorme" into "The black cat sleeps". Click each English word to see WHERE the model actually looked while producing it, the attention weights over the Italian input.
Click an English word to see where the model looked.
Drop the recurrence entirely
Attention began as a patch for RNN translation. Then, in 2017, a paper asked the question its own title answers: if attention gives every position direct access to every other position… what is the recurrence still FOR? "Attention Is All You Need" deleted it, no hidden state carried step by step, no conveyor belt, just attention layers. The architecture was named the Transformer.
Look what falls out when recurrence goes:
- The keyhole is gone: no single vector bottleneck; every position reads every other directly.
- Vanishing-through-time is gone: no per-step multiplication chain for the gradient to die in. Word 2 to word 402: one direct connection.
- The sequence trains in PARALLEL, no waiting for step t−1. GPUs digest whole sequences at once, which is precisely what made training on internet-scale text affordable.
That architecture is the ancestor of GPT, Claude, and essentially every model making headlines today. The next protocol: NLP & Transformers: opens it up bolt by bolt: how attention is computed, multi-head attention, positional encoding. Walk in knowing you have already used its heart today.
RNN or attention?
Six properties. Attribute each to the mechanism it describes.
Must finish processing word 12 before it can even begin word 13.
Connects word 2 and word 402 as directly as two neighbors.
Its entire knowledge of the past lives in one fixed-size vector.
Processes all positions of the sequence at once on a GPU.
To learn a long-range dependency, its gradient must survive one multiplication per intervening step.
For each output, computes a learned weighted look over ALL inputs.
Check your understanding
What is the core advantage of attention over a recurrent hidden state?
What to remember
- ✓Encoder–decoder RNNs push whole sentences through a keyhole: one fixed vector. Quality measurably decays with length.
- ✓Attention: for each output, a learned weighted look over ALL encoder states. Crossed word orders and long distances become trivial.
- ✓The 2017 move: drop recurrence entirely, keep attention, the Transformer. No bottleneck, no vanishing-through-time, full parallelism.
- ✓That architecture is the direct ancestor of GPT and Claude, and you just used its core mechanism by hand.
- ✓Module complete: sequences, RNNs, BPTT, gates, attention. One module left in the protocol, networks that CREATE.