LSTM and GRU
Give the memory a vault, and learned keys
The vanilla RNN's sin was structural: EVERY step, the entire memory gets shoved through the same blending multiplication, whether that helps or not. Old information doesn't die because it should, it dies because the architecture never offered a way to leave it alone.
The Long Short-Term Memory cell (Hochreiter & Schmidhuber, 1997, two decades before it conquered your phone) fixes exactly that. It splits the state in two: a working memory like before, plus a protected CELL STATE, a conveyor belt that information can ride across time steps untouched. No mandatory blending. Riding the belt is the default, not a privilege to re-earn each step.
Access to the belt is controlled by three gates, tiny learned valves, each outputting a number between 0 (shut) and 1 (open):
- Forget gate: how much of the current cell state survives this step?
- Input gate: how much of the incoming information gets written onto the belt?
- Output gate: how much of the belt is revealed into the working memory right now?
The crucial word is LEARNED: each gate is itself a small neural function of the current input and state, trained by backprop. The network learns its own memory policy, when a token deserves the vault, and when it is noise to ignore. Time to feel why that beats a fixed rule: play the gates yourself.
You are the gates
Mission: the FIRST number of the sequence must still be in the memory cell at the end, everything after it is noise trying to get in. Before each step, set the two gates, then let the token through. A vanilla RNN (fixed ½-blend rule, no choices) runs alongside for comparison.
GRU, and what the gated decade built
The LSTM's famous sibling, the GRU (2014), streamlines the design: two gates instead of three, working memory and belt merged into one state. Fewer parameters, faster training, and on many tasks it matches the LSTM, a classic engineering trade-off with no universal winner. Try the cheap one first is a respectable default.
Calibrate what gates actually bought the world. From roughly 2013 to 2018, gated recurrent networks WERE applied AI: Google Translate's 2016 leap in quality, Siri and Alexa's speech recognition, autocomplete, music generation. When the media said "AI reads text", the machinery underneath was an LSTM riding its conveyor belt.
And yet two limits survived every gate. The network still processes strictly step-by-step, no parallelism, remember?, and everything it knows about the past must still squeeze through fixed-size states. Gates decide WHAT to keep; they cannot keep everything, and they cannot make time run in parallel. Both walls fall in the next lesson, and what knocks them down is already the heart of the next protocol.
Check your understanding
During training, an LSTM's forget gate outputs 0.97 for a particular feature. What is the cell doing?
What to remember
- ✓Vanilla RNNs forget by force: mandatory blending every step. LSTMs make preservation the default, a cell-state conveyor belt information rides untouched.
- ✓Three learned gates: forget, input, output, are valves between 0 and 1, trained by backprop into a memory POLICY, not a fixed rule.
- ✓The belt also protects the backward pass: gradients get a path that the per-step multiplication chain cannot erode.
- ✓GRU: the 2-gate streamlined sibling, cheaper, often equal. An engineering trade-off, not a ranking.
- ✓Gated RNNs powered the 2013–2018 AI wave (Translate, Siri), but stayed sequential, and still compress the past into fixed-size state. Next: the idea that dropped both constraints.