Backprop through time
Unroll the loop
How do you backpropagate through a loop? You straighten it. Take the RNN processing a 20-word sentence and UNROLL it: draw the cell once per time step, each copy passing its hidden state to the next. What appears on paper is something you already know intimately, a 20-layer feedforward network, with one twist: every "layer" is the SAME cell, sharing the same weights.
Training is then plain backpropagation applied to the unrolled graph, the error flows backwards from the last step toward the first, exactly like flowing down layers. The technique is called backpropagation through time (BPTT), and the name is honest: time has become depth.
Say that sentence again slowly, because it should trigger an alarm from the deep networks module: the gradient must cross one multiplication per time step. A 100-word sentence is a 100-layer network. You know EXACTLY what happens to a signal multiplied 100 times by factors below 1, you watched it dissolve once already. It is about to happen again, on a new axis.
The vanishing gradient: through time
The network mis-predicts at the last word, and the correction must travel back to teach word 1. Each step back multiplies the gradient by ≈ 0.6 (recurrent weight × activation slope). Stretch the distance and watch how much lesson actually arrives.
A dozen steps and the lesson arrives as a whisper: learning long-range patterns is now hundreds of times slower than short ones.
What this breaks, and the duct tape
Put the failure into a sentence: "The cat, which had been hiding under the porch since the storm scared it, WAS hungry." To conjugate WAS, the network must remember CAT across 12 words. The gradient connecting them crosses 12 multiplications, by your interactive, a whisper. Vanilla RNNs reliably learn short-range grammar and reliably fail long-range dependencies: subject–verb agreement across clauses, a variable defined 80 tokens ago, the attacker who logged in 500 log-lines before the breach.
The exploding twin also returns, factors above 1 compound into NaN across long sequences, and the fix is the same gradient clipping from the deep networks module. Symmetry preserved.
The practical duct tape everyone uses: truncated BPTT, only backpropagate k steps into the past (say 35) and stop. It caps the cost and the explosions, but read the fine print: dependencies longer than the truncation window get literally zero learning signal. The window is a confession: "we do not expect to learn anything longer than this."
Duct tape is not a cure. The real fix changed the CELL itself, giving memory a protected pathway that multiplication cannot erode. That is the next lesson, and it carried the entire field for twenty years.
Check your understanding
A vanilla RNN keeps failing to learn subject–verb agreement across 15 intervening words. Why?
What to remember
- ✓Training an RNN = unrolling it: a 100-step sequence becomes a 100-layer network whose layers all share one cell.
- ✓BPTT is ordinary backprop on the unrolled graph, so the gradient crosses one multiplication per time step.
- ✓The vanishing gradient returns on the time axis: long-range dependencies receive ~zero teaching signal. Exploding twin → gradient clipping, as before.
- ✓Truncated BPTT caps cost by refusing to look further than k steps, a confession, not a cure.
- ✓The cure changes the cell: a memory pathway that multiplication cannot erode. Enter the LSTM.