The vanishing gradient problem
The whisper game
Last lesson ended on a mystery: depth is clearly the right idea, yet for decades deep networks simply refused to train. The killer wasn't the architecture. It was the learning signal itself dying on the way down.
Recall how backpropagation works: you did it by hand in the Fundamentals protocol: the error signal travels backwards, layer by layer, and at each layer it gets multiplied by that layer's local derivative. The chain rule is literally a chain of multiplications. Now the fatal detail: the sigmoid activation, the classic choice for decades, has a derivative that never exceeds 0.25.
Multiply the error signal by at most 0.25, ten times in a row: 0.25¹⁰ ≈ one millionth. The gradient reaching layer 1 is a whisper of a whisper of a whisper. The last layers hear the error loudly and learn fine, which makes the network LOOK like it's training, but the early layers, the ones that build the edges and textures everything else depends on, receive effectively zero and never move. Deep on paper, shallow in practice.
Watch the gradient dissolve
The error starts at the output with strength 1.0 and travels back, multiplied by the activation’s derivative at every layer. Add layers. Switch activations. Watch what actually arrives at layer 1.
Getting faint. Layer 1 still learns, but orders of magnitude slower than the top layers, hours of training for it, seconds for them.
The exploding twin, and why ReLU is not the whole fix
The same chain of multiplications has an evil twin. If each factor is slightly ABOVE 1, say 1.5, thirty times, the gradient doesn't vanish, it explodes: 1.5³⁰ ≈ 190,000. Weights get catapulted, the loss shoots to NaN, the exact signature you learned to diagnose in the learning rate lesson. The standard duct tape is gradient clipping: cap the gradient's size before applying it.
You saw in the interactive why ReLU changed everything: its derivative is exactly 1 wherever the neuron is active, so the signal passes through untouched, no shrinking per layer. This is THE deep reason ReLU won, beyond being cheap to compute.
But ReLU alone didn't unlock true depth, for reasons you can already name:
- Dead ReLUs: a neuron stuck in the flat zone has derivative 0, a broken link that kills the whole chain through it (the dead neuron problem from the activation functions lesson).
- Real gradients also get multiplied by the WEIGHTS between layers, not just activations, the chain can still shrink or blow up through them.
- Even with ReLU, networks beyond ~20 layers stubbornly refused to improve, something else was broken. That mystery, and its beautiful fix, is the next lesson.
Check your understanding
A 20-layer sigmoid network trains for days: the last layers improve, the first layers never change. What is happening?
What to remember
- ✓Backprop is a chain of multiplications, the gradient crosses every layer on the way back, shrinking or growing at each step.
- ✓Sigmoid’s derivative ≤ 0.25 makes deep chains vanish: early layers receive ~zero signal and silently never learn.
- ✓The symptom is sneaky: the network seems to train (late layers move) while its foundation stays frozen.
- ✓Factors above 1 give the exploding twin, NaN loss, tamed by gradient clipping.
- ✓ReLU passes gradient ×1 where active and made depth possible, but not sufficient: dead units, weight factors, and a wall around ~20 layers remained. Enter residual connections.