Residual connections
The 56-layer embarrassment
In 2015, researchers at Microsoft ran an experiment that should have been boring: train a 20-layer network and a 56-layer network on the same data. Deeper sees more, deeper should win. The 56-layer network was WORSE.
Your first instinct is the right reflex, overfitting, too much capacity. Except it wasn't: the 56-layer network was worse on the TRAINING data too. An overfit model crushes the training set and fails on new data; this one couldn't even memorize. That is not a generalization problem, it is an optimization problem: gradient descent physically failed to find good weights that were provably there.
Provably: because here is the paradox that made everyone stop and stare: the 56-layer network could imitate the 20-layer one perfectly. Use the first 20 layers, then have the remaining 36 layers simply copy their input to their output. Identity. Do nothing. If the optimizer could find "do nothing", deeper would never be worse. It could not find "do nothing". Passing a signal unchanged through a stack of weights and activations is, for gradient descent, a surprisingly contorted function to learn.
Open the highway
Worst case chain: every block multiplies the gradient by 0.25. Stack blocks until the signal is dead, then flip on skip connections and look at layer 1 again.
Dead signal. This is exactly where deep learning was stuck before 2015. Now switch the skip connections ON.
Learn the change, not the whole recipe
The fix: Kaiming He and colleagues, the ResNet paper, is two ASCII characters: y = F(x) + x. Each block no longer produces the whole output; it produces a CORRECTION (a residual) that gets added to its own input, which flows around it on a skip connection.
That one plus sign solves both halves of the problem at once:
- The identity becomes the default instead of a contortion: to do nothing, the block just learns F = 0, push the weights toward zero, the easiest thing gradient descent does. Extra layers can no longer hurt.
- The gradient gets a highway: differentiating F(x) + x gives F′(x) + 1, and that "+1" means the backward signal always has an unobstructed path. No more chains of shrinking multiplications, the whisper game from last lesson is over.
The result made history: ResNet-152: 152 layers, trained cleanly and won ImageNet 2015. And the idea escaped vision entirely: skip connections sit inside essentially every modern architecture, including the transformers behind ChatGPT. When you reach the NLP protocol you will find them there, waiting, doing exactly this job.
Check your understanding
Why does the skip connection y = F(x) + x let very deep networks finally train?
What to remember
- ✓The 2015 mystery: 56 layers lost to 20, on TRAINING data. Not overfitting: optimization failure.
- ✓The paradox that exposed it: deeper could just copy the shallower net, but gradient descent cannot easily learn "copy the input".
- ✓Residual block: y = F(x) + x, learn only the correction, let the input flow around on a skip connection.
- ✓The +x is a gradient highway (F′(x) + 1: always a clear path back) and makes identity the default (F = 0 is easy). Depth stops being dangerous.
- ✓ResNet-152 won ImageNet 2015; skip connections now live in nearly every serious architecture, transformers included.