Dropout & weight decay
Millions of parameters, and an old enemy returns
Residual connections and batch norm made depth trainable. Congratulations: you can now fit a 50-million-parameter network. Which revives the oldest enemy in this course, overfitting. You met it in Fundamentals with a wiggly curve chasing noisy points; a deep network is that curve with fifty million knobs.
How bad is it? A famous experiment (Zhang et al., 2017) took a standard image network and trained it on CIFAR-10 with the labels replaced by RANDOM ones, cat photos labeled "truck", planes labeled "frog". The network reached ~100% training accuracy anyway. Pure memorization of pure nonsense, executed flawlessly. THAT is the memorization power idling inside every deep net you train.
Deep networks also overfit in a specifically social way: co-adaptation. Neuron 47 learns to be useful only in combination with neuron 112, a private pact that happens to cancel errors on the training set. Fragile, brittle knowledge that collapses on new data. The cure is delightfully brutal: during training, randomly silence neurons. Every step, each neuron has (say) a 50% chance of simply not existing. No neuron can build its function on a colleague that might be gone, every neuron must learn features that stand on their own. That's dropout.
Watch the network lose neurons, on purpose
A small network in training. Every step, dropout silences a random subset of the hidden layer. Step through and watch: a different sub-network trains each time, and no neuron can count on any other.
Dropout ON. Run a training step to see who gets silenced.
Weight decay, and the rest of the toolkit
The second workhorse attacks a different symptom. Remember, from the complexity slider in Fundamentals, what an overfit curve looks like: wild swings. Wild swings need huge coefficients, big weights are how a network buys wiggliness. Weight decay (L2 regularization) makes it pay for them: add λ·Σw² to the loss, so every large weight now costs loss points. Gradient descent, forced to balance fitting against the bill, keeps weights small, and small weights mean smooth functions where tiny input changes cannot cause output explosions.
One subtlety completes dropout: training silences neurons, but at test time ALL of them are active (scaled to keep magnitudes consistent). Randomness while learning, full ensemble while answering.
You now hold the complete standard toolkit, worth seeing in one place:
- Dropout: breaks co-adaptation; forces self-sufficient features.
- Weight decay: taxes large weights; buys smoothness.
- Early stopping: watch validation loss, stop at its minimum (the epoch trainer from Fundamentals).
- Data augmentation: flip, crop, recolor, rotate: multiply your dataset with label-preserving edits.
- More real data: the only cure that attacks the disease instead of the symptoms. Everything else constrains the model; data actually informs it.
Prescribe the right medicine
Five training rooms, five overfitting situations. Pick the tool each one is begging for.
Image classifier overfitting; flipping, cropping and recoloring your photos would cost nothing and preserve every label.
Validation loss bottomed out at epoch 30 and is now climbing while training loss keeps falling. You want the epoch-30 model.
You suspect a clique of neurons has formed, each useless alone, jointly canceling errors on the training set only.
The learned function swings wildly, enormous positive and negative weights amplify tiny input changes into huge output changes.
You want the loss function itself to charge a fee for every large weight, automatically, at every step.
Check your understanding
Dropout randomly silences neurons during training. What happens at test time?
What to remember
- ✓Deep nets can memorize ANYTHING, even pure random labels, so regularization is not optional at 50M parameters.
- ✓Dropout: randomly silence neurons each training step. Co-adaptation becomes impossible; features must stand alone. Test time: everyone plays.
- ✓Weight decay: loss + λ·Σw². Large weights buy wiggliness; make wiggliness expensive and functions come out smooth.
- ✓Full toolkit: dropout, weight decay, early stopping, augmentation, more data, and more (real or augmented) data is the only one that cures instead of constrains.
- ✓Module complete: depth is worth it (hierarchy), what broke it (vanishing gradients), and the three pillars that fixed it (skips, batch norm, regularization). Next module: the architecture that taught networks to SEE.