Batch normalization
Training on quicksand
Skip connections fixed the gradient's journey. But deep networks had a second, sneakier disease, and this one strikes on the way FORWARD.
Think about what layer 30 is actually doing during training. It's learning from the activations that layer 29 hands it. But layer 29 is ALSO training, its output distribution changes with every single update. Layer 30 is a student preparing for an exam whose subject changes every week: whatever it adapted to yesterday is stale today. Every layer above the first is learning on quicksand.
The concrete damage: as weights shift, each layer's activations drift, the whole batch sliding right, spreading out, collapsing. Drift far enough and the values land in the activation function's flat saturation zones, where the derivative is zero, and you know from the vanishing gradient lesson what a zero in the chain does. The classic survival strategy was a tiny learning rate: drift slowly, learn slowly. Deep training crawled.
Pull the batch back into the sweet spot
Each dot is one neuron’s activation for the current batch of data. The teal band is the sweet spot where the activation function still has slope; the red zones are saturation, flat ground, zero gradient. Feed new batches, watch the drift, then normalize.
Normalize: then let the network overrule you
Batch normalization (2015) is the idea you just performed, made into a layer: take the batch, subtract its mean, divide by its standard deviation. Every layer now receives inputs centered on the sweet spot, every step, no matter how much the layer below just changed.
But there's a subtle danger: what if the network NEEDED activations away from zero, a mostly-negative regime, a wide spread? Forcing everything to mean 0, spread 1 would amputate expressiveness. So batch norm adds two learnable parameters per feature: γ (scale) and β (shift), applied after normalizing. The network can re-widen, re-center, even undo the whole normalization if that's what the loss prefers. You stabilize by default, the network overrules by exception, and γ and β are trained by backprop like any other weight.
What it bought in practice:
- Much higher learning rates without divergence, the drift that made them lethal is gone. Training that took weeks took days.
- Far less sensitivity to weight initialization, bad starting points get normalized away.
- A mild regularization bonus: each sample sees statistics that depend on its random batch-mates, a little noise, dropout-flavored, for free.
One caveat worth owning: batch norm needs the batch. At inference there is no batch (one user, one image), so it switches to running averages saved from training. And its famous cousin: LayerNorm, which normalizes across features instead of across the batch, is the one transformers use. You will meet it in the NLP protocol.
Check your understanding
After normalizing to mean 0 and spread 1, batch norm applies learnable γ (scale) and β (shift). Why add them?
What to remember
- ✓Deep training’s hidden treadmill: every layer learns from a layer that is itself changing, inputs drift into saturation, forcing crawling learning rates.
- ✓Batch norm: at each layer, each step, subtract the batch mean, divide by the batch spread. Inputs land in the gradient-rich sweet spot.
- ✓γ and β make it a suggestion, not a straitjacket: the network can re-scale, re-shift, or undo it.
- ✓Bought: big learning rates, fast convergence, initialization tolerance, a pinch of free regularization. Cost: batch dependence, inference uses stored running averages.
- ✓LayerNorm, its across-features cousin, is what transformers use. Skip connections + normalization: two of the three pillars of trainable depth. The third: keeping millions of parameters honest, is next.