Learning rate
The most important dial you'll ever set
left one thing unspecified. The gradient gives a direction and a steepness, but how far do you actually move in one step? That scaling factor is the learning rate, and it comes with a new word worth owning: it's a hyperparameter. Parameters (the weights) are what the model learns by itself; hyperparameters are the settings YOU choose before training, learning rate, number of layers, batch size. Of all of them, the learning rate is hyperparameter number one: get it wrong and nothing else you did matters.
The trade-off is a pure Goldilocks problem:
- Too small: every step is safe but microscopic; training crawls, burns compute for hours, and can effectively stall on flat ground.
- Too large: you leap clean over the valley floor and land on the far slope, sometimes higher than you started; the loss oscillates or outright explodes.
- Right: bold progress while the ground is steep, and it still settles cleanly into the minimum.
Why does too-large fail so violently? Because the gradient is local, it describes the ground where you stand, and says nothing about the ground three valleys away. A huge step is an act of blind faith in a slope that was only measured under your feet. Same fog as last lesson: take steps small enough that the ground can't change too much mid-air.
The learning-rate lab
Same valley as last lesson, but now you pick the learning rate before stepping. Try all three and compare how the runs end.
Diagnose the training run
In a real run there's no ball to watch, you watch the loss curve and read the symptoms. Diagnose each of these runs.
The loss climbs after every update and within seconds becomes NaN (not a number).
After hours of training the loss is decreasing, but has barely moved from where it started.
The loss drops fast at first, then keeps bouncing up and down around the same value without ever settling.
The loss falls quickly in the early epochs, slows down, and levels off at a low value.
The loss curve is perfectly smooth and steadily falling, but the model is learning in slow motion, and the budget meter is not.
Real runs change the dial mid-flight
One fixed rate for a whole run leaves value on the table, because the ideal step size changes as you descend: far from the valley you want bold strides, near the floor you want fine adjustments. So real training uses learning-rate schedules, start higher, decay as training progresses (step drops, smooth cosine curves, many shapes, same idea).
Two more names you'll meet constantly, planted here so they're familiar: warmup means spending the first few hundred steps ramping the rate UP from near zero, which keeps big models from blowing up while their weights are still random chaos. And optimizers like Adam give every single weight its own adaptive step size instead of one global dial. How they do it is a story for the Deep Learning protocol, for now it's enough to know the dial doesn't have to be one number, or a constant.
Check your understanding
A "hyperparameter" is…
What to remember
- ✓The learning rate scales every gradient step, it is a hyperparameter (chosen by you), not a parameter (learned by the model).
- ✓Too small: safe, smooth, and unaffordably slow. Too large: overshoot, oscillation, or a NaN explosion. Right: fast early, settles cleanly.
- ✓Oversized steps fail because the gradient is local, a big leap trusts the slope far beyond where it was measured.
- ✓You can diagnose the dial from the loss curve alone: NaN or bouncing means too high, glacial smoothness means too low.
- ✓Real runs move the dial mid-flight: schedules decay it, warmup ramps it up at the start, and optimizers like Adam adapt it per weight.