Gradient descent
Downhill in the fog
Picture this: every possible setting of ALL the network's weights is one point on a landscape, and the altitude at that point is the loss you'd get with those weights. Training is a search for a low valley. For a real network the landscape has billions of dimensions, impossible to picture, but the math doesn't care, and the 2D intuition carries over surprisingly well.
Here's the catch: you can't see the map. Checking the altitude of even a tiny fraction of all weight settings is hopeless, you felt that in the nudge game two lessons ago. All you can do is feel the slope under your feet. That slope is the gradient, and backpropagation hands it to you for the price of one backward pass.
Gradient descent is then the simplest walking strategy imaginable:
- Compute the gradient: which direction is uphill, and how steep it is.
- Step in the exact opposite direction: downhill.
- Make the step proportional to the steepness, scaled by a factor called the (next lesson).
- Repeat from wherever you land.
No map, no plan, no memory, just "feel the ground, step downhill, repeat". It sounds too dumb to work. Watch it work.
Walk down the loss curve
One weight only, so the landscape is a simple curve: the weight's value runs along the horizontal, the loss sits on the vertical. Each click takes one gradient step with a fixed learning rate. Watch the size of the hops as the ground flattens.
Steep ground: the next step will be big.
Valleys, traps, and why noise helps
The obvious worry: what if you roll into a small dip that isn't the deepest valley, a , and get stuck, feeling flat ground in every direction while a much better valley sits one hill away?
In 2D that intuition is deadly right. In billions of dimensions it mostly isn't, for a strange reason: to be truly trapped, the ground must curve upward in every one of those billions of directions at once. Far more common are saddle points, up in some directions, down in others, and a saddle always offers an escape route. High dimensionality, weirdly, works in your favor.
The second helper is noise. Real training computes the gradient on a small batch, not the full dataset, that's the "stochastic" in stochastic gradient descent (SGD). Each batch gives a slightly different, slightly wrong slope, so the ball jitters as it rolls. That jitter is useful: it shakes the ball out of shallow dips that would trap a perfectly smooth roller.
And an honest note to close: nobody finds THE global minimum of a billion-dimensional landscape, and nobody needs to. Training finds A valley low enough that the model works, and in practice that turns out to be plenty.
Check your understanding
The gradient at the current weights tells you…
What to remember
- ✓The loss landscape: every full setting of the weights is a point, the loss is its altitude, and training is a descent.
- ✓You can never see the map, only feel the local slope, the gradient, which backprop computes.
- ✓Gradient descent steps opposite the gradient, proportional to its steepness: it brakes automatically as the ground flattens near a minimum.
- ✓Real training is stochastic (per-batch) gradient descent: the noise shakes the ball out of shallow dips, and in high dimensions true traps are rarer than saddle points.
- ✓You don't need THE lowest valley, just a low-enough one, and that's what training actually finds.