Backpropagation intro
The blame game
The forward pass ends with a prediction. Compare it with the true answer and you get one number: the error, how wrong the network was on this example. So far, so good. But that single number hides the real problem of learning.
Say the network predicted 8 and the truth was 16. The error came out of a chain of thousands (or billions) of weights, all multiplying and adding together. Which of them caused the miss? Should weight #3,481,229 go up or down? By a lot or a little? The error doesn't say: it's one number describing the failure of the whole team, with no names attached.
This is called the credit assignment problem, and it's THE problem of training neural networks. Before you see the algorithm that solves it, it's worth feeling the problem in your own hands, on a network small enough to fix manually.
Be the training algorithm
This tiny neuron computes prediction = w₁·x₁ + w₂·x₂ + bias. The inputs are fixed: x₁ = 3 and x₂ = 2. The target output is 16. Nudge the three knobs until the error hits zero, and pay attention to what information you use to decide each nudge.
Error: 8
One backward sweep instead of billions of experiments
What you just did has a name: trial and error, one weight at a time. Its cost is brutal: every experiment needs its own forward pass, so a network with N weights needs on the order of N passes just to figure out one round of nudges. With billions of weights and millions of training examples, that plan is dead on arrival.
Backpropagation's insight: don't experiment. Compute. The network is just math, sums, multiplications, activations, and calculus has a tool, the chain rule, that can trace exactly how much each weight contributed to the final error. Not approximately: exactly, for every weight at once. The recipe:
- Run the forward pass and measure the error at the output.
- Walk backwards, layer by layer: each layer computes how much each of its weights, and each of its inputs, contributed to the error, using the chain rule.
- By the time you reach the input layer, every single weight holds its own instruction: "move me this way, by this much, to shrink the error." That signed instruction is the weight's gradient.
The cost? Roughly one extra backward pass, about as expensive as the forward pass itself. That's the miracle: the blame assignment for all 175 billion weights, for the price of two passes instead of billions.
The chain rule itself is centuries-old math. The breakthrough was a 1986 paper by Rumelhart, Hinton and Williams showing it could train multi-layer networks in practice. Remember the XOR wall from the perceptron lesson? Minsky's real objection wasn't that multi-layer networks couldn't represent XOR, it's that nobody knew how to train them. Backpropagation was the missing piece, and it's still the engine inside every network trained today.
Which way should the weight move?
A gradient is, at its core, a signed answer to "which way?", and for one neuron you can reason it out by hand. Remember: this weight's contribution to the prediction is weight × input. For each situation, pick the right nudge.
The prediction came out too LOW. This weight's input was a positive number.
The prediction came out too HIGH. This weight's input was a positive number.
The prediction came out too LOW. This weight's input was a negative number.
The prediction came out too HIGH. This weight's input was exactly 0.
The prediction came out too HIGH. This weight's input was a negative number.
Check your understanding
Nudging weights one at a time worked fine for 3 knobs. Why do real networks need backpropagation instead?
What to remember
- ✓After a forward pass the network knows how wrong it was, one number, but not who's to blame or which way each weight should move. That's the credit assignment problem.
- ✓Fixing weights by trial and error means one experiment per knob: fine for 3, impossible for billions.
- ✓Backpropagation sends the error backwards through the layers using the chain rule, computing every weight's gradient, its direction and share of the blame, exactly.
- ✓The cost is roughly one extra backward pass: all the blame assignments for the price of two passes total.
- ✓Popularized in 1986 by Rumelhart, Hinton and Williams, backprop is what finally made multi-layer networks trainable, and broke the XOR wall for good.