The training loop
Four steps, repeated until it works
You now hold every piece this module built: the neuron's weighted sum, activation functions, the forward pass, and backpropagation. Training a neural network is nothing more than snapping them together into a loop:
- 1. Forward pass: push a batch of examples through the network and get predictions.
- 2. Measure the loss: compare predictions with the true answers; one number says how wrong.
- 3. Backward pass: backpropagation assigns every weight its share of the blame.
- 4. Update the weights: nudge each weight a small step in the direction that shrinks the loss.
Two vocabulary words make the loop precise. A batch is the small group of examples processed in one iteration, not one example (too noisy, and it wastes the GPU's appetite for parallel matrix math) and not the whole dataset (too slow per update). An epoch is one full pass over the entire training set: once every example has been seen once, that's one epoch done.
That really is the complete recipe. GPT-class models and a 20-line toy script run the exact same four steps, the difference is only how many weights, how much data, and how many times the loop spins.
Assemble the loop
Click the four steps in the order they happen inside one training iteration.
Train it yourself, one epoch at a time
Each click runs one full epoch of the loop above. Watch both curves: the loss on the training data, and the loss on validation data the model never trains on. Decide when YOU would stop.
Epoch 0: the weights are still random. Run the first epoch.
When do you stop?
The naive answer: "when the training loss is low", is a trap you just watched happen. Training loss almost never turns around: given enough epochs, a big network will keep squeezing it down by memorizing individual examples. The number that tells the truth is the validation loss, because it's measured on data the loop never touches.
So the practical stopping rule used everywhere is early stopping: keep training while validation loss improves, keep a saved copy of the best-so-far model, and stop when validation has been getting worse for a while. You ship the saved copy, not the final one.
And with that, the module is complete. You have the whole machine: neurons doing weighted sums, activations bending straight lines, the forward pass producing predictions, backprop assigning blame, and the loop that repeats it until the loss gives in. The next module: Training and Evaluation: hands you the instrument panel: what exactly the loss measures, how gradient descent really moves, how big the steps should be, and how to judge a trained model honestly.
Check your understanding
What exactly is an "epoch"?
What to remember
- ✓The training loop is four steps, forward pass, measure the loss, backward pass, update the weights, repeated thousands or millions of times.
- ✓A batch is the group of examples processed per iteration; an epoch is one full pass over the whole training set.
- ✓Improvement is front-loaded: early epochs buy a lot, later ones less and less.
- ✓Watch validation loss, not training loss. When validation turns upward the network is memorizing, early stopping keeps the model from its best epoch, not its last.
- ✓Module complete: you can now trace the entire life of a neural network, from a single weighted sum to a full training run.