Loss functions
The number the whole game is played against
Every turn of the training loop starts from one number: the loss. Training is nothing but the attempt to make that number smaller. The model doesn't "want to be smart" or "try to understand", it descends its loss, and that's the entire motivation it has. Which means something worth engraving: the loss function is the definition of what the model cares about. Choose it badly and you'll get a model that optimizes the wrong thing, perfectly.
For regression: predicting numbers, like the house prices you tuned by hand in the machine learning lesson, the default is MSE, mean squared error: take each miss (prediction − truth), square it, average over the batch. The squaring is a personality trait, not a detail:
- A miss of 1 costs 1, but a miss of 10 costs 100, big mistakes dominate the total, so the model works hardest on its worst errors.
- Squaring erases the sign: overshooting by 5 and undershooting by 5 hurt exactly the same.
- It keeps the loss smooth, a tiny weight change causes a tiny loss change, which is precisely the signal feed on.
You've already met MSE's temper, even if it wasn't named: in the supervised learning lesson, adding one outlier yanked the MSE fit while the MAE fit barely moved. That was the squaring at work, one huge miss, squared, outshouting all the small honest ones.
The price of being confidently wrong
Classification models don't output numbers to miss by, they output probabilities. Their loss, cross-entropy, reads: take the probability the model assigned to the TRUE answer, and charge −log of it. Drag the slider and watch what happens at the extremes.
Confident and correct: the loss barely registers.
This asymmetry is the whole point. A model that hedges pays a small, steady price; a model that says "99% sure" and is wrong gets hammered with a loss dozens of times larger. Training against cross-entropy is what teaches a network to output honest probabilities instead of blind bravado.
Why not just train on accuracy?
A fair objection: for a classifier we ultimately care about accuracy, the percentage of correct answers. Why not optimize that directly and skip the cross-entropy ceremony?
Because accuracy is a staircase. Nudge a weight slightly and the model's probability for some example moves from 0.62 to 0.63, but the predicted class only flips when it crosses the 0.5 line, so accuracy usually doesn't change at all. A function that's flat almost everywhere has a gradient of zero almost everywhere, and starves: no slope, no signal, no learning. It's the same disease that killed the step function in the activation lesson, and the same cure, replacing a cliff with a slope.
Cross-entropy is that slope. When the probability of the right answer creeps from 0.62 to 0.63, the loss improves immediately, rewarding every tiny step in the right direction long before the prediction actually flips. The division of labor is clean: the loss is for the machine, smooth, differentiable, always giving gradients something to chew on. Metrics like accuracy are for humans, we'll give them their own lesson shortly.
Pick the right loss
For each task, choose the loss function you would train with.
Predicting tomorrow's temperature in °C.
Deciding whether an email is spam or legitimate.
Estimating a house price from size, location and floor.
Classifying a photo as cat, dog, or horse.
Predicting how many minutes a food delivery will take.
Flagging a network login as legitimate or an intrusion.
Check your understanding
Why do we train classifiers on cross-entropy instead of directly on accuracy?
What to remember
- ✓The loss function is the definition of what the model cares about, training minimizes that number and absolutely nothing else.
- ✓MSE (square each miss, average) is the regression default: big errors dominate, sign doesn't matter, and it's smooth.
- ✓Cross-entropy (−log of the probability given to the true answer) is the classification default: hedging costs little, confident wrongness explodes.
- ✓You can't train on accuracy, it's a staircase with no gradient. Loss is for the machine; metrics are for humans.
- ✓Choose the loss to match the task: continuous number → MSE, classes with probabilities → cross-entropy.