Accuracy, precision, recall
The 99%-accurate useless model
Loss is what the machine trains on; once training ends, humans need to judge the result, that's what metrics are for. The most natural one is accuracy: the percentage of answers the model got right. Intuitive, universal… and it lies, hard, the moment your classes are imbalanced.
Concrete case. A disease screening dataset: 1000 patients, of whom 10 are actually sick. Now consider the dumbest possible "model", a single line of code that answers healthy for everyone. It gets 990 of 1000 right: 99% accuracy. It also catches exactly zero sick patients. Same story for fraud detection (999 legit transactions per stolen card), intrusion detection, spam, any domain where the interesting class is rare. A 99% accuracy headline can hide a model that has never once done its actual job.
To see through the headline, split "being right" into its four possible outcomes. Convention: positive means the model raised the alarm (sick, spam, fraud); negative means it stayed quiet.
- True positive (TP): it raised the alarm, and there really was a fire.
- False positive (FP): alarm, but no fire: a false alarm.
- False negative (FN): silence, but there WAS a fire: a miss.
- True negative (TN): silence, and there was truly nothing.
The memory hook: positive/negative is what the MODEL said; true/false is whether it was right. Every metric worth knowing is built from these four boxes.
Sort the four outcomes
Positive = the model raised the alarm. For each event, pick the box it belongs in.
A spam filter sends a real phishing email to the junk folder.
A spam filter junks the email where your school confirms your exam date.
An antivirus stays quiet while actual malware runs on the machine.
A fraud detector approves a normal grocery purchase.
A covid test comes back positive for a perfectly healthy person.
An intrusion detection system flags a real attacker's login attempt.
A medical screening tool tells a sick patient they're healthy.
A smoke detector stays silent through an ordinary, smoke-free afternoon.
Precision and recall: two questions, two metrics
From the four boxes come the two questions that actually matter, each with its own metric:
- Precision: when the model raises the alarm, how often is it right? TP / (TP + FP). High precision = alarms you can trust, few false alarms.
- Recall: of all the real cases out there, how many did the model catch? TP / (TP + FN). High recall = few misses.
Run the useless screening model through them: it raises no alarms, so it catches none of the 10 sick patients, recall = 0/10 = 0%. One number, and the 99% accuracy fraud collapses instantly.
Here's the uncomfortable part: precision and recall pull against each other. Make the model alarm at the slightest suspicion and you'll catch nearly everything (recall up) while drowning in false alarms (precision down). Make it alarm only when certain, and the alarms become trustworthy (precision up) while misses pile up (recall down). When one number is needed for comparing models, the F1 score combines both, but the real decision, which side of the trade to favor, is not a math question at all. It depends on which mistake costs more in the real world. That's the next exercise.
Which mistake is worse here?
For each system, decide which metric you would push first, by asking: which error hurts more, a false alarm or a miss?
Cancer screening, where every flagged patient simply gets a more accurate follow-up test.
The spam filter on your school inbox.
An intrusion detection system that pages the on-call security engineer at 3 AM.
Airport screening for weapons in carry-on luggage.
A tool that automatically deletes any file it classifies as malware.
Check your understanding
A disease affects 5 people in 1000. A model answers "healthy" for everyone. Which number exposes it as useless?
What to remember
- ✓Accuracy lies under class imbalance: the always-say-no model scores 99%+ while never doing its job.
- ✓Four outcomes: TP, FP, FN, TN. Positive/negative = what the model said; true/false = whether it was right.
- ✓Precision = how trustworthy the alarms are. Recall = how many real cases got caught.
- ✓They trade off against each other; choosing which to favor depends on which real-world mistake costs more, not on math. Destructive automatic actions demand precision; catastrophic misses demand recall.
- ✓Loss trains the model, metrics judge it, and the metric must be chosen as carefully as the loss.