Train / validation / test split
You can't grade yourself on questions you've memorized
A classmate grinds through 200 exercises before the math final. To "check his level", he re-solves those same 200 exercises, scores 100%, and walks in confident. The exam has different problems. He collapses. Nobody is surprised: testing yourself on questions you've already memorized measures memory, not skill.
Evaluating a model on its training data is exactly that mistake, and the overfitting lesson showed you why: a flexible enough model can score perfectly on data it has seen while having learned nothing usable. A model's grade only counts on data it has never touched.
So the data gets split before training ever starts, classically around 70–80% / 10–15% / 10–15%, into three sets with three jobs:
- Training set: the exercises. The model computes gradients on this; it is the only data that ever changes the weights.
- Validation set: the mock exams. Never trains, but you check it as often as you like while making choices: which , which architecture, when to stop.
- Test set: the real final. Sealed in an envelope until every decision is made, opened once, and its score is the number you report.
Why three sets and not two?
The subtle trap hides in the middle set. Suppose you kept only train + test. You try learning rate 0.1, test says 84%. You try 0.01, test says 87%, keep it. New architecture: 89%. Thirty experiments later, every choice was made by peeking at the test set.
Notice what happened: the test data never touched the weights, but it steered YOUR decisions. You, the human, became the overfitting mechanism, thirty peeks selected whatever combination happens to shine on those particular examples. The test score is no longer an honest forecast of the real world; it's partly a score you curated. That's why in the training-loop lesson watched validation loss: validation exists precisely to absorb all the peeking, so the test set can stay sealed and stay honest.
The general disease is called data leakage: any route by which test information sneaks into training decisions. Classic leaks: duplicate rows that land on both sides of the split; normalizing the data using statistics computed on everything before splitting; shuffling a time series so the model trains on Tuesday's future to predict Monday's past. The symptom is always the same, glorious numbers in the lab, faceplant in production. In security terms: the model saw the answer sheet.
Which set does this touch?
For each situation, pick the split it belongs to.
The data backpropagation computes gradients on.
You compare learning rate 0.1 against 0.01 and keep whichever scores better on this set.
The single score you publish in the final report.
Early stopping watches the loss on this set to decide when training should halt.
Opened exactly once, after every decision has already been made.
The only split whose examples actually change the weights.
You checked your score on this set 40 times while tweaking the model, whatever it was called at the start, that behavior makes it this.
Check your understanding
You tuned your model 50 times, checking the test score after each tweak, and finally hit 96%. What can you honestly claim?
What to remember
- ✓A model's grade only counts on data it never saw, evaluating on training data measures memory, not learning.
- ✓Three sets, three jobs: train = the exercises (changes weights), validation = the mock exams (absorbs your peeking and tuning), test = the sealed final (opened once, reported once).
- ✓Two sets are not enough: repeated peeks at the test set make YOU the overfitting mechanism, even though the weights never touched it.
- ✓Data leakage = test information sneaking into training decisions. Lab numbers soar, production faceplants, the model saw the answer sheet.
- ✓Protocol complete. You started at "will AI take my job?" and you can now read a training run like an engineer: loss, gradients, learning rate, metrics, splits. Next protocol: Deep Learning: opens the box of what happens when networks get deep.