Overfitting and generalization
The problem nobody's mentioned yet
Three lessons back you learned three totally different ways for a model to learn: from labeled examples, from structure with no labels, from reward signals after acting. Different sources, but so far you've judged every single one of them the same way, by how well they matched the data you gave them. That's about to bite you.
The bite: a model that matches its training data perfectly is not automatically a good model. It's entirely possible to build something that scores a flawless zero error on every example it was shown, and then falls apart the moment it sees anything new. That gap: training performance vs. real performance, is what this lesson is about.
- Underfitting: a model too simple to capture the real pattern, so it does badly everywhere, even on the data it trained on.
- Overfitting: the opposite failure, a model that captured the training data so precisely it memorized the noise along with the signal, and then can't handle anything slightly different.
- Generalization: what you actually want, sitting between the two, the ability to perform well on data the model has never seen.
You're about to see all three happen to the exact same dataset, live, just by changing one setting.
Fit the curve
Below is a dataset: ad spend on the x-axis, conversions on the y-axis. The blue dots are training data, what the model gets to see and fit. The orange squares are test data, new points the model never saw while fitting, used only to check how well it actually generalizes. Drag the slider to change how complex the fitted curve is allowed to be, from a straight line up to a wildly flexible curve, and watch both errors.
This curve is too rigid to capture the real relationship, it's missing the pattern everywhere, training data included. Notice the training error itself is already high. That's the signature of underfitting: it's not even doing well on the data it's allowed to see.
Why you need data the model has never seen
Here's the uncomfortable part: if all you ever look at is training error, overfitting is invisible. The training error kept dropping the entire time you dragged that slider to the right, a model tracking only that number would tell you the wildest, most overfit curve was the best one you built. It's wrong, and training error alone can't tell you that.
The fix is exactly what you just used: hold out a chunk of data the model never gets to see while fitting, and only check performance on it afterward. That's the test set. The moment training error and test error start pulling apart, training kept falling, test started climbing, you're watching overfitting happen in real time.
In practice this split gets used more carefully than just "train" and "test": there's usually a third slice too, used while you're still tuning a model, so you don't quietly overfit to the test set itself by peeking at it too many times. That three-way split: and exactly how to use it correctly, is a lesson on its own, later in this Academy.
Underfitting, overfitting, or good fit?
Same three labels every time. Read the situation and decide which failure mode, if any, it describes.
A student memorized the exact wording of every answer on the practice exam, word for word, and then panicked when the real exam asked about the same concepts in different phrasing.
A student only skimmed the chapter titles the night before the exam, and did poorly on both the practice questions and the real exam.
A student worked through the underlying concepts, and handled both the practice questions and unfamiliar exam questions about equally well.
A house-price model uses only "number of bedrooms" to predict price, and its predictions are noticeably off even on the houses it was trained on.
A house-price model uses 200 variables, down to the color of the front door, and predicts the exact sale price of every house in its training data, but its predictions on newly listed houses are wildly off.
A spam filter flags spam correctly about 95% of the time on both the emails it was trained on and on brand-new emails it's never processed before.
Module 2, closed
That closes this module. In five lessons you've gone from "what is machine learning" to running all three learning paradigms yourself, supervised, unsupervised, reinforcement, and now the failure mode that can quietly break any of them: overfitting.
One thread ties all five lessons together: a model is only useful to the extent it generalizes. Not how well it matches the data you already have, but how well it holds up on the data you don't have yet. Every technique in the rest of this course, different architectures, different training tricks, different evaluation methods, exists in service of that one goal.
Next module goes one level deeper: Neural Networks Intro, where you'll open up what's actually inside the models you've been training as black boxes so far, starting from a single artificial neuron.
What you learned
- ✓Underfitting: a model too simple to capture the real pattern, it performs badly everywhere, training data included.
- ✓Overfitting: a model that captured noise along with signal in its training data, it looks flawless on training data and falls apart on anything new.
- ✓Generalization is the actual goal: consistent performance on data the model has never seen, not a perfect score on the data it already has.
- ✓Training error alone can hide overfitting completely, it keeps dropping even as a model gets worse in every way that matters. A held-out test set is what makes the problem visible.