Scaling laws
The graph that moved billions
In 2020, researchers at OpenAI (Kaplan and team) published a finding that reads like physics envy fulfilled: plot a language model's loss against its parameter count on log-log axes, and you get a straight line. More parameters, lower loss, following a power law that holds across SEVEN orders of magnitude. Same story for training data and total compute.
Pause on how abnormal that is. Deep learning was a field of alchemy: try it, pray, tune. Scaling laws made one quantity, loss, PREDICTABLE before spending the money. Train a family of small models, fit the line, read off what a 1000x bigger run will score.
That predictability is the quiet answer to "why did anyone dare spend $100M on a training run?" Nobody gambles nine figures on vibes. They extrapolated a straight line and the line kept its promise: GPT-4's loss was forecast from runs a ten-thousandth its size, and the forecast landed.
Ride the line
Loss versus parameter count, log scale, power-law shape. Drag from one million to one trillion parameters and watch what the curve does, and what it never does.
Small-model land: every 10x in size buys a fat chunk of loss. Cheap wins everywhere.
The Chinchilla correction, and what the law does not promise
In 2022, DeepMind's Chinchilla paper caught the field mid-mistake: everyone was scaling parameters and starving models of data. The balanced recipe: roughly 20 training tokens per parameter. Their 70B model, fed properly, beat a 280B model trained on less data. Overnight, "how big is it?" became the wrong question; "how big, on how much data, for how much compute?" is the real one.
And one boundary to keep sharp, because the next lesson lives on it:
- What the law predicts: LOSS, the average next-token error, a smooth aggregate.
- What it stays silent about: which ABILITIES show up at which scale. Whether a model can suddenly do three-digit arithmetic or follow instructions is invisible in the aggregate.
Predictable loss, unpredictable talents. Hold that asymmetry; it is the door into emergence.
Check your understanding
What made scaling laws economically decisive rather than just scientifically neat?
What to remember
- ✓Loss follows power laws in parameters, data, and compute: straight lines on log-log axes, across orders of magnitude.
- ✓That made frontier training a forecastable investment instead of a prayer.
- ✓Returns are logarithmic: each equal improvement costs ~10x the previous one, and an irreducible loss floor waits at the end.
- ✓Chinchilla: scale data with parameters, ~20 tokens per parameter. Balanced beats big.
- ✓The laws predict aggregate loss, not abilities. What talents arrive at what scale is a separate, messier story: next lesson.