Sampling: temperature and top-p
The last decision belongs to a dice roll
Everything the model computes ends as a probability distribution over the vocabulary. One question remains, every single token: which one gets written? That choice is not made by the neural network. It is made by a sampling rule bolted on after it, and you, the user of an API, own its dials.
The obvious rule, always take the top token, is called greedy decoding, and it writes prose like a hostage: safe, repetitive, and prone to loops ("the the the" pathologies are greedy artifacts). Pure sampling by the distribution's own weights fixes the monotony and introduces a new disease: with ~100,000 tokens in the vocabulary, even tiny per-token probabilities add up to a fat tail of nonsense that eventually gets drawn.
The production answer is two dials on top of the distribution: temperature reshapes it, top-p prunes it. Thirty seconds with each and they stop being mysterious API parameters.
Reshape, prune, roll
A real softmax over five candidate tokens, live. Drag temperature to reshape the distribution, tighten top-p to prune the tail, and sample repeatedly to feel the dice change.
The capital of France is ▍
Low temperature exaggerates the leader's advantage (at 0.1, Paris is effectively certain); high temperature taxes the leader and subsidizes the tail (at 2.0, watch banana climb from decimal dust to a real threat). Top-p attacks the tail directly: at 0.9, banana is simply expelled from the menu no matter how hot things get. Temperature bends the odds; top-p guards the door.
Settings that match jobs
The mechanics in one line each: temperature divides the scores before the softmax (below 1 sharpens toward the favorite, above 1 flattens toward uniform, 0 collapses to greedy). Top-p keeps only the smallest set of tokens whose probabilities sum to p, then renormalizes and rolls among the survivors.
What to actually type into the API:
- Extraction, code, math, anything with a right answer: temperature 0-0.3. You want the spike, not the adventure.
- General assistant work: 0.5-0.8 with top-p ~0.9. The factory defaults, and they earned it.
- Brainstorming, fiction, naming things: 0.9-1.2. You are paying for the tail; occasional weirdness is the product.
Two corrections to popular beliefs. Temperature 0 does not make the model MORE CORRECT, it makes it more repeatable: the favorite token can be a confidently wrong one, and now you get that mistake deterministically. And randomness is not a defect to engineer away: on flat distributions (remember the fiction round in the pretraining lesson) the tie-breaking dice are what keep the model from writing the same email forever.
Check your understanding
Your extraction pipeline runs at temperature 1.2 and occasionally outputs invented fields. First move?
What to remember
- ✓The network outputs a distribution; a sampling rule you control picks the token.
- ✓Greedy is repetitive and loop-prone; raw sampling leaks the long tail. Both extremes lose.
- ✓Temperature reshapes (sharpen below 1, flatten above); top-p prunes the tail and rolls among survivors.
- ✓Match the dial to the job: ~0 for right-answer tasks, defaults for assistants, hot for creative work.
- ✓Temperature 0 buys repeatability, not truth: the favorite token can still be wrong, now consistently.