Byte-pair encoding
Nobody wrote the vocabulary
A natural assumption: some linguist at OpenAI curated the token list, choosing sensible prefixes and suffixes. Wrong. No human picked a single token. The vocabulary is learned by an algorithm from 1994 built for file compression, byte-pair encoding, and the recipe fits in one sentence.
Count every adjacent pair of symbols in your corpus, glue the most frequent pair into a new symbol, repeat. Start from single characters, run it 50,000 times, and the merge list IS the vocabulary. "t"+"h" merges early because "th" is everywhere. Later "th"+"e" merges into "the". Frequency decides everything.
This is a pattern you will keep meeting in this protocol: wherever you expect hand-crafted linguistic rules, you find brute statistics instead. BPE knows nothing about grammar, prefixes, or meaning. It knows what co-occurs, and that turns out to be enough.
Run BPE by hand
A micro-corpus of eight words, already split into characters. The table shows the most frequent adjacent pairs right now. Merge your way to a vocabulary, one click per merge, exactly what the real algorithm does 50,000 times.
From toy to production
The real numbers: GPT-2 ran BPE to 50,257 tokens. GPT-4o runs a descendant of it to roughly 200,000. Llama, Claude, Gemini: all BPE variants, different corpora, same merge-the-frequent-pair engine.
The corpus choice has consequences people rarely connect. BPE trained on mostly-English text spends its merge budget on English. The result:
- "The cat is on the mat" → 6 tokens. Efficient, cheap, fast.
- "Il gatto è sul tappeto", same meaning in Italian → roughly 9-10 tokens on GPT-2-era vocabularies.
- Thai, Hindi, or Georgian text can cost 3-5x the tokens of equivalent English.
Same sentence, same model, but the Italian speaker pays more per thought, gets less effective context window, and waits longer per sentence. This is not a moral failing of the model; it is arithmetic falling out of a merge table. But it is the kind of arithmetic you should know exists before you deploy a product in twelve languages.
Check your understanding
Why does "lower" tokenize into fewer pieces than "wixer", even though both are 5 letters?
What to remember
- ✓BPE builds the vocabulary automatically: merge the most frequent adjacent pair, repeat ~50,000 times.
- ✓It is a 1994 compression algorithm, not a linguistic theory. It rediscovers prefixes and suffixes from pure counting.
- ✓The merge list is trained once, on a specific corpus, then frozen. The vocabulary is a snapshot of that corpus's statistics.
- ✓Languages underrepresented in the corpus tokenize worse: more tokens for the same meaning, higher cost, smaller effective context.
- ✓GPT-2: 50,257 tokens. GPT-4o: ~200,000. Different scale, same engine.