Why models can't read
The illiterate machine
A GPU does one thing at scale: it multiplies grids of numbers. That is the entire physical reality of every AI you have ever talked to. Somewhere between your keyboard and that grid of numbers, the word "cat" has to stop being a word.
This protocol opens the machine that does the stopping. Before attention, before GPT, before any of the famous parts, there is an unglamorous conversion pipeline: text becomes numbers, numbers become vectors, vectors become predictions. Skip this layer and half of a language model's weird behavior looks like magic or madness.
Most people skip it. Then they meet a model that fails to count the letters in "strawberry" and conclude the AI is stupid. The AI never saw the letters. You are about to see exactly what it saw instead.
See what the machine sees
Type anything, or use the presets. Top row: your view. Bottom row: the only thing a computer ever stores, the UTF-8 bytes.
Numbers, yes. But which numbers?
Bytes prove text can be numeric. They are also a terrible unit for a language model, because the model has to learn patterns over these units, and the choice of unit decides how hard that job is.
Three candidates were on the table, and the field tried all of them:
- Characters: a tiny alphabet (fits in a few hundred symbols), but "cat" becomes 3 separate steps with no meaning in any of them, and a page of text becomes thousands of steps to process.
- Whole words: each unit carries meaning, but the list explodes. English has 600,000+ words, then plurals, typos, names, and new slang. The day someone types "rizz", a word-level model is blind.
- Subwords: chunks somewhere between letters and words. Common words stay whole, rare words get built from pieces. A fixed list of ~50,000 chunks covers anything typeable.
Subwords won, and not by taste: they won on math. A fixed, medium-sized vocabulary that can still spell out anything unseen is the only option that scales to the entire messy internet. Those chunks have a name: tokens. They are the next lesson, and they explain more day-to-day AI behavior than any other single concept in this protocol.
Check your understanding
Why do language models not work directly on raw characters or bytes?
What to remember
- ✓A language model computes on numbers. It never sees letters, words, or meaning. Ever.
- ✓Raw bytes are honest numbers but terrible learning units: no meaning per unit, enormous sequence lengths.
- ✓Whole words fail in the other direction: infinite, ever-growing vocabulary.
- ✓Subword chunks (tokens) are the compromise that won: ~50k pieces, common words whole, rare words assembled.
- ✓When a model does something weird with spelling, counting letters, or unusual strings, suspect the units before you suspect the intelligence.