What is a token?
Chunks, not words
A token is one entry from a fixed list of text chunks, the model's entire alphabet. Everything you type gets sliced into pieces from that list before the model touches it. In English the pieces average about 4 characters, roughly three quarters of a word.
The slicing follows frequency, not grammar. "the" appears billions of times online, so it earned its own token. "Antidisestablishmentarianism" did not, so it gets assembled from pieces like a ransom note cut out of newspaper headlines.
One detail that surprises everyone: the space before a word usually belongs to the token. " cat" (with the space) and "cat" (at the start of a text) are two different tokens with two different IDs. The model treats them as different symbols, full stop.
Slice text yourself
A miniature tokenizer, same logic as the real ones (greedy match against a fixed vocabulary, longest piece first). Type or use the presets, and watch where the knife falls.
The ␣ symbol marks a space that lives INSIDE a token. This toy vocabulary has a few hundred entries; GPT-4's has about 200,000, but the slicing logic you are watching is the same.
Why tokens run your AI bills and your AI bugs
Tokens are not trivia. They are the unit in which everything about a language model is measured:
- Money: APIs charge per token, in and out. A prompt that tokenizes badly costs real euros more, every single call.
- Memory: the context window ("128k") counts tokens, not words or pages. Bad tokenization means less actual text fits.
- Speed: models generate one token at a time. More tokens per word = fewer words per second on your screen.
And they explain the famous failures. Ask a model how many r's are in "strawberry" and it may miss, because it never received s-t-r-a-w-b-e-r-r-y. It received something like "str" + "aw" + "berry": three opaque symbols. Counting letters inside a token is like asking someone how many bricks are inside a wall they have only seen from a distance, painted over.
My advice: any time a model stumbles on spelling, rhymes, character counts, or exact string manipulation, run the text through a tokenizer viewer before blaming the model. Nine times out of ten the mystery dissolves at the token boundary.
Check your understanding
A model handles "unhappiness" fine but writes gibberish when asked to reverse the string "xk7qzp". The most likely reason?
What to remember
- ✓A token is a chunk from a fixed vocabulary: common words whole, rare words in pieces, ~4 characters each in English.
- ✓The leading space is part of the token: " cat" and "cat" are different symbols to the model.
- ✓Tokens are the billing unit, the context-window unit, and the generation unit. All three.
- ✓Spelling and letter-counting failures are usually tokenization stories: the model never sees individual characters.
- ✓Frequency in the training corpus decides what deserves its own token. The vocabulary is a fossil of internet statistics.