Vocabulary and token IDs
The codebook
Once BPE has done its merging, the result is just a numbered list: the vocabulary. Row 464 holds "The", row 7342 holds " cat". Tokenizing a text means replacing each chunk with its row number, so "The cat is hungry" leaves the tokenizer as something like [464, 7342, 318, 8949].
That list of integers is the ONLY thing the model receives. Your prompt, in its entirety, is a sequence of row numbers from a codebook you have never seen.
The step is pure bookkeeping, a lookup table, and that is precisely why it deserves five minutes of attention: people imagine meaning sneaking in here, and it does not. 7342 is not "catlike" in any way. It could have been 12 or 49,000. The number is an arbitrary name tag, nothing more.
Decode the transmission
A sequence of token IDs arrived. Decode it by clicking the matching vocabulary row for each ID, left to right. The codebook is right below.
The reserved seats
A few rows of the codebook are special: tokens that never come from your text, inserted by the system to structure the sequence.
- <|endoftext|>: GPT-2's ID 50256, the last row. Marks "this document is over". During training it separates unrelated texts so the model does not blend a recipe into a news article.
- Padding tokens: filler to make batched sequences the same length. The model learns to ignore them.
- Chat markers: modern assistants wrap turns in special tokens (user turn, assistant turn, system). The "chat" format you know is these invisible tokens all the way down.
Special tokens are also a security surface. If an application naively pastes user text into a prompt and that text can smuggle in chat-marker tokens, the user can impersonate the system role. Keep that thought warm: it returns in the AI Security protocol.
Check your understanding
Token ID 7342 maps to " cat". What does the model know about cats from this number?
What to remember
- ✓The vocabulary is a numbered list of chunks; tokenization = text in, row numbers out.
- ✓The model's true input is a sequence of integers. Nothing else crosses the border.
- ✓IDs are arbitrary name tags: no meaning in the number, no meaning in numeric closeness.
- ✓Special tokens (<|endoftext|>, padding, chat markers) structure the sequence and are a real security surface.
- ✓Open question carried to the next module: if IDs are meaningless, where does meaning come from?