Tokenizer quirks and failures

A tokenizer is frozen statistics from a corpus scraped years ago. The model on top of it can be brilliant, but it computes over those frozen chunks, and at the seams between chunks, strange things happen. Learn the seams and a whole family of "AI is being weird" moments becomes predictable.

Predictable is the key word. These are not random glitches. Each one traces back to a specific property of the vocabulary, which means you can anticipate them, test for them, and design around them.

The recurring failure modes, and where each comes from:

• Character blindness: counting letters, reversing strings, spelling backwards. The model receives multi-character chunks, so character-level questions are answered from memory, not perception.
• Digit chopping: numbers split into 1-3 digit groups misaligned with place value, quietly taxing arithmetic.
• Space sensitivity: "hello" and " hello" are different tokens, which is one reason a prompt can behave differently after an invisible whitespace edit.
• Language tax: non-English text costs more tokens for the same content. Budget 1.5-3x when planning multilingual products.
• Glitch tokens: strings that earned a vocabulary slot but almost never appeared in training afterward. The infamous "SolidGoldMagikarp" (a Reddit username) made GPT-3 hallucinate, insult users, or refuse to repeat it: a token with a slot but nearly untrained meaning, a ghost in the codebook.

The pattern behind all five: the tokenizer was optimized for compressing yesterday's corpus, and your input is not yesterday's corpus. Whenever input looks unlike scraped web text (codes, IDs, tables, novel strings), expect the seams to widen.