NLP & Transformers
Module 1 · NLP & Transformers · ~33 min

From Text to Tokens

Models can't read words: how text becomes numbers.

  1. Why models can't read7 minUp next
  2. What is a token?7 min
  3. Byte-pair encoding7 min
  4. Vocabulary and token IDs6 min
  5. Tokenizer quirks and failures6 min