Bag of words

You need to turn a sentence into a vector. The obvious move, and for decades the standard one: make a giant checklist of every word in the language, and for each sentence, count. "The cat chased the cat" becomes: the=2, cat=2, chased=1, everything else zero. One sentence, one long sparse vector of counts.

This is bag of words, and the name is the confession. Tear the sentence apart, throw the words in a bag, shake. Order? Gone. Grammar? Gone. What survives is pure inventory.

Before laughing at it, respect what it ran: spam filters that actually worked (count "viagra" and "winner", weigh the evidence), search engines, sentiment analysis, topic detection. Through the 2000s this bag powered most of the text-processing economy.

Put the failures side by side and they share one root:

• Order blindness: "dog bites man" = "man bites dog". The bag stores counts, and counts commute.
• Synonym blindness: "movie" and "film" occupy two unrelated slots in the checklist. Similarity between slots is exactly zero, always.
• Dimension bloat: one slot per vocabulary word means vectors with 100,000+ entries, almost all zeros.

The root: in bag of words, every word is a direction of its own, at right angles to every other word. The representation has no concept of "these two words resemble each other". Similarity between words is not weak; it is undefined.

The fix requires a different kind of vector: short, dense, and positioned so that related words sit NEAR each other. Build that, and synonyms stop being strangers. That vector is called an embedding, and it is the rest of this module.