Word2Vec and GloVe

Word2Vec (Mikolov and team, Google, 2013) trained a deliberately tiny network on a deliberately dumb game: given a word, predict its neighbors within a few positions. Billions of rounds of that game, and the embedding table that made the predictions work became the product. The network was scaffolding; the coordinates were the point.

The expected outcome happened: similar words clustered. Then came the unexpected one. Researchers tried ARITHMETIC on the coordinates: take the vector for "king", subtract "man", add "woman". The nearest word to the resulting point was "queen".

Sit with how strange that is. Subtraction and addition, operations nobody designed into the training game, turned out to MEAN something. The displacement from "man" to "woman" is a consistent direction in the space, and applying that same displacement to "king" lands on "queen". Relationships had become directions.

Two honest corrections to the legend, then the real limitation.

• The analogies are the postcard, not the terrain: famous examples were selected because they work. Plenty of analogy arithmetic lands on garbage. The space has strong regularities, not clean algebra everywhere.
• GloVe (Stanford, 2014) reached similar spaces from the opposite door: instead of playing prediction games, it factorizes global co-occurrence counts. Two methods, one lesson: the signal was in co-occurrence all along.

The real limitation is structural: ONE vector per word, frozen after training. "bank" gets a single point, stretched between riverbanks and checking accounts, serving both meanings and nailing neither. Every polysemous word in the language has this problem.

A word does not have one meaning; it has a meaning per sentence it appears in. A single static point can never represent that. What you would want is a vector computed fresh, per occurrence, with the sentence taken into account. Hold that thought for two lessons.