Word embeddings
Give every word an address
An embedding is a short list of coordinates, typically 300 to a few thousand numbers, that positions a word as a point in space. The design goal is one sentence long: words with similar meaning get nearby addresses. "cat" and "kitten" end up neighbors; "cat" and "carburetor" end up in different neighborhoods entirely.
This also answers the cliffhanger from the tokens module. Token ID 7342 is a meaningless row number, but row 7342 of the embedding TABLE holds a vector of learned numbers. The lookup "ID → vector" is the exact border where arbitrary labels become geometry. Meaning does not live in the ID; it lives in the coordinates the ID points to.
Nobody assigns these coordinates. They are parameters, tuned by gradient descent like every weight you met in the earlier protocols. The map draws itself during training.
Walk the map
A 2D shadow of an embedding space (real ones have hundreds of dimensions). Click words and check who their nearest neighbors are. Distances are computed live from the coordinates.
Click any word to measure who lives nearby.
Where do the coordinates come from?
From context. The operating principle of all embedding learning is the distributional hypothesis, stated by the linguist J.R. Firth in 1957: "You shall know a word by the company it keeps."
"cat" and "kitten" appear surrounded by the same kinds of words: purring, vet, fur, sleeping. Training nudges words with shared contexts toward each other, millions of nudges across billions of sentences, and neighborhoods condense out of raw text like dew. No dictionary is ever consulted.
Two calibrations before you trust the pretty 2D picture:
- Real embeddings use 300 to 12,000 dimensions. A 2D map is a shadow of that object; genuine neighbor relationships survive the flattening only approximately.
- The space reflects the corpus, verbatim. Train on text where "nurse" appears mostly near female pronouns and the geometry absorbs that bias, faithfully and silently. The map is the territory the data described, not the territory as it should be.
Check your understanding
Why do "cat" and "kitten" end up close in embedding space?
What to remember
- ✓An embedding is a learned coordinate list that turns a token ID into a position in space.
- ✓The embedding table is the border crossing: arbitrary row numbers on one side, geometry with meaning on the other.
- ✓Similar context → similar coordinates. Firth's "company it keeps" is the whole trick.
- ✓Real spaces have hundreds to thousands of dimensions; every 2D picture is a lossy shadow.
- ✓The space inherits its corpus, including the corpus's biases. Geometry is honest about data, not about the world.