Contextual embeddings

Static embeddings answer "where does this word live?" once, at training time, forever. But you never read a WORD; you read an occurrence of a word inside a sentence, and the sentence decides what it means. "bank" next to "river" and "bank" next to "loan" are, for every purpose that matters, two different words wearing the same spelling.

The repair, arriving around 2018 with ELMo and then BERT: compute the vector fresh for every occurrence. Start from a static-ish lookup, then let the surrounding words push and pull the vector before anyone uses it. Same token, different sentence, different final coordinates.

The word for this is contextual, and it is not an incremental upgrade. It dissolves the ambiguity problem instead of managing it: nobody has to enumerate the meanings of "bank" because the sentence itself computes which one is present.

A definition is not a mechanism. Saying "context adjusts the vector" raises the real question: WHAT reads the other words, decides that "river" matters more than "the", and applies the pull, for every token in the sentence at once?

You already brushed against the answer at the end of the deep learning protocol: attention. Each token looks at every other token, scores how relevant each one is, and mixes their information into its own vector, with learned weights. Run that mixing once and "bank" absorbs a hint of "river". Stack the operation in layers and representations get progressively sharper: literal at the bottom, contextual in the middle, task-shaped at the top.

The next module opens that machine bolt by bolt: the scoring, the mixing, the multiple heads doing it in parallel, and the architecture built around it. Module name: The Transformer. Everything you have built so far, tokens and geometry, is its raw material.