The attention mechanism
The promise comes due
Two lessons ago you watched "bank" slide across the semantic map when its sentence changed, and the mechanism doing the pulling stayed a black box. At the end of the deep learning protocol you used attention weights in a translation without seeing how they were computed. Today the box opens.
The computation answers one question, asked by every token about every other token: how relevant are you to me, right now? Three ingredients per token, each produced by multiplying its with a learned matrix:
- Query: what I am currently looking for. The pronoun "it" carries a query shaped like "seeking a thing recently mentioned".
- Key: what I advertise to others. "ball" carries a key shaped like "concrete object, recently mentioned".
- Value: the information I actually hand over if someone picks me.
Relevance = dot product of my query with your key. Big score, high compatibility. The scores then pass through a (the same squash-into-percentages you met with cross-entropy) and become weights that sum to 1. Each token's new vector is the weighted average of everyone's values. That is the entire mechanism.
Compute one attention step
The classic test sentence. Pick a highlighted word as the query and walk the three stages: raw scores, softmax weights, final blend. The numbers are real; the softmax runs live.
Pick a highlighted word: it becomes the query.
Why three matrices instead of none
A fair question: tokens already have embeddings, why not score raw embedding against raw embedding and skip Q, K, V entirely?
Because asking and advertising are different jobs. "it" needs to MATCH "animal" without MEANING "animal": a pronoun and a noun have distant embeddings but must score high in this specific relationship. The query matrix and key matrix learn projections where "seeker of antecedents" and "available antecedent" land close, even when the words themselves do not. Separating value lets a token hand over information different from the bait it advertised.
Two footnotes that complete the real formula. Scores get divided by a constant (√ of the vector size) before the softmax, plain plumbing to stop the softmax from saturating when vectors are long. And in this lesson each token attended to the entire sentence, which suits understanding tasks; generation needs a restriction you will meet in the GPT lesson.
Check your understanding
In attention, what role does the softmax play?
What to remember
- ✓Attention = every token scoring every other token for relevance, then blending values by those weights.
- ✓Query (what I seek), key (what I advertise), value (what I deliver): three learned projections of the same embedding.
- ✓Score with dot products, normalize with softmax, blend the values. All differentiable, all trained by backprop.
- ✓Separate Q and K matrices let "it" match "animal" without meaning "animal": asking and advertising are different skills.
- ✓One attention pass = one round of context absorption. The next lesson runs many of them in parallel.