Multi-head attention

A single attention pass produces ONE weighting per token. But a token has several relationships running at once: "it" needs its antecedent, its verb, and its position in the phrase, simultaneously. One softmax must average those needs into a single compromise pattern, and compromises blur.

The fix is almost insultingly direct: run several attention operations in parallel, each with its own Q, K, V matrices, each free to specialize. Every parallel copy is called a head. GPT-2 small runs 12 heads per layer across 12 layers: 144 independent spotlights sweeping the same text for different things.

No head is told what to specialize in. Training pressure alone differentiates them, the same way neurons in your CNN protocol ended up as edge detectors nobody designed.

The three heads you just used are cleaned-up archetypes, but they are drawn from reality. When researchers dissected trained transformers, they found:

• Positional heads: attend to the previous token, or two back, or the start of the sentence.
• Syntactic heads: track subject-verb, verb-object, preposition attachments, with surprising precision.
• Rare-token heads: lock onto unusual words, useful for copying names and codes.
• Induction heads: find earlier occurrences of the current pattern and predict "same thing as last time", a mechanism believed central to in-context learning.

And a bucket of heads that do nothing identifiable: prune them and the model barely notices. Training buys capacity in bulk and does not spend it all. The honest summary: multi-head attention is not twelve neat specialists, it is a mob of learned patterns, some sharp, some redundant, whose UNION covers the relationships language needs.

After every head has gathered its evidence, the results are concatenated and passed through one more learned matrix that decides how much of each head's opinion survives. Specialists propose, the projection disposes.