Positional encoding

Here is an embarrassing property of the elegant mechanism from the last two lessons: attention cannot see order. The computation is score-everyone, softmax, blend, and none of those steps knows who came first. Shuffle the input tokens and you shuffle the outputs identically; nothing in the math protests.

To attention, a sentence is a SET of tokens, a bag. And you already know from module 2 what bags do to meaning: "the dog bit the postman" and "the postman bit the dog" become the same input.

RNNs never had this problem: processing one token at a time makes order physically inescapable. Transformers deleted the conveyor belt to win parallelism, and order information went into the trash with it. It has to be bought back, explicitly.

The injection is one addition: before layer 1, each token's embedding gets a position vector added to it. Token 5 receives the "I am position 5" vector on top of its meaning vector. From then on, position rides INSIDE the representation, and attention heads can learn to read it (the previous-token head from last lesson exists only because of this).

What the position vectors look like has gone through three generations:

• Sinusoidal (the 2017 original): each position gets a unique fingerprint built from sine waves of different frequencies, like a clock with many hands. Chosen so relative offsets are easy for the network to compute.
• Learned positions (GPT-2 era): forget the formula, make each position vector a trainable parameter and let backprop shape it. Simple, works, but only for positions seen in training.
• Rotary embeddings, RoPE (Llama and most modern models): rotate query and key vectors by an angle proportional to position, so attention scores depend on relative distance directly. Currently the default choice.

The differences matter for extending context windows (sinusoids extrapolate poorly, RoPE stretches better), but the principle is identical across all three: make position part of the vector, because the mechanism reading the vectors is order-blind by construction.