Why transformers beat RNNs

The tempting story: transformers won because attention is smarter than recurrence. The true story is blunter: transformers won because they made GPUs profitable. Architecture quality mattered, but throughput decided the war.

An RNN must finish token 41 before touching token 42; a 10,000-token document is a 10,000-step queue, and a GPU (a machine built to do millions of things at once) idles in that queue. A transformer processes all 10,000 tokens simultaneously during training. Same hardware, orders of magnitude more learning per hour.

Add the path-length bonus: between token 3 and token 9,000, an RNN's gradient crosses ~9,000 multiplications (you watched signals die in far shorter chains), while attention connects them in ONE step, at every layer. Cheaper training AND healthier gradients over distance. That combination is why "train on the whole internet" went from fantasy to budget line.

Attention's price tag is printed right in the race panel: every token scores every token, so compute grows with the SQUARE of length. Double the context, quadruple the attention bill. The 128k-token context windows you use today are a stack of engineering workarounds (sparse patterns, KV caches, memory tricks) built to blunt that square.

And at generation time the tables partially turn: an RNN carries one compact state forward, while a transformer generating token 10,001 must still attend over the 10,000 before it. Inference cost is the transformer's ongoing tax, paid on every token it ever writes.

Which is why the story is not closed. State-space models (Mamba and relatives) are recurrence rebuilt on modern math: parallel to train, RNN-cheap to run. Whether they dethrone attention or merely haunt it, remember the lesson of THIS lesson: architectures win on hardware economics, not elegance. Judge every challenger by that yardstick.