RNN basics
A loop with a memory
The recurrent neural network is the "process step by step, carry a memory" recipe made literal. At each time step it takes two things, the new token x_t and its own memory from the previous step h_(t−1), blends them through learned weights, and produces the updated memory: h_t = f(W_x·x_t + W_h·h_(t−1)). Read a word, update your mental summary, move on. That single cell, applied in a loop, is the whole architecture.
The memory h is called the hidden state, and precision matters here: it is a FIXED-SIZE vector that must summarize everything seen so far. After 5 words or 5,000, the summary is the same handful of numbers. It is learned compression under brutal constraints, which is both the trick and, as you are about to feel, the flaw.
And notice a familiar move: the SAME weights W_x and W_h are used at every single time step. You have seen this exact economy before, a CNN shares one filter across every position in SPACE; an RNN shares one cell across every position in TIME. Same trick, different axis: a pattern learned once works at step 3 and at step 3,000.
Watch the memory fade
A tiny RNN with a deliberately crude update rule: new memory = ½ old memory + ½ new token. Feed it a sentence one word at a time, and watch how much of each word survives in the hidden state.
The fine print on that memory
A real RNN is smarter than the ½-blend toy, its weights are TRAINED, so it learns what to keep and what to discard for its task. But the structure of the problem you just watched does not train away:
- Lossy by design: a fixed-size vector cannot hold an unbounded past. Something is always being crushed to make room.
- Recency-biased: each update mixes the past DOWN. Old information survives only by being re-encoded again and again, step after step.
- Strictly sequential: h_t needs h_(t−1) first. No parallelism across time: a GPU that convolves a whole image at once must crawl through a sequence token by token.
Hold onto the fade you just watched. Next lesson we ask the training question, can an RNN even LEARN to preserve early information?, and the answer will reuse, almost line for line, the darkest math of the deep networks module.
Check your understanding
What exactly is the hidden state h in an RNN?
What to remember
- ✓The RNN cell: h_t = f(W_x·x_t + W_h·h_(t−1)), read a token, blend it with the memory, pass the memory on.
- ✓The hidden state is fixed-size learned compression of the entire past: lossy and recency-biased by construction.
- ✓Weight sharing across TIME mirrors CNN sharing across SPACE, one cell, reused at every step, any sequence length.
- ✓Strictly sequential: each step waits for the previous one. Remember this cost when attention shows up.
- ✓Next: unroll the loop and train it, and meet an old enemy from the deep networks module, now stalking the time axis.