Modeling sequences
Data that flows
Every network so far had a luxury you probably never noticed: the whole input arrived at once, in a fixed size. An image is always 224×224. The perceptron's inputs were always the same list. But look at what most of the world's data actually is: sentences, audio, stock prices, sensor readings, keystrokes, network traffic. It arrives over TIME, one piece after another, in sequences of wildly different lengths.
And in a sequence, ORDER is the meaning. "Dog bites man" and "man bites dog" contain identical words; one is a Tuesday, the other is a headline. Feed both to a model that averages or counts its inputs and they become the same data point, the meaning lived entirely in the arrangement, and the arrangement is gone.
A fixed-size network faces two impossible demands: inputs of length 5 and length 5,000 in the same architecture, and sensitivity to order without an input slot "position 3,412". The recipe that answers both, and defines this whole module: process the sequence ONE STEP AT A TIME, carrying along a memory of what came before. The architecture stays fixed; the sequence can be any length; order matters because each step sees the memory of every step before it.
The four shapes of sequence problems
Before building the machinery, learn to see the shapes. Sequence tasks come in four input→output silhouettes, and naming the silhouette is the first step of designing any system:
- One to one: no sequence at all: one input, one output. Classic image classification. Everything before this module.
- Many to one: read a whole sequence, emit one verdict. A review → its star rating. A network session → hostile or benign.
- One to many: one seed, a sequence out. A photo → a caption, word by word.
- Many to many: sequence in, sequence out. Italian in, English out; audio in, transcript out.
Train your eye on real systems: your spam filter is many-to-one, autocomplete is many-to-many running live, an intrusion detection system watching packets is many-to-one on an endless stream. The silhouette tells you where the memory has to live and when the network must speak.
Name the silhouette
Six real tasks. Classify each by its input→output shape.
Read a full product review, output one star rating.
One photo in, a descriptive sentence out.
Italian sentence in, English sentence out.
One image in, one label out: cat or dog.
Watch a stream of network packets; raise a single alert if the session turns hostile.
The first notes of a melody in, a full continuation out.
Check your understanding
Why can't a model that counts or averages its inputs distinguish "dog bites man" from "man bites dog"?
What to remember
- ✓Sequences break fixed-size networks twice: variable length, and meaning that lives in the order.
- ✓The recipe of the whole module: process step by step, carry a memory. Fixed architecture, any length, order preserved.
- ✓Four silhouettes: one-to-one, many-to-one, one-to-many, many-to-many. Name the silhouette before designing the system.
- ✓Real systems wear these shapes: spam filters and IDS are many-to-one; translation and chat are many-to-many.
- ✓Next: build the memory itself, the recurrent neural network, and the surprisingly simple update rule inside it.