Why depth matters
One layer can do anything, so why stack fifty?
Here's a fact that should make deep learning sound pointless: a neural network with a single hidden layer can, in theory, approximate any function you want. It's a proven theorem. One layer. Done. So why does every network that actually changed the world, image recognition, translation, GPT, stack dozens or hundreds of layers instead?
The catch hides in two words the theorem whispers: "wide enough". That single layer might need an absurd, exponentially exploding number of neurons, one neuron per special case, like memorizing every possible sentence instead of learning grammar. It can represent the function the way a phone book "represents" a city: technically complete, completely unusable.
Depth is the alternative: build features out of features. Layer one learns strokes. Layer two combines strokes into letters. Layer three combines letters into words. Each level REUSES everything below it, a stroke detector serves every letter, a letter detector serves every word. That reuse is why a deep network gets away with millions of neurons where the wide one would need more than there are atoms available.
Build the hierarchy
A deep vision network builds its understanding in stages, each one made of the pieces below it. Click the stages in order, from raw input to full understanding.
We opened deep networks up, this is what layers actually see
This hierarchy is not a metaphor. When researchers visualized what individual neurons in trained networks respond to (Zeiler and Fergus, 2014, among the most famous experiments in the field), the stages were just… there:
- Early layers: edges at various angles, color gradients. Eerily similar to neurons in V1, the first visual area of your own brain, the same one from the biological inspiration lesson.
- Middle layers: textures, repeated motifs, simple shapes: honeycombs, fur, round things.
- Late layers: task-specific concepts: dog faces, car wheels, human eyes.
Nobody programmed that division of labor. It emerges, every time, because it's the efficient way to see. And notice something valuable: the early layers are generic. Edges are edges, whether the task is recognizing cats or tumors. Keep that thought: it becomes an entire lesson later in this module's sibling, transfer learning.
Which layer does this neuron live in?
Six neurons from a trained image network, described by what makes them fire. Place each one at its depth.
Fires for a diagonal edge, anywhere in the image, on anything.
Fires for fur-like texture: dense, soft, repeating strands.
Distinguishes a husky from a wolf.
Fires for a horizontal stripe of strong contrast.
Fires for round, spoked, rubber-rimmed shapes, wheels, mostly.
Activates strongest when the whole image is a bicycle seen from the side.
Check your understanding
A single hidden layer can theoretically approximate any function. Why do practical networks go deep instead?
What to remember
- ✓One wide-enough layer can approximate anything, but "wide enough" can mean exponentially many neurons. Depth is the efficient path, not the only path.
- ✓Deep = compositional: each layer builds features from the layer below and offers them to the layer above. Reuse is the superpower.
- ✓Visualization studies confirm the hierarchy: edges → textures → parts → objects. Nobody designs it; it emerges.
- ✓Early-layer features are generic across tasks, a fact worth money, as the transfer learning lesson will show.
- ✓Next: if depth is so great, why did deep networks fail to train for decades? Meet the vanishing gradient.