Activation functions
Why the step function had to go
The perceptron's step function did its job, output 1 or 0 based on a threshold, but it has two problems that only show up once you try to build something bigger than one neuron.
- No usable slope: a step function is flat everywhere except at one single point, where it jumps straight up. Training a network means nudging every weight a little in whichever direction reduces error, and to know which direction, you need a gradient (a slope) to follow. A function that's flat almost everywhere gives you nothing to follow.
- No added power from stacking: a weighted sum is a linear operation. Stack ten linear layers with nothing but weighted sums between them, and the result is mathematically still just one linear layer, you'd still be stuck drawing a single straight line, exactly the same wall the perceptron hit with XOR.
An activation function's job description is short: it has to be non-linear (so stacking layers adds new shapes the network can draw) and it has to have a usable derivative (so , coming up in a couple of lessons, has a signal to follow). Three functions became the standard answers.
The three you'll actually meet
Sigmoid: σ(x) = 1 / (1 + e⁻ˣ). Squashes any real number into a smooth curve between 0 and 1, historically the first non-linear replacement for the step function, and still useful whenever you want an output that reads as a probability.
Tanh: a rescaled sigmoid that squashes into (-1, 1) instead, centered on 0. Same S-shape, but zero-centered outputs tend to make training slightly better-behaved.
ReLU (Rectified Linear Unit): f(x) = max(0, x). Almost embarrassingly simple: pass positive numbers through unchanged, zero out anything negative. One comparison, nothing else.
Each one solves the differentiability and non-linearity problem. But each one also comes with its own failure mode, which is exactly what the next exercise walks through.
Match the property to the function
For each description, pick which activation function it's describing.
Squashes any input into a range between 0 and 1, useful when you want to interpret the output as a probability.
Squashes any input into a range between -1 and 1, and is zero-centered instead of centered on 0.5.
Outputs the input completely unchanged if it's positive, and exactly 0 if it's negative, no squashing at all.
Was the default choice before deep learning took off, but saturates, flattens out, for large positive or negative inputs, leaving almost no gradient to learn from.
Is the cheapest of the three to compute, just one comparison against zero, which is part of why it became the default in modern hidden layers.
Can suffer from 'dying' neurons: if training pushes a neuron's input to always be negative, its gradient stays at exactly 0 forever and it never updates again.
A natural choice for the final layer of a binary yes/no classifier, since its output can be read directly as a probability.
So which one should you actually use?
In practice, almost nobody agonizes over this from scratch, the field has converged on a rule of thumb:
- Hidden layers: ReLU (or a small variant of it) by default. It's cheap, avoids saturation for positive values, and is what nearly every modern deep network uses under the hood.
- Output layer: it depends entirely on the task. A yes/no prediction wants sigmoid. A choice between several categories wants a related function called (a topic for a later module). A plain number prediction: like a price, often wants no activation at all, just the raw weighted sum.
- Sigmoid and tanh haven't disappeared, they still show up in specific places, like the gates inside LSTM networks (a later module), but as the default choice buried inside every hidden layer of a network, ReLU is what won.
Check your understanding
Suppose every layer of a deep network used the step function as its activation. What would go wrong during training?
What to remember
- ✓An activation function needs two things: non-linearity (so stacking layers actually adds power) and a usable derivative (so backpropagation has a gradient to follow).
- ✓Sigmoid squashes to (0, 1), historically first, still standard for yes/no output layers, but saturates for large inputs.
- ✓Tanh squashes to (-1, 1), a zero-centered version of sigmoid with the same saturation problem.
- ✓ReLU outputs max(0, x): cheap, no saturation for positive inputs, the default choice in modern hidden layers, but can "die" if pushed permanently negative.
- ✓Rule of thumb: ReLU (or a variant) in hidden layers, a task-specific function in the output layer.