Enter ReLU (Rectified Linear Unit)
ReLU is currently the most popular activation function in the world for hidden layers. Its formula is brilliantly simple: if the input is negative, output 0. If the input is positive, output the input unchanged.
Mathematically: f(x) = max(0, x)
In a biological sense, think of a node as a neuron. If the incoming signal (pre-activation) is negative, the neuron decides it's not relevant and does not "fire" (outputs 0). If the signal is positive, it fires proportionately to the signal's strength.
The simplest useful non-linearity
ReLU(x) = max(0, x)
Negative values become zero; positive values pass through unchanged. That is the entire function, and it replaced far more sophisticated alternatives because of what it does to training.
Its derivative is equally simple: exactly 1 for positive inputs, exactly 0 for negative ones.
That derivative of 1 is the crucial property. The chain rule multiplies one factor per layer, so a network of sigmoids — whose derivative peaks at 0.25 — loses a factor of at least four per layer, and after ten layers the gradient reaching the input is a millionth of what left the loss. ReLU contributes a factor of exactly 1 wherever it is active, so gradients pass through depth intact.
This is not a small optimisation. Networks deeper than a handful of layers were not practically trainable before ReLU, and much of the deep learning era rests on this one-line function.
Why any non-linearity is needed at all
Without one, stacked layers collapse. Two linear layers compute W₂(W₁x) = (W₂W₁)x, which is a single matrix — so a hundred linear layers can express exactly what one can: a straight boundary.
Inserting a non-linear function between layers is what lets a network represent curves, corners and disjoint regions. Depth only buys expressiveness when something bends between the layers.
ReLU bends in the simplest possible way — one kink at the origin — and that turns out to be enough. A network of ReLUs is a piecewise-linear function, and with enough pieces it can approximate anything.
The other properties that matter
Speed. A comparison against zero, against an exponential for sigmoid or tanh. On billions of activations that difference is real, and it applies to both the forward and backward pass.
Sparsity. Roughly half the activations in a randomly initialised layer are zero, and trained networks often reach 50–80% sparsity. Sparse activations are cheaper, and there is evidence they make representations more disentangled.
No saturation on the positive side. Sigmoid flattens at both ends, so a strongly activated neuron learns almost nothing. ReLU's positive side never flattens.
The price is the negative side. A neuron whose inputs always produce a negative pre-activation outputs zero, receives a gradient of zero, and stops updating — permanently. This is the dying ReLU problem, and a network can lose a substantial fraction of its neurons this way, usually after a too-large learning rate has pushed weights into a bad region.
Guided Experiments with This Interactive
- Observe the Clipping:
Look at the Pre-Activation sliders on the left. Set a few to negative numbers and a few to positive. Notice the Outputs on the right immediately clip the negative ones to exactly 0.00 (turning grey), while the positive ones carry over perfectly.
- Watch the Forward Pass:
Click the Forward Pass button. Watch the animation on the center canvas. You will see exactly where each input lands on the $y = \max(0, x)$ curve. Notice that any point on the left side of the Y-axis drops straight to the bottom floor line.
- Identify Sparsity:
Click the "Randomize Values" button a few times. On average, about half of the outputs will be zero. This creates a "sparse" network where only a subset of neurons is active at any given time. This makes the network highly efficient and helps prevent neurons from becoming overly dependent on each other.
The "Dying ReLU" Problem
Because ReLU outputs exactly 0 for any negative input, its gradient (slope) for negative values is also 0. During backpropagation, if a neuron consistently receives negative inputs, its weights will never update. It becomes a "dead" neuron.
To solve this, variants like Leaky ReLU were invented, which allow a tiny, non-zero gradient for negative numbers (e.g., $f(x) = 0.01x$ when $x < 0$).
What ReLU does to a feature map
ReLU is one line -- max(x, 0) -- and it is the only reason a deep stack of convolutions is worth building. This shows what it does to real feature-map values, why the alternative is not a deeper network but a pointless one, and the failure mode it brings with it.
Worth remembering
- ReLU stands for Rectified Linear Unit.
- It replaces negative values with 0 and keeps positive values unchanged.
- It provides essential non-linearity to CNNs.
- It is computationally extremely cheap compared to functions like Sigmoid or Tanh.
- It promotes network sparsity (many inactive nodes).
The variants, and when they help
| Activation | Negative side | Notes |
|---|
| ReLU | Exactly 0 | The default; fast; can die |
| Leaky ReLU | 0.01x | Keeps a small gradient alive |
| PReLU | αx, with α learned | Slightly better, slightly more to fit |
| ELU | α(eˣ − 1) | Smooth, mean closer to zero, more expensive |
| GELU | x · Φ(x) | Smooth; standard in transformers |
| SiLU / Swish | x · sigmoid(x) | Smooth; common in modern vision models |
For convolutional networks, plain ReLU remains the sensible default. Leaky ReLU is the first thing to try if you observe many dead units. GELU and SiLU dominate in transformers and in recent efficient vision architectures, where their smoothness appears to help slightly — at a small computational cost.
Preventing dead units matters more than choosing among the variants:
- Use He initialisation, which is designed for ReLU and scales the initial weights by √(2/fan_in). Xavier initialisation, designed for tanh, leaves ReLU networks with activations that shrink through depth.
- Lower the learning rate if you see a sudden collapse in the fraction of active units.
- Add batch normalisation before the activation, which keeps pre-activations centred and makes the dead region much harder to fall into.
Where ReLU sits in a block
The conventional order in a convolutional network is:
Conv → BatchNorm → ReLU
Batch normalisation before the activation is the arrangement the original paper used and it remains standard. It keeps the input to ReLU centred so that roughly half the units are active, which is what the initialisation assumed.
Note that the convolution's bias is redundant when batch normalisation follows, because the normalisation subtracts the mean anyway — which is why bias=False is common on convolutions in ResNet-style code.
In residual blocks the placement is more delicate: the pre-activation variant (BN → ReLU → Conv) trains deeper networks better, because it leaves the identity path completely clean, and that detail was worth a paper of its own.
Questions people ask
How do I know if my ReLUs are dying? Log the fraction of zero activations per layer. Around 50% is healthy; 90%+ that keeps rising is a problem.
Is ReLU differentiable at zero? Not strictly — there is a corner. Frameworks define the derivative there as 0 by convention, and it causes no practical trouble because exactly zero almost never occurs.
Should I use ReLU on the output layer? Only for regression targets that must be non-negative. Classification uses softmax or sigmoid; unbounded regression uses no activation.
Does ReLU cause exploding gradients? It does not damp them, since the positive-side derivative is 1. Combined with poor initialisation or a high learning rate that can allow explosion, which is what gradient clipping and normalisation address.
Why does ReLU work despite discarding half the information? Because the network compensates by learning filters in both polarities, and the sparsity it produces appears to be useful rather than merely lossy.
Is GELU better than ReLU? Marginally, in transformers. In convolutional networks the difference is usually within noise, and ReLU is faster.
Recap in one screen
max(0, x): negatives become zero, positives pass through.- Its derivative is exactly 1 on the positive side, which is what lets gradients survive depth.
- Without a non-linearity, stacked layers collapse into one linear map.
- It is fast and produces sparse activations; the cost is dead neurons on the negative side.
- He initialisation, batch normalisation and a sensible learning rate prevent most dying-ReLU problems.
- Conv → BatchNorm → ReLU is the standard block order.