Vanishing and Exploding: Two Sides of One Coin
- w < 1 → vanishing: gradients decay exponentially; early timesteps stop learning; the network only captures short-range patterns.
- w > 1 → exploding: gradients grow exponentially; updates become huge and training destabilizes into NaNs. (The blunt fix — gradient clipping — just caps the magnitude.)
- w = 1 exactly: stable — but nothing keeps a trained weight matrix pinned there. The knife-edge is why vanilla RNNs are fundamentally fragile on long sequences.
The Way Out: Gates
LSTM and GRU cells attack the product itself: they add a cell state with additive updates and learned gates, creating a path where the effective factor stays near 1 for as long as the gates choose. Gradients ride this "constant error carousel" across hundreds of steps — which is why LSTMs dominated sequence learning for two decades, until attention offered an even more direct shortcut.
Multiplication through time
Training a recurrent network means unrolling it. A 50-step sequence becomes a 50-layer network sharing one weight matrix, and backpropagation multiplies one Jacobian factor per step.
The gradient reaching step 1 from a loss at step 50 is a product of 49 factors. If they average below 1, the product collapses:
| Steps back | Factor 0.5 | Factor 0.9 | Factor 1.1 |
|---|
| 10 | 0.001 | 0.35 | 2.6 |
| 25 | 3e-8 | 0.07 | 10.8 |
| 50 | 9e-16 | 0.005 | 117 |
At 50 steps with a factor of 0.5 the gradient is one part in a quadrillion — indistinguishable from zero in floating point. The network cannot learn that step 1 mattered, so it cannot learn long-range dependencies at all.
Two things make factors below 1 the default. Tanh's derivative is at most 1 and usually well below it. And the recurrent weight matrix's largest singular value must be near 1 for the product to be stable, which nothing enforces.
What it looks like in practice
The symptoms are specific enough to diagnose:
- The model learns local patterns and ignores distant ones. It predicts the next word well from the previous two and fails on agreement across a clause.
- Early-layer or early-timestep gradients are orders of magnitude smaller than late ones.
- Loss falls and then plateaus at a level that local context alone can explain.
- Increasing the sequence length does not improve results, because the extra context is unreachable.
The diagnostic is to log gradient norms by time step or by layer:
loss.backward()
for name, p in model.named_parameters():
if p.grad is not None:
print(f"{name:<30} {p.grad.norm().item():.3e}")
A steady decay from output to input is the signature. Similar magnitudes throughout is healthy.
The mirror problem
The same multiplication explodes when the factors exceed 1. A factor of 1.1 over 50 steps gives 117; a factor of 2 gives 10¹⁵.
The result is one enormous weight update that pushes the model far outside any sensible region, the next forward pass produces inf, the loss becomes NaN, and the run is over — nothing recovers from NaN weights.
Exploding gradients are, in one sense, the easier problem: they are loud and immediately visible, and gradient clipping fixes them completely. Vanishing gradients are silent — the model trains, converges, and simply fails to learn what you wanted.
| | Vanishing | Exploding |
|---|
| Symptom | Slow or no learning of long range | NaN loss |
| Visibility | Silent | Immediate |
| Fix | Gating, architecture | Gradient clipping |
| Onset | From the start | Often within a few steps |
The five fixes, measured against each other
The problem is a product of terms that is almost never exactly 1. Each standard remedy attacks a different factor in that product, and this measures how much each one buys.
Guided experiments
- Start at w = 0.70, T = 20. The bars die out about a third of the way back — most of the sentence is unreachable by the learning signal.
- Enable log scale. The decay becomes a straight line — the visual signature of an exponential. Now drag T to 40 and read the survival value: it falls off a cliff.
- Push w to 1.3. The diagnosis flips to EXPLODING and the bars blow through the top of the chart. Then park w exactly at 1.0 and notice how artificial that stability feels — one nudge either way and it's gone.
What to remember
The vanishing gradient problem is not a bug or a tuning issue — it is the inescapable arithmetic of multiplying T numbers that aren't exactly 1. It caps how far back a vanilla RNN can learn, and it is the direct reason LSTMs, GRUs, and ultimately attention-based Transformers exist.
How gating fixes it
The LSTM's answer is an additive path. Its cell state updates as:
cᵗ = fᵗ ⊙ cᵗ₋₁ + iᵗ ⊙ gᵗ
Differentiating with respect to cₜ₋₁ gives the forget gate fₜ. If the network has learned to keep this information, fₜ is near 1, and the gradient passes back through that step essentially unchanged.
Contrast with a plain RNN, where the factor is the derivative of a tanh times a weight matrix — a quantity nothing keeps near 1.
The GRU achieves the same with its update gate: hₜ includes (1 − zₜ) ⊙ hₜ₋₁, so the derivative includes (1 − zₜ), near 1 when the state is being preserved.
This is structurally the same trick as a residual connection: an additive path whose derivative is 1, giving gradients a route that multiplication cannot crush. The transformer's x + sublayer(x), the LSTM's cell state and the GRU's interpolation are three expressions of one idea.
Gating reduces the problem rather than solving it. LSTMs handle hundreds of steps reliably and thousands unreliably, which is part of why attention — where any two positions are one step apart regardless of distance — eventually displaced them.
The full set of countermeasures
For vanishing:
- Use an LSTM or GRU rather than a plain RNN. This is the main fix, and it is free.
- Orthogonal initialisation of the recurrent matrix keeps its singular values at 1, so the product neither grows nor shrinks initially.
- Shorter sequences, or truncated backpropagation through time, which bounds how far gradients must travel.
- Skip connections across time steps in deeper recurrent stacks.
- Attention, which removes the distance dependence entirely.
For exploding:
- Gradient clipping by norm. Essentially mandatory for recurrent models;
max_norm=1.0 to 5.0 is typical. - A lower learning rate.
- Layer normalisation inside the recurrent cell.
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
optimizer.step()
Questions people ask
Why does tanh cause vanishing gradients? Its derivative is at most 1 and approaches 0 when saturated, so the per-step factor is usually below 1.
Would ReLU fix it in an RNN? It removes the saturation problem and makes explosion far more likely, since the derivative is exactly 1 on the positive side and the same matrix is applied repeatedly. Gating is the better answer.
Do LSTMs eliminate the problem? They greatly reduce it. Very long dependencies remain hard, and clipping is still needed for the exploding direction.
How do transformers avoid it? Any two positions are one attention step apart, so there is no long product of factors to decay.
What is truncated BPTT? Backpropagating only k steps back rather than to the beginning. It bounds memory and compute, and prevents learning dependencies longer than k.
Is this the same problem as in deep feed-forward networks? Structurally identical — a product of many factors — and the fixes rhyme: ReLU and residual connections there, gating and clipping here.
Recap in one screen
- Backpropagation through time multiplies one factor per step, so anything not near 1 compounds.
- Factors below 1 vanish the gradient and make long-range learning impossible — silently.
- Factors above 1 explode it into
NaN — loudly, and clipping fixes it completely. - Gating creates an additive path whose derivative is the gate, near 1 when information is preserved.
- That is the same idea as a residual connection, and attention removes the distance dependence altogether.