When one function feeds another, their slopes multiply. Nudge the input and watch the change travel down the chain — this is backpropagation, stripped to its essentials.
The chain diagram above the plot shows each link's local derivative. Their product is the total.
Derivatives of nested functions multiply — and that fact is the entire mechanism of backpropagation.
When a function feeds into another — y = f(g(x)) — a change in x must travel through both to reach y. The chain rule says the sensitivities simply multiply.
Picture two gears. Turning the first makes the second turn 3× as fast; that second gear drives a third at 2×. Turn the first gear once and the last one spins 6 times — the ratios multiplied.
Derivatives are exactly these ratios: how much does the output move per unit of input? Press Nudge x and watch a small change in x produce a change in u, which produces a change in y — each link scaling the one before.
The chain strip shows x → u → y with each link's local derivative underneath. The app also computes a numeric check — the slope measured by actually nudging x by a tiny amount — and it always matches the product. The rule is not an approximation.
A neural network is a deeply nested function. Input goes into layer 1, whose output goes into layer 2, and so on until the loss:
loss = L(f₃(f₂(f₁(x))))
To update a weight in layer 1 you need the derivative of the loss with respect to that weight — and it is a chain of factors, one per layer between the weight and the loss.
Backpropagation is that computation done efficiently. Rather than recomputing the chain separately for every weight, it works backwards from the loss, carrying the accumulated product with it and handing each layer the factor it needs. This is why training a network costs roughly twice a forward pass rather than one pass per parameter.
Multiplying many numbers together is unstable. If each local derivative is less than 1, the product shrinks toward zero — the vanishing gradient. If each is greater than 1, it explodes.
Choose the sigmoid outer function and slide x outwards. Its derivative never exceeds 0.25, so stacking ten sigmoid layers scales the gradient by at most 0.25¹⁰ ≈ 0.00000095. That is precisely why ReLU replaced sigmoid in deep networks, and why LSTMs added a gated path that avoids repeated multiplication.
The chain rule handles functions inside functions, and the intuition is a chain of gears.
If a car travels twice as fast as a bicycle, and the bicycle travels three times as fast as a walker, then the car is six times faster than the walker. Rates through a chain multiply.
Written formally, for y = f(g(x)):
dy/dx = dy/du × du/dx where u = g(x)
The du cancels, which is a helpful (if slightly informal) way to remember it.
A worked example. Let y = (3x + 1)².
Check it by expanding first: (3x + 1)² = 9x² + 6x + 1, whose derivative is 18x + 6. The same answer, and the chain rule got there without expanding — which matters when the expansion is impossible.
The rule composes for any depth. For y = f(g(h(x))):
dy/dx = f′(g(h(x))) × g′(h(x)) × h′(x)
Every layer contributes one factor, evaluated at the value flowing into it. Take y = esin(x²):
Multiply all three: dy/dx = esin(x²) · cos(x²) · 2x.
The procedure never changes: differentiate the outermost function keeping its inside intact, then multiply by the derivative of the inside, and repeat.
Because the chain rule multiplies, a deep network multiplies many numbers together — and multiplication of many small or large numbers is unstable.
If each layer contributes a factor around 0.25 (the maximum slope of the sigmoid), then after ten layers the gradient reaching the first layer is 0.25¹⁰ ≈ 0.000001. The early layers barely move, and the network effectively refuses to train. That is the vanishing gradient problem.
If each factor is around 2 instead, ten layers give 2¹⁰ = 1024, and weights explode into NaN. That is exploding gradients.
Everything in the standard modern toolkit is a response to this arithmetic:
Nested functions multiply their slopes. That gives you the derivative of arbitrarily deep compositions — and because multiplication compounds, it also explains why gradients vanish or explode in deep networks.
Take a single neuron: z = wx + b, a = sigmoid(z), and squared-error loss L = (a − y)².
To find how the loss changes with the weight, chain three factors:
dL/dw = dL/da × da/dz × dz/dw
So dL/dw = 2(a − y) · a(1 − a) · x.
Put numbers in: x = 2, w = 0.5, b = 0, y = 1. Then z = 1, a = 0.731, and the loss is 0.072.
dL/dw = 2(0.731 − 1) × 0.731 × 0.269 × 2 = −0.211.
Negative, so increasing w reduces the loss — and gradient descent duly moves w upwards. Notice too that a(1 − a) is at most 0.25, and is far smaller when the neuron is saturated near 0 or 1. A confidently wrong sigmoid neuron therefore learns very slowly, which is precisely the argument for cross-entropy loss, whose derivative cancels that term.
The chain rule's answer against a numerical one, and the two factors it multiplies shown separately.
Why is it called the chain rule? Because the derivatives form a chain of factors, one per nested function.
Do I need to apply it manually in PyTorch? No — loss.backward() does it. Knowing the rule is what lets you diagnose vanishing gradients and understand why architectures are shaped as they are.
What is the multivariable version? When a variable influences the output through several paths, you sum the contributions of each path. That is what makes backpropagation through a branching graph work.
How does this relate to the product rule? They answer different questions: the product rule is for functions multiplied together, the chain rule for functions nested inside each other. Deep networks use both.
Why do residual connections help? They add an identity path whose derivative is 1, so the product along the shortest route never shrinks — the gradient always has a clean way back.
Can gradients vanish with ReLU? Less easily, but yes — a neuron stuck on the negative side contributes a factor of exactly 0 and stops learning entirely. Leaky ReLU exists to avoid that.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Before the details”?
What does this module say about “The Gear Analogy”?
What does this module say about “Reading the Diagram”?
When one function feeds another, their slopes multiply. Nudge the input and watch the change travel down the chain — this is backpropagation, stripped to its essentials.