One step, worked out
Three points: (1, 2), (2, 4), (3, 6). The answer is obviously m = 2, c = 0, but start from m = 0, c = 0 and let the maths find it.
With m = c = 0 every prediction is 0, so the residuals are 2, 4 and 6.
∂L/∂m = −(2/3)[(1×2) + (2×4) + (3×6)] = −(2/3)(28) = −18.67
∂L/∂c = −(2/3)(2 + 4 + 6) = −8.00
At a learning rate of 0.01: m ← 0 + 0.187 and c ← 0 + 0.08. The line has tilted slightly toward the data. Repeat a few hundred times and it lands on m = 2, c = 0.
The simplest thing gradient descent can do
Fit a line, y = wx + b, by iteratively adjusting w and b to reduce the squared error. It is the smallest complete example of everything training does, with two parameters instead of millions.
The loss is mean squared error:
L = (1/n) Σ (ŷᵢ − yᵢ)² where ŷ = wx + b
Its two gradients, from the chain rule:
∂L/∂w = (2/n) Σ (ŷᵢ − yᵢ) xᵢ ∂L/∂b = (2/n) Σ (ŷᵢ − yᵢ)
Read those two expressions and the algorithm becomes intuitive. Both are driven by the error (ŷ − y). The weight's gradient is additionally multiplied by x, so points far from the origin push the slope harder. The bias's gradient is just the average error, so the bias tracks the overall offset.
Three steps by hand
Data: (1, 2), (2, 4), (3, 6) — the true line is y = 2x. Start at w = 0, b = 0, learning rate 0.1.
Step 1. Predictions are 0, 0, 0; errors −2, −4, −6.
- ∂L/∂w = (2/3)[(−2)(1) + (−4)(2) + (−6)(3)] = (2/3)(−28) = −18.67
- ∂L/∂b = (2/3)(−12) = −8.0
- w ← 0 + 1.867 = 1.867, b ← 0 + 0.8 = 0.8
Step 2. Predictions are 2.67, 4.53, 6.40; errors +0.67, +0.53, +0.40 — now overshooting slightly, and much smaller.
Step 3 onwards. The corrections keep shrinking. After a few hundred steps w approaches 2 and b approaches 0.
Two lessons visible in three steps. The first step is enormous because the error is enormous, and the steps shrink automatically as the fit improves — no schedule needed for that. And the loss overshot from negative to positive error: gradient descent oscillates towards a solution rather than approaching monotonically.
Ordinary least squares has an exact closed-form solution — the normal equation — so iterating towards the answer looks like a step backwards. Three reasons it is not:
Scale. The closed form requires inverting a matrix whose size is the number of features, at roughly cubic cost, and it needs the whole dataset in memory. Gradient descent works on mini-batches and scales to any size.
Generality. Change the loss to absolute error, or add a non-linear activation, and the closed form disappears. Gradient descent needs only a derivative.
It is the same algorithm. This is the point of the exercise: what happens here is exactly what happens in a network with a billion parameters. Same loss, same chain rule, same update rule — only the number of parameters differs.
Try this above
- Turn on Show Target OLS Fit so you can see where gradient descent should end up.
- Set the learning rate very small and step repeatedly — correct direction, painfully slow.
- Raise it steadily. There is a point where each step overshoots the minimum and lands further away than it started; the loss rises instead of falling.
- Set m and c by hand to something far from the answer and watch how the first few steps are large and later ones shrink — the gradient itself gets smaller as you approach.
What usually goes wrong
Divergence from too high a learning rate is the classic failure, and its signature is unmistakable: the loss increases every step, often to infinity within a dozen iterations. If you see that, the fix is almost always η, not the model.Unscaled x values cause a subtler problem. If x is in the thousands, ∂L/∂m is thousands of times larger than ∂L/∂c, so a learning rate that suits one wrecks the other and the path zig-zags down a narrow valley.
In one line
Two parameters, the same optimiser as a deep network, and slow enough to watch every step.
The learning rate, demonstrated
With two parameters the effect of the learning rate is easy to see directly.
| Learning rate | Behaviour on the example above |
|---|
| 0.001 | Converges, very slowly — thousands of steps |
| 0.1 | Converges smoothly in a few hundred steps |
| 0.5 | Oscillates around the answer before settling |
| 1.5 | Diverges — the loss grows until it overflows |
The divergence case is worth understanding: if the step overshoots far enough that the new error is larger than the old one, the next step is larger still, and the process runs away. That is the same mechanism behind NaN losses in deep networks, visible here in two dimensions.
Feature scaling matters for the same reason as in networks. With x values in the thousands, the gradient with respect to w is thousands of times larger than the gradient with respect to b, so the loss surface becomes a narrow ravine and no single learning rate suits both parameters.
import numpy as np
x = np.array([1, 2, 3], dtype=float)
y = np.array([2, 4, 6], dtype=float)
w, b, lr = 0.0, 0.0, 0.1
for step in range(500):
pred = w * x + b
err = pred - y
w -= lr * (2 / len(x)) * (err * x).sum()
b -= lr * (2 / len(x)) * err.sum()
print(w, b) # ~2.0, ~0.0
From this to a neural network
Three additions turn the code above into a network, and nothing else changes conceptually.
More parameters. Replace the scalar w with a matrix, and the multiplication with a matrix product. The gradient formula generalises directly.
An activation function. Wrap the output in a non-linearity and the model can fit curves. The chain rule gains one factor per activation.
Layers. Feed one layer's output into the next. Backpropagation is the bookkeeping that computes all the gradients efficiently rather than recomputing shared factors.
That is genuinely all of it. Everything else — Adam, batch normalisation, dropout, schedules — is a refinement of the same loop: predict, measure error, compute gradients, step.
The smallest possible training loop
One weight, one bias, and a loop -- then the same problem solved exactly, so you can watch gradient descent arrive at an answer you already know.
Questions people ask
Why the factor of 2 in the gradients? It comes from differentiating the square. Some texts use 1/(2n) in the loss so it cancels; it only rescales the effective learning rate.
Does it always converge? For linear regression the loss is convex, so with a small enough learning rate it converges to the unique global minimum. Networks are not convex, and there is no such guarantee.
How many iterations? Until the loss stops improving. For this problem, hundreds; for networks, thousands to millions of steps.
Do I need to scale the inputs? For gradient descent, yes — unscaled features create a ravine. The closed-form solution is unaffected.
What if I use absolute error instead? The gradient becomes the sign of the error, so every step is the same size regardless of how wrong you are. It is more robust to outliers and slower to settle.
Is this how scikit-learn's LinearRegression works? No — it uses the closed form. SGDRegressor is the gradient-descent version.
Recap in one screen
- Fit
y = wx + b by stepping both parameters against the gradient of squared error. - Both gradients are driven by the error; the weight's is additionally scaled by the input.
- Steps shrink automatically as the fit improves; too large a rate diverges.
- The closed-form solution exists but does not scale, does not generalise to other losses, and teaches nothing.
- Add a matrix, an activation and a second layer, and this is exactly how a neural network trains.