Forward, Then Backward
- Forward pass: tokens enter left to right; each cell computes hₜ from xₜ and hₜ₋₁; the loss L is measured at the end.
- Backward pass: the gradient ∂L/∂h starts at the loss and travels right to left. Crossing each cell multiplies it by the local derivative ∂hₜ/∂hₜ₋₁ — the chain rule, applied once per timestep.
- Weight update: because every unrolled copy shares the same W and U, their gradient contributions from all timesteps are summed before the single weight update.
Unrolling the loop
A recurrent network is a loop, and gradients cannot flow through a loop directly. So training unrolls it: a 50-step sequence becomes a 50-layer feed-forward network in which every layer shares the same weight matrix.
Ordinary backpropagation then applies to that unrolled graph. The only special feature is what happens to the shared weights.
Each weight's gradient is the sum of its gradients from every time step, because it was used at all of them:
∂L/∂W = Σᵗ ∂Lᵗ/∂W
That summation is the entire difference between BPTT and standard backpropagation. Everything else — the chain rule, the storage of activations, the update — is the same.
Two consequences follow directly. Memory grows with sequence length, because every step's activations must be kept for the backward pass. And the gradient for early steps is a long product of factors, which is where vanishing and exploding gradients come from.
Truncated BPTT
Backpropagating through a 10,000-step sequence is impossible: the memory required is proportional to the length.
Truncated BPTT bounds it by propagating only k steps back, then detaching the hidden state and continuing forward:
hidden = None
for chunk in chunks_of(sequence, size=50): # k = 50
out, hidden = model(chunk, hidden)
loss = criterion(out, targets_for(chunk))
optimizer.zero_grad()
loss.backward()
optimizer.step()
hidden = tuple(h.detach() for h in hidden) # cut the gradient path
The detach() is the whole mechanism. The hidden state continues forward, carrying information indefinitely, while the gradient stops at the chunk boundary.
The trade is explicit: the model can use context from arbitrarily far back in the forward direction, and it cannot learn dependencies longer than k, because no gradient connects them. Typical k is 20–100 — long enough for useful structure, short enough to be affordable.
Forgetting to detach is a real bug: the graph grows without bound, memory climbs every step, and eventually the process is killed.
Cost, and where it binds
| Aspect | Scaling |
|---|
| Forward compute | Linear in sequence length |
| Backward compute | Linear in sequence length |
| Activation memory | Linear in sequence length |
| Parallelism across steps | None — strictly sequential |
The last row is the one that mattered historically. Because step t needs hₜ₋₁, the steps cannot be computed simultaneously, so a GPU's thousands of cores sit mostly idle and training time scales with length.
That is a hardware-utilisation limitation rather than a quality one, and gating cannot fix it. It is the main reason transformers — where every position is computed in parallel — displaced recurrent networks for large-scale work, independent of any argument about long-range dependencies.
Exploration guide
- Run the forward pass. Cells light up green in sequence, ending at the loss node — this is the network "reading" the sentence.
- Backpropagate. Amber arrows sweep right to left. Watch the chain-rule product in the sidebar: 1.0, then 0.7, then 0.49... each timestep multiplies again.
- Set T = 7 and repeat. By the time the gradient reaches t = 1 it is 0.7⁶ ≈ 0.12 — barely a whisper. The bar chart makes the decay unmistakable. This shrinkage is the seed of the vanishing gradient problem, covered next.
Unrolling the loop to differentiate it
Training an RNN means treating the loop as a deep network with shared weights. This runs the full backward pass by hand, checks it numerically, and shows why the gradients from every timestep are summed rather than averaged.
Worth remembering
BPTT = unroll the loop, backpropagate through the chain, sum the shared-weight gradients. It makes RNNs trainable — but it also chains together T multiplications, and a long product of numbers below 1 races toward zero. That arithmetic inevitability is the vanishing gradient problem.
Practical notes
Clip gradients. The summation over time steps plus the product of factors makes large gradients common. clip_grad_norm_ with a threshold of 1–5 is standard, and omitting it eventually produces a NaN run.
Pack variable-length sequences. pack_padded_sequence stops the recurrence at each sequence's real end. Without it, gradients flow through padding steps and the final hidden state summarises the padding rather than the sentence.
Watch memory with sequence length. Doubling the sequence doubles the stored activations. If training fits at 200 steps and fails at 400, this is why — truncate, or reduce the batch size.
Detach the carried state between chunks in stateful training, or the graph never gets freed.
Consider gradient checkpointing for long sequences: store activations only every few steps and recompute the rest during the backward pass, trading about 30% more compute for a large memory saving.
How it differs from feed-forward backpropagation
| | Feed-forward | Through time |
|---|
| Weights | Distinct per layer | Shared across steps |
| Gradient for a weight | From one place | Summed over all steps |
| Depth | Fixed by architecture | Set by sequence length |
| Memory | Fixed | Grows with sequence length |
| Parallelism | Across the batch | Across the batch only |
The shared weights are what create both the summation and the long product. A network with distinct weights per step would have neither problem — and would need a separate parameter set for every possible sequence length, which is exactly what parameter sharing exists to avoid.
Questions people ask
Why is it called "through time"? Because the unrolled layers correspond to time steps rather than to distinct network layers.
What value of k should I use? 20–100 typically. Long enough to cover the dependencies you care about, short enough to fit in memory.
Does truncation hurt? It prevents learning dependencies longer than k. The forward state still carries information further, so it is a limit on learning rather than on use.
Do transformers need BPTT? No — there is no recurrence, so ordinary backpropagation applies and all positions are computed in parallel.
Why does my memory grow every batch? A carried hidden state that was never detached, so the graph accumulates.
Is gradient clipping always needed? For recurrent models, effectively yes.
Recap in one screen
- BPTT unrolls the recurrence into a deep network sharing one weight matrix, then backpropagates normally.
- A shared weight's gradient is the sum of its contributions from every time step.
- Memory and compute grow linearly with sequence length, and the steps cannot be parallelised.
- Truncated BPTT bounds cost by detaching the state every k steps — and caps what can be learned at k.
- Clip gradients, pack variable-length batches, and always detach carried state.