PyTorch has two key ideas: tensors as the data structure for all numerical computation, and autograd for automatic differentiation via dynamic computation graphs. Together they make implementing backpropagation trivial.
A tensor is a generalisation of arrays to arbitrary dimensions: a scalar is a 0-D tensor, a vector is 1-D, a matrix is 2-D, and a batch of RGB images is a 4-D tensor (batch × channels × height × width). In PyTorch, every piece of data — inputs, weights, outputs, gradients — is a tensor. Tensors can live on CPU or GPU and support the same operations as numpy arrays, plus automatic differentiation.
0-D (scalar)
tensor(3.14)
1-D (vector)
[1, 2, 3]
2-D (matrix)
[[1,2],[3,4]]
4-D (image batch)
(N,C,H,W)
When you set requires_grad=True on a tensor, PyTorch tracks every operation performed on it, building a dynamic computation graph. When you call .backward() on a loss scalar, PyTorch traverses the graph in reverse — applying the chain rule at each node — and accumulates gradients in the .grad attribute of each leaf tensor.
Dynamic vs. static graphs: TensorFlow 1.x used static graphs — you defined the computation graph first, then ran data through it. PyTorch uses a dynamic graph: the graph is re-built from scratch each forward pass. This makes debugging with standard Python tools trivial and supports variable-length inputs naturally (e.g., variable-length sequences in NLP).
Every PyTorch training loop has the same four steps per batch:
1. Forward pass
Run input through the model to get a prediction. The computation graph is built here.
2. Compute loss
Compare prediction to ground truth. Loss is a scalar tensor.
3. Zero gradients
optimizer.zero_grad() — clear accumulated gradients before backward. Forgetting this adds gradients from previous batches.
4. Backward + step
loss.backward() computes gradients; optimizer.step() updates weights using the gradients.
Why zero_grad() matters: PyTorch accumulates (adds) gradients in the .grad attribute by default — it does not overwrite. If you forget to call zero_grad(), you get a running sum of gradients across batches, which blows up. Always call zero_grad() before backward() in every training iteration.
nn.Module is the base class for all neural network layers and models in PyTorch. A Module has a forward() method you override, and it automatically tracks all registered parameters. Layers (Linear, Conv2d, LSTM) are all Modules. A model is itself a Module that contains other Modules — the tree structure enables automatic parameter discovery and clean save/load.
Say the answer out loud before revealing — recalling is what makes it stick. ➜ Next: ANN in PyTorch
What is a tensor and how does it differ from a numpy array?
What is the computation graph in PyTorch and why is it called dynamic?
Why must you call optimizer.zero_grad() before loss.backward() in every iteration?
What is nn.Module and why is it the base class for neural network layers?