A GPU is not a faster CPU. It is a wider one - thousands of slow cores instead of a few fast ones - and that shape decides which workloads it transforms and which it does not.
Multiplying a 1×512 input by a 512×512 weight matrix uses a tiny fraction of a GPU’s cores; the rest sit idle. Multiply a 256×512 batch by the same weights and the work grows 256-fold while the time barely moves, because that work fills capacity that was already there and already paid for.
This is why batch size dominates GPU utilisation. Below a threshold the GPU is latency-bound and a CPU can genuinely be competitive; above it, throughput takes over and the GPU wins by a wide margin. The same reasoning explains why wider layers use a GPU better than narrow ones.
A CPU has a handful of powerful cores optimised for doing different things quickly, one after another. A GPU has thousands of simple cores optimised for doing the same thing to many values simultaneously.
Neural network training is almost entirely matrix multiplication, which is exactly that pattern: multiply and add, thousands of independent times, with no branching. It maps onto GPU hardware nearly perfectly.
The gap in practice:
| Task | CPU | GPU |
|---|---|---|
| Small tabular network | Seconds | Seconds — no benefit |
| ResNet-50 on ImageNet, one epoch | Days | Minutes |
| Fine-tuning BERT | Days | ~1 hour |
| Inference, single request | Often fine | Faster, and often unnecessary |
The pattern is that GPUs win when the model is large and the batches are large enough to keep thousands of cores busy. For a two-layer network on 10,000 rows, the overhead of moving data to the device can make the GPU slower.
Rarely the arithmetic. Three other things usually bind first.
Memory. Activations from the forward pass must be kept for the backward pass, and that dominates — a ResNet-50's weights are about 100MB while its activations at batch size 64 run to several gigabytes. This is why out-of-memory errors arrive when you raise the batch size, not when you load the model.
The data loader. If the CPU cannot decode and augment images fast enough, the GPU idles. Watch nvidia-smi: utilisation oscillating between 100% and near zero means the loader is the bottleneck, and more num_workers is the fix.
Data transfer. Moving tensors between host and device costs time. Keep them on the GPU, use pin_memory=True, and avoid .cpu() or .item() calls inside the training loop — each one forces a synchronisation that stalls the pipeline.
Modern GPUs have dedicated hardware for 16-bit matrix multiplication, several times faster than 32-bit. Mixed precision uses 16-bit for the heavy arithmetic and keeps 32-bit copies of the weights for accuracy.
scaler = torch.cuda.amp.GradScaler()
for batch in loader:
opt.zero_grad()
with torch.autocast("cuda", dtype=torch.float16):
loss = criterion(model(batch.x), batch.y)
scaler.scale(loss).backward()
scaler.step(opt)
scaler.update()Two to three times faster and roughly half the memory, with essentially no accuracy cost. It should be the default for any GPU training.
The GradScaler exists for a specific reason: 16-bit numbers underflow to zero below about 6e-8, and many gradients are smaller than that. Scaling the loss up before the backward pass and unscaling afterwards keeps them representable. On newer hardware, bfloat16 has a wider exponent range and needs no scaler at all.
CPU and GPU have separate memory. Every tensor must cross the PCIe bus, and that bus is slow relative to on-device bandwidth — roughly 16–32 GB/s against 900+ GB/s of GPU memory bandwidth.
So a computation is only worth moving to the GPU if the arithmetic saved exceeds the transfer cost. Copying a batch across to run one cheap elementwise operation is slower than doing it on the CPU. This is why the standard advice is to move the model and the batch to the device once and keep everything there for the whole step, rather than moving tensors back and forth — and why an innocuous .cpu() or .item() inside a training loop can dominate the profile, since it also forces a synchronisation that stalls the pipeline.
A GPU trades per-core speed for thousands of cores, which suits neural networks because matrix multiplication is embarrassingly parallel. The speedup is real only when there is enough work in flight to occupy the device, so batch size and layer width matter more than anything else, and the separate memory space means transfers and synchronisations can quietly become the bottleneck. Small model, small batch: the CPU may well win.
Not everything needs a GPU, and reaching for one reflexively wastes money.
Tabular data with gradient boosting. XGBoost and LightGBM on CPU are extremely fast and frequently beat neural networks on this data anyway.
Small networks. Below a few hundred thousand parameters and with modest batches, the transfer overhead can exceed the compute saving.
Inference at low volume. A single prediction from a modest model takes milliseconds on a CPU. Many production systems serve models on CPU deliberately: no GPU to provision, simpler deployment, lower cost.
Data preprocessing. Loading, decoding and augmenting are CPU work, and that is where the parallelism should go.
The practical rule: profile before buying. If GPU utilisation sits at 20%, the answer is a better data pipeline, not a bigger GPU.
Data parallelism is the common approach: each GPU holds a full copy of the model, processes a different slice of the batch, and the gradients are averaged across devices. Use DistributedDataParallel rather than the older DataParallel, which is significantly slower.
Note that the effective batch size is multiplied by the number of GPUs, so the learning rate needs scaling accordingly.
Model parallelism splits the model itself across devices, and is only needed when the model does not fit on one. Pipeline parallelism splits it by layers and streams micro-batches through. Sharded training (ZeRO, FSDP) splits the optimiser state, gradients and parameters across devices, which is what makes training models with tens of billions of parameters possible — Adam's state alone is twice the model size.
For most work, one GPU with mixed precision and a well-tuned data loader is enough, and reaching for distributed training before saturating a single device is premature.
A GPU is not a fast CPU. It is thousands of slow cores, which makes it enormously better at some shapes of work and no better at all at others -- and that distinction decides how you write training code.
How much GPU memory do I need? 8GB trains most vision models at modest batch sizes; 24GB is comfortable; large language model fine-tuning wants 40GB or more, or parameter-efficient methods like LoRA.
Why is my GPU only 30% utilised? The data loader, almost always. Raise num_workers, pre-resize images, and check for .item() calls in the loop.
Does mixed precision hurt accuracy? In practice no, with the gradient scaler in place. It is standard.
Can I train on Apple Silicon? Yes — PyTorch's MPS backend uses the integrated GPU. Slower than a discrete card, and much faster than CPU for medium models.
Should I use TPUs? They are excellent for very large-scale training on Google's stack, and involve more code changes than moving between GPUs.
Is more VRAM or a faster GPU better? VRAM decides what you can train; speed decides how long it takes. Memory is usually the binding constraint first.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Latency versus throughput”?
What does this module say about “Why batch size is the variable that matters”?
Watch a head-to-head race! The CPU has low latency but processes sequentially. The GPU has a memory-transfer delay, but processes massive batches in parallel. Adjust the batch size to see who wins.
AI.Cruz builds and maintains AI.Cruz. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.