L1 and L2, and why one is sparse
L2 (ridge, weight decay) penalises the sum of squared weights:
penalty = Σ wi² → gradient = 2wi
L1 (lasso) penalises the sum of absolute values:
penalty = Σ |wi| → gradient = sign(wi)
The gradients explain the difference completely. L2’s pull is proportional to the weight, so as a weight approaches zero the force pulling it there fades — weights shrink toward zero and never quite arrive. L1’s pull is constant regardless of size, so a weight near zero is pushed just as hard as a large one and is driven exactly to zero, where it stays.
That is why L1 produces sparse models and performs feature selection, while L2 keeps every feature with a smaller coefficient. Elastic net uses both, getting L1’s selection with L2’s stability when features are correlated.
Constraining the model on purpose
A network with millions of parameters can fit almost any training set exactly, including its noise. Regularisation is any deliberate constraint that makes that harder, trading a little training accuracy for better generalisation.
The methods available, and what each actually does:
| Method | Mechanism | Typical setting |
|---|
| Weight decay (L2) | Penalises large weights | 0.01–0.1 with AdamW |
| L1 | Penalises absolute weights, drives some to zero | Rare in networks |
| Dropout | Randomly zeroes activations | 0.1–0.5 |
| Early stopping | Stops at the validation minimum | Patience of 5–20 epochs |
| Data augmentation | Expands the effective dataset | Task-specific |
| Batch normalisation | Normalises, and adds batch noise | Standard in CNNs |
| Label smoothing | Softens the targets | ε = 0.1 |
| Mixup / CutMix | Blends examples and labels | Images |
| Gradient clipping | Caps the update size | max_norm 1.0 |
Only some of these are regularisers in the strict sense; several are stabilisers that happen to reduce overfitting. In practice they are used together and their effects overlap.
Weight decay, and what it does geometrically
Adding a penalty on the squared size of the weights changes the objective:
loss = prediction error + λ × Σw²
The gradient of the penalty is 2λw, so every step pulls each weight slightly towards zero — hence the name "decay". Weights survive only if the data justifies them.
Two implementation notes that matter.
Use AdamW, not Adam with weight_decay. Plain Adam adds the penalty into the gradient, which then gets divided by the adaptive scaling — so parameters with small gradients are penalised more, which was never the intent. AdamW applies the decay directly to the weights, decoupled from the scaling.
Exclude biases and normalisation parameters. Decaying a batch-norm γ towards zero fights what the layer is for, and biases carry no capacity worth penalising.
decay, no_decay = [], []
for name, p in model.named_parameters():
(no_decay if p.ndim <= 1 or "bias" in name else decay).append(p)
opt = torch.optim.AdamW([
{"params": decay, "weight_decay": 0.01},
{"params": no_decay, "weight_decay": 0.0},
], lr=3e-4)
That one-line rule — parameters with one dimension or fewer are exempt — captures biases and all normalisation parameters, and is what most reference implementations use.
What to reach for first
Not everything at once. A sensible order:
- Early stopping — free, and it needs no tuning beyond a patience value.
- Weight decay — cheap, always applicable, and it should essentially always be on.
- Data augmentation — the largest single gain on image and audio tasks with limited data.
- Transfer learning — if a pretrained model exists for your domain, this outperforms every regulariser on a small dataset.
- Dropout — if a gap remains after the above.
- A smaller model — last, because it also lowers the ceiling.
Stacking regularisers has diminishing and sometimes negative returns. Batch normalisation already injects noise, so heavy dropout on top of it often hurts. Add one at a time and measure.
Six ways to stop a model memorising
L2, L1, dropout, early stopping, augmentation and more data all fight overfitting by different mechanisms. Here they are on one problem, so the differences are visible rather than described.
Guided experiments
- Train with no penalty. Set Penalty Strength (λ) to its minimum and press Train Network. Weights grow freely and the model fits the targets as closely as it can, noise included.
- Apply L2. Enable the L2 toggle, set Penalty Strength (λ) to about 0.1, press Reset & Randomize Targets and train. Every weight shrinks, and none reaches zero — that smooth shrinkage is the whole behaviour of L2.
- Switch to L1. Enable the L1 toggle instead at the same λ. Now some weights collapse to exactly zero while others stay large. The model has selected a subset of connections and switched the rest off.
- Turn it up too far. Set Penalty Strength (λ) to 0.8 and train. Nearly everything is crushed toward zero and the model can no longer fit even the real structure — regularisation causing underfitting, which is the failure mode at the far end.
Weight decay is not quite L2
The two are used interchangeably and are only equivalent for plain SGD. Weight decay multiplies the weights by a factor slightly below 1 at each step; L2 adds a term to the loss, so its contribution passes through the optimiser’s gradient machinery.
With Adam that difference is real. Adam divides each gradient by a running estimate of its magnitude, which also rescales the L2 term — so parameters with large gradients end up effectively less regularised. AdamW fixes this by applying the decay directly to the weights, outside the adaptive step, and it is the reason AdamW is now the default optimiser for transformers.
What usually goes wrong
- Regularising a model that is underfitting. If training loss is already high, adding a penalty makes both curves worse. Confirm the model can overfit before trying to stop it.
- Penalising the biases. Biases shift the output rather than scale the input, so shrinking them limits what the model can represent without reducing overfitting. Regularise weights only.
- Using L2 with Adam and expecting weight decay. Use AdamW when you want true decay.
- Failing to scale features first. The penalty treats all weights equally, so a feature on a large scale needs a small weight and is under-penalised relative to the rest. Standardise before regularising.
- Tuning λ linearly. It spans orders of magnitude; search it on a log scale, typically 10−5 to 10−1.
In one line
Regularisation adds a weight-size penalty to the loss so the optimiser trades fit against simplicity, with λ setting the rate. L2 shrinks all weights proportionally and keeps every feature; L1 applies constant pressure and drives weights to exactly zero, producing a sparse, self-selecting model. Use it when the model overfits, scale the features first, and reach for AdamW if the optimiser is Adam.
Knowing whether it worked
Regularisation is tuned by watching two curves, not by reasoning about the settings.
Too little: training loss keeps falling while validation loss rises. The gap widens with epochs.
About right: both curves flatten near each other, and validation reaches its best value late in training.
Too much: both curves plateau high, and training loss never gets low. The model is being prevented from learning the signal along with the noise.
That last case is real and easy to miss, because "the model is not overfitting" looks like success. Check the training loss as well: if the model cannot fit its own training data, the constraint is too tight.
The other measurement worth making is the effective one: does validation performance improve? A regulariser that reduces the gap without improving validation has achieved nothing except making training worse.
Implicit regularisation
Several things regularise without being labelled as such, and they explain why large networks generalise better than their parameter counts suggest.
Stochastic gradient noise. Mini-batch gradients are noisy estimates, and that noise biases the optimiser towards flatter minima, which tend to generalise better than sharp ones. Very large batches reduce this noise, which is part of why they sometimes generalise slightly worse.
Early stopping is implicit L2. For linear models this is provable: stopping gradient descent early is equivalent to an L2 penalty of a particular strength.
Architecture itself. Convolution's weight sharing and locality are constraints — a strong prior that image statistics are translation-invariant. That is why CNNs beat transformers on small image datasets: the constraint is doing work that the transformer must learn from data.
Parameter sharing in recurrent and graph networks does the same along their own axes.
Questions people ask
Which regulariser matters most? On images, data augmentation. On tabular data, weight decay and early stopping. Across the board, more data.
What weight decay value? 0.01 with AdamW is a reasonable default; 0.1 for large transformers. Tune it on a log scale.
Do I need dropout with batch normalisation? Often not. If you use both, lower the dropout rate.
Does regularisation slow training? In epochs, usually yes. In final quality on unseen data, that is the point.
Can I regularise too much? Yes — the symptom is a training loss that will not come down.
Is a bigger model with more regularisation better than a smaller one? Frequently, yes, especially with pretrained weights. Capacity plus constraint often beats a small model outright.
Recap in one screen
- Regularisation trades training accuracy for generalisation by constraining the model.
- Weight decay is the baseline — use AdamW, and exclude biases and normalisation parameters.
- Early stopping is free; augmentation is the biggest win on images; transfer learning beats them all on small datasets.
- Add one at a time and judge by validation performance, not by the size of the gap alone.
- Batch noise, early stopping and the architecture itself all regularise implicitly.