Patience, and why you cannot stop at the first uptick
Validation loss is noisy. Mini-batch sampling, dropout and a finite validation set mean it rises and falls between epochs even while the underlying trend is still improving. Stopping at the first increase would stop almost immediately and almost always too early.
Patience is the number of epochs to keep training without improvement before giving up. With patience 10, the run continues 10 epochs past the best score and stops only if nothing beats it in that window.
The other half is restoring weights. Because training continued past the best point, the final weights are not the best weights — the best checkpoint must be saved when it occurs and reloaded at the end. Skipping that gives you a model that is deliberately patience epochs overfit.
Stopping at the right moment
Train a network long enough and training loss keeps falling while validation loss stops improving and then rises. That turning point is where the model stops learning the pattern and starts memorising the noise.
Early stopping watches validation performance and halts when it stops improving. It is the cheapest regulariser available — free in compute, since it saves time — and it requires no tuning beyond one number.
The mechanism has three parts:
- Monitor a validation metric after each epoch.
- Track the best value seen, and save a checkpoint whenever it improves.
- Stop when it has not improved for
patience epochs, and restore the best checkpoint.
That third detail is the one people miss. Stopping without restoring leaves you with the weights from patience epochs past the best point — the worst of both worlds.
Patience, and why it is not zero
Validation loss is noisy. It dips and recovers, sometimes for several epochs, particularly with small validation sets or aggressive augmentation. Stopping at the first non-improvement would end most runs prematurely.
| Patience | Behaviour |
|---|
| 0–1 | Stops on noise; almost always too early |
| 5–10 | The usual range |
| 20+ | For noisy metrics, or long schedules |
Patience also interacts with the learning-rate schedule. With a cosine decay, the final low-rate epochs often produce a distinct improvement — so patience must be long enough to survive the plateau before it, or you will stop just short of the best result the schedule was building towards.
best, wait = float("inf"), 0
for epoch in range(max_epochs):
train_one_epoch()
val = validate()
if val < best - 1e-4: # min_delta: ignore trivial changes
best, wait = val, 0
torch.save(model.state_dict(), "best.pt")
else:
wait += 1
if wait >= patience:
break
model.load_state_dict(torch.load("best.pt")) # restore the best, not the last
What to monitor
Validation loss is the default, and it is not always the right choice.
Loss is smooth and sensitive, which makes it a good early-stopping signal. But it can rise while accuracy holds steady — a model becoming overconfident on examples it already classifies correctly. Stopping there discards a model that was still improving on the thing you care about.
The task metric — accuracy, F1, IoU, AUC — is what you will report, so stopping on it aligns the decision with the goal. It is noisier, so it needs more patience.
The practical answer: monitor the metric that matters for the decision, with enough patience to ride out its noise, and log both.
Note that the validation set is now part of the training procedure. Every early-stopping decision uses it, so its score is no longer an unbiased estimate of generalisation. Keep a separate test set for the number you report.
Interactive Exploration Guide
- Watch the curves diverge. Press Start Training and follow both lines. They fall together, then separate — training keeps dropping while validation flattens and turns up. That gap is overfitting, drawn directly.
- Stop too eagerly. Set Patience (Epochs) to 5 and press Reset Model, then train. The run often halts during a temporary plateau, before the model has finished improving — underfitting caused by impatience.
- Give it room. Set Patience (Epochs) to 60 and train again. Now the run survives noisy stretches and stops near the true minimum, at the cost of extra epochs after the best point.
- Find the middle. Try 20 and compare where it halts against the visible minimum of the validation curve. Patience is a bias–variance trade of its own: too little stops on noise, too much wastes compute.
Why it works as regularisation
Early stopping restricts how far the weights can travel from their initialisation. With small initial weights and a bounded number of gradient steps, the reachable region of weight space is limited — which is a constraint on effective model capacity, not merely a stopping heuristic.
For linear models with gradient descent this can be made precise: early stopping is equivalent to L2 regularisation, with the number of epochs playing the role of 1/λ. Fewer epochs correspond to a stronger penalty. Deep networks are not linear, but the intuition transfers — which is why early stopping and weight decay are partly redundant.
What usually goes wrong
- Not restoring the best weights. Stopping is only half the technique. Without restore_best_weights=True or an explicit checkpoint reload, you keep the overfit weights you trained past the minimum.
- Monitoring the training loss. Training loss almost never rises, so it will never trigger a stop. Monitor validation loss, or validation accuracy if that is the deployment metric.
- Using the test set as the validation set. Choosing the stopping epoch by test performance leaks the test set and makes the reported number optimistic.
- Patience of 1 or 2. Guaranteed to stop on noise. Scale patience to how noisy the curve is — 10 to 20 epochs is a common default.
- A validation set too small to be informative. With a few hundred samples the curve is so noisy that the stopping point is essentially random.
Key takeaway
Early stopping monitors validation loss, waits patience epochs after the best score, then halts and restores the best checkpoint. It regularises by limiting how far weights travel from initialisation — provably equivalent to L2 for linear models — and it costs nothing beyond the validation split you already have. The two things that make it fail are monitoring the wrong curve and forgetting to restore the best weights.
Why it works as a regulariser
Early stopping is not merely a convenience. For linear models it is provably equivalent to L2 regularisation of a particular strength: limiting the number of gradient steps limits how far the weights can travel from their small initial values, which is exactly what a weight penalty does.
The same intuition carries to networks. Weights grow during training, and larger weights mean sharper, more confident functions that fit finer detail — including noise. Stopping early keeps them smaller.
That also explains why early stopping and weight decay partly overlap, and why adding both usually gives less than the sum of their individual effects.
Where it fits with the other regularisers
| Method | Cost | Tuning |
|---|
| Early stopping | Negative — saves compute | One patience value |
| Weight decay | Free | One coefficient |
| Dropout | Slows convergence | One rate per layer group |
| Augmentation | CPU time | Task-specific choices |
Early stopping should essentially always be on. It costs nothing, it protects against the most common failure mode, and the checkpoint it keeps is the model you would have wanted anyway.
The one situation where it is set aside is very large-scale pretraining, where a fixed step budget and a fixed schedule are decided in advance and the model is not trained to convergence at all — there is no overfitting to stop, because the data is effectively unlimited.
Common mistakes
- Not restoring the best checkpoint, which discards the whole benefit.
- Patience of one, stopping on noise.
- Monitoring training loss, which never rises and so never triggers.
- Using the test set to decide when to stop, which makes the test score optimistic.
- A validation set too small to give a stable signal — a few hundred examples produce metrics that swing by several percent.
- Stopping before a scheduled learning-rate drop has had a chance to help.
Stop at the bottom, not at the end
Validation loss falls, turns, and rises. Early stopping keeps the weights from the turn -- and the two parameters that decide when to stop are where the subtlety lives.
Questions people ask
What patience should I use? 5–10 epochs for most work; more if the metric is noisy or the schedule is long.
Should I monitor loss or accuracy? Whichever you will report, with patience matched to its noise. Log both.
Does early stopping replace other regularisation? No. It complements weight decay and augmentation, and overlaps partly with weight decay.
Can I combine it with a learning-rate schedule? Yes, and reduce-on-plateau plus early stopping is a common pairing — cut the rate on the first plateau, stop on the second.
Should I retrain on train plus validation afterwards? Some do, for the extra data, using the epoch count found by early stopping. It gives up the ability to detect that the retrained model is worse, so it is a judgement call.
What is min_delta for? To ignore improvements too small to be meaningful, so a metric creeping by 1e-6 does not reset the patience counter for ever.
Recap in one screen
- Watch validation performance, keep the best checkpoint, and stop when it stops improving.
- Restore the best weights — stopping without restoring is the common mistake.
- Patience of 5–10 epochs rides out normal noise; patience of 1 stops on it.
- Monitor the metric you will report, since loss and accuracy can diverge.
- It is provably related to L2 regularisation, costs nothing, and should almost always be enabled.