Learning Curves
Two lines that tell you whether to collect more data or change the model — and they give different answers.
Overview
What is plotted
Train the model on 40 examples, record the error on those examples and on a held-out validation set. Repeat with 80, 120, and so on. Plot both against training-set size.
Grey is training error, orange is validation error. The shape of the pair is the diagnosis, and the diagnosis is more useful than either number alone.
Learning Curves
Worth knowing
Learning Curves
The plot that answers whether to collect more data, and saves you from collecting it pointlessly.
Underfitting
Set capacity to 1 or 2. Both curves rise quickly to a plateau, sit close together, and stay high.
Training error is high, which is the tell. The model cannot fit the data it has already seen, so nothing about unseen data is the problem. This is bias: the model is too simple for the structure present.
The important consequence is negative. More data will not help. The curves have already converged; adding examples moves neither. Money spent on collection here is wasted, and this plot is the cheapest way to find that out before spending it.
What helps instead: more capacity, better features, less regularisation, a different model class.
Overfitting
Set capacity to 8 or 9. Training error drops close to zero, validation error stays well above it, and a wide gap opens between them.
The model fits its training data nearly perfectly and generalises poorly. This is variance: capacity is being spent memorising noise.
Here more data *does* help, and the curve says so — the validation line is still falling at the right-hand edge. Every additional example makes the noise harder to memorise. Other options: reduce capacity, add regularisation, augment the data, or ensemble.
About right
Set capacity to 4 or 5. The curves converge to a small gap at a low error, and both have flattened.
The readout says as much. There is no obvious bias and no obvious variance, and the flattening is the answer to "should we collect more data": no, not for this model. Improvement would have to come from features or from a different model class, not from volume.
The summary
| Training error | Gap | Diagnosis | Do |
|---|---|---|---|
| High | Small | Bias | more capacity, better features |
| Low | Large | Variance | more data, regularisation, less capacity |
| Low | Small | Fine | features or a different model |
| High | Large | Something is wrong | check the split and the labels |
That last row is worth keeping. High training error *and* a large gap should not happen from bias or variance alone, and usually means a bug — leakage in reverse, mislabelled data, or a validation set drawn from a different distribution.
Where the curve is still falling
The single most valuable thing here is being able to answer "will more data help" with evidence.
If the validation curve is still descending at your current dataset size, collecting more will improve the model, and the slope gives a rough sense of how much. If it has been flat for the last few points, more data buys nothing, and the effort belongs elsewhere.
That question is otherwise answered by intuition, and it usually involves real money.
Practical notes
The curves are noisy at small sizes, because a model trained on 40 examples depends heavily on which 40. Average over several splits, and plot the spread if you can.
Use the same validation set at every size. Growing it alongside the training set confounds two changes.
And do not confuse this with a validation curve, which plots error against a hyperparameter at fixed data size. Both are useful and they answer different questions: learning curves ask about data, validation curves ask about a setting.
Reading the four shapes
The diagnosis is the *pair* of curves, not either number, and four combinations cover every case:
| training error | gap to validation | diagnosis | what helps |
|---|---|---|---|
| high | small | bias (underfit) | more capacity, better features |
| low | large | variance (overfit) | more data, regularisation, less capacity |
| low | small | about right | features or a different model, not volume |
| high | large | a bug | check the split, the labels, for leakage |
The one that saves real money is the top row against the second. Both look like a model that could be better, but they answer "should we collect more data" in opposite directions. In the bias case both curves have already flattened and sit close together and high — the model cannot even fit what it has, so more of the same data moves nothing, and the plot is the cheapest way to learn that before paying for collection. In the variance case the validation curve is still falling at the right-hand edge, so every extra example makes the noise harder to memorise and the gap narrows. The rule is precise: a *converged* gap is variance you cannot buy your way out of; a still-descending validation curve is the one that justifies the data budget.
Two practical cautions keep the diagnosis honest. The curves are noisy at small training sizes, because a model fitted on 40 examples depends heavily on *which* 40, so average over several splits before reading a shape into them. And use the same validation set at every training size — growing it alongside the training set confounds two changes and muddies the very gap you are trying to read. The learning curve, which varies the amount of data, is also not to be confused with a validation curve, which varies a hyperparameter at fixed data size: both are useful and they answer different questions, one about collection and one about a setting.
Where it goes wrong
Reading a single run. Small-sample noise looks like structure.
Plotting accuracy on imbalanced data. The curves will be flat and uninformative. Plot the metric you actually care about.
Concluding 'more data' from a gap alone. Check the validation curve is still falling. A converged gap means variance you cannot fix with volume.
The chart that tells you whether more data will help
Two models on the same problem, scored as the training set grows. One curve says collect more data; the other says do not bother. Reading the difference is the whole skill.
Check yourself
0 of …Answer without scrolling back up.
Training error is high and the two curves have converged. What does that mean?
What tells you whether collecting more data will help?
High training error AND a large gap suggests what?
Learning Curves
Train the model on 40 examples, record the error on those examples and on a held-out validation set. Repeat with 80, 120, and so on. Plot both against training-set size.