Probability Calibration
A model that says 0.8 should be right 80% of the time. Most are not, and AUC will never tell you.
Overview
Two different questions
Discrimination asks whether the model ranks positives above negatives. AUC and [ROC](precision_recall_vs_roc.html) measure this.
Calibration asks whether the numbers mean what they say. Of all the cases where the model said 0.8, were about 80% actually positive?
A model can be excellent at one and terrible at the other, and this is not an edge case — it is the normal situation.
Probability Calibration
Worth knowing
Probability Calibration
Whether a model's confidence means anything, and why every ranking metric is blind to it.
The slider proves it
Drag the temperature control and watch the curve deform. That control performs temperature scaling: it divides the logit by a constant before the sigmoid. Dividing by a number less than one sharpens every score toward 0 or 1; dividing by a number greater than one flattens them toward 0.5.
Crucially, this is a monotonic transformation. It never changes the order of any two predictions. Every ranking metric — AUC, average precision, anything based on ordering — is identical at every setting of that slider, while the reliability curve swings from badly over-confident to badly under-confident.
Which is the point. If you only ever look at AUC, you have no information about calibration at all.
Reading the diagram
Predictions are binned by confidence, and each bin plots mean confidence against observed frequency.
On the diagonal: calibrated.
Below the diagonal: over-confident. The model says 0.9 and is right 70% of the time. This is the common failure, and the damaging one.
Above the diagonal: under-confident. The model hedges when it should not.
ECE — expected calibration error — is the average vertical distance, weighted by how many points fall in each bin. It is the number in the readout, and it is a summary that can hide compensating errors: a curve that is above the diagonal in one region and below in another can report a small ECE. Look at the curve, not only the number.
Why models come out miscalibrated
Modern neural networks are over-confident, and increasingly so as they get larger. Training to minimise cross-entropy on data it can fit perfectly pushes outputs toward 0 and 1 long after the accuracy has stopped improving.
Support vector machines produce distances from a hyperplane, not probabilities, and passing them through a sigmoid does not make them probabilities.
Naive Bayes is famously over-confident because its independence assumption multiplies correlated evidence as though it were independent.
Random forests tend to be under-confident at the extremes, because averaging many trees pulls predictions toward the middle.
Logistic regression, fitted on well-specified features, is usually close to calibrated — it optimises exactly this.
Fixing it
All three methods fit a correction on a held-out set, never on the training data.
Platt scaling. Fit a logistic regression to map scores to probabilities. One or two parameters, works on small validation sets, assumes the distortion is sigmoid-shaped.
Temperature scaling. The single-parameter version, and the standard for neural networks. Because it changes no rankings, accuracy and AUC are guaranteed untouched — a rare free lunch.
Isotonic regression. Fits any monotonic mapping. More flexible, and needs considerably more data or it overfits the validation set.
When it matters
Whenever a probability feeds a decision with costs. Expected value is probability times payoff, so a wrong probability is a wrong decision even when the ranking is perfect.
Medical and risk contexts, where "30% chance" is communicated to a person.
When scores are combined across models or over time.
When a threshold is set from a target rate. [Choosing a threshold](threshold_tuning.html) from calibrated probabilities gives the rate you asked for; from uncalibrated scores it does not.
If all you do is rank — show the top 100 results — calibration is irrelevant.
Expected calibration error, worked
ECE puts a single number on "do the confidences mean anything". Bin the predictions by stated confidence, and in each bin compare the average confidence against the fraction actually positive; ECE is the average gap, weighted by bin size:
| bin | mean confidence | observed accuracy | gap |
|---|---|---|---|
| high | 0.90 | 0.70 | 0.20 |
| mid | 0.70 | 0.65 | 0.05 |
| low | 0.55 | 0.55 | 0.00 |
With 100 points in each bin the ECE is (0.20 + 0.05 + 0.00)/3 = 0.083 — the model is eight points miscalibrated on average, driven almost entirely by the top bin, where it says 0.90 and is right 70% of the time. That over-confidence is the common and damaging failure, and note what it does *not* touch: every one of those predictions could still rank correctly, so AUC and average precision are unchanged. Temperature scaling — dividing the logit by a constant fitted on held-out data — pulls the top bin down toward its true accuracy without reordering anything, which is why it can fix calibration for free while leaving every ranking metric identical. The one caution is that ECE is an average and can hide compensating errors, so a curve above the diagonal in one region and below in another can post a small ECE; read the reliability curve, not only the number.
Why models miscalibrate is worth knowing, because it predicts which fix to reach for. Modern neural networks are over-confident and increasingly so as they grow, because minimising cross-entropy on data they can fit perfectly keeps pushing outputs toward 0 and 1 long after accuracy has stopped improving; temperature scaling is the standard remedy. Naive Bayes is over-confident because its independence assumption multiplies correlated evidence as if it were independent. Random forests are usually *under*-confident at the extremes, because averaging many trees pulls predictions toward the middle. Logistic regression on well-specified features tends to come out close to calibrated, because that is exactly what it optimises.
Where it goes wrong
Calibrating on the training set. It will look perfect and generalise nothing.
Isotonic regression on a small validation set. It overfits.
Assuming a good AUC implies good probabilities. The slider on this page exists to disprove exactly that.
Recalibrating without re-checking after data drift. A calibration fitted last year describes last year's distribution.
When 0.9 does not mean 90%
A model can rank perfectly and still lie about its confidence. This measures the gap, then closes it two different ways.
Check yourself
0 of …Answer without scrolling back up.
Why is AUC unchanged as the temperature slider moves?
A reliability curve sits below the diagonal. The model is:
When does calibration not matter?
Probability Calibration
Crucially, this is a monotonic transformation. It never changes the order of any two predictions. Every ranking metric — AUC, average precision, anything based on ordering — is identical at every setting of that slider, while the reliability curve swings from badly over-confident to badly under-confident.