Confidence Intervals
Draw a thousand intervals from a known population and count how many contain the answer. The confidence is a property of the procedure.
Confidence Intervals
Worth knowing
z · SE, so it shrinks like √n.Confidence Intervals
What the ninety-five percent actually refers to, demonstrated by drawing a thousand of them.
The construction
Take a sample, compute its mean, and put an interval around it:
mean +/- z * (sample SD / sqrt(n))The z depends on the confidence level — 1.96 for 95%. The rest is the [standard error](sampling_distributions.html).
What the confidence refers to
Here is the statement people usually give: *there is a 95% probability the true mean lies in this interval.*
That is wrong, and the visualisation shows why. The population mean is fixed at 10 — the dashed line. It is not random. Your interval is either side of it or it is not; there is no probability left once both are determined.
What is random is the interval, because it is computed from a random sample. Draw again and you get a different one. The 95% describes how often that procedure lands on the truth.
The panel draws forty intervals, from a thousand generated. Most cross the dashed line; the highlighted few do not. Nothing distinguishes a missing interval from a covering one when you can only see it from the inside — and in practice you have exactly one, and no idea which kind you have.
The count is real
The readout gives the actual coverage of all thousand. At n = 30 with a 95% level it lands near 94%, and every part of that gap is instructive.
Drag n down to 5 and coverage falls to about 89%. That is not sampling noise. This page uses a z critical value with the sample's *estimated* standard deviation, and at small n that estimate is unreliable enough that the interval comes out too narrow.
That is exactly why the t-distribution exists. Gosset's correction widens the interval to account for having estimated the SD, and the correction is large at small n and negligible past about 30. The multiplier it puts in place of 1.96 is worth seeing, because it explains both halves of that sentence:
| n | t multiplier | wider than z by |
|---|---|---|
| 5 | 2.776 | 42% |
| 10 | 2.262 | 15% |
| 30 | 2.045 | 4% |
| 100 | 1.984 | 1% |
At n = 5 the honest interval is nearly half as wide again as the one this page draws, which is where the missing coverage went. By n = 30 the correction is 4% and the distinction stops mattering, which is where the rule of thumb comes from. Drag n upward and watch the gap close on its own — the page is demonstrating the problem the t-distribution solves, rather than quietly using t and hiding it.
Width and what buys it
Two things set the width.
The confidence level. Switch from 95% to 99% and the intervals lengthen noticeably. A higher level does not locate the parameter better; it hedges more. A 100% interval would be "somewhere between minus infinity and infinity" — perfectly reliable and perfectly useless.
The sample size, through sqrt(n). Quadrupling the sample halves the width.
Reading one properly
"Between 42% and 48%" means the procedure that produced it covers the truth 95% of the time.
A wide interval is a result. It says the data does not pin the answer down, which is genuine information rather than a failed experiment.
Overlapping intervals do not imply no difference. Two 95% intervals can overlap while a direct test of their difference is significant. Test the difference; do not eyeball the bars.
An interval containing zero corresponds to a two-sided test not rejecting at that level. That correspondence is exact, and it is why many people prefer reporting intervals to reporting [p-values](hypothesis_testing_and_p_values.html): the interval carries the effect size and the uncertainty together, where a p-value carries neither.
Two intervals that overlap, and a difference that is real
The rule above — do not eyeball the bars — is worth doing rather than taking on trust, because the arithmetic is short and the result is genuinely counter-intuitive.
An experiment with 100 observations in each arm and a standard deviation of 10, so the standard error in each arm is 10 / √100 = 1.
| arm | mean | 95% interval |
|---|---|---|
| A | 10.0 | [8.04, 11.96] |
| B | 13.2 | [11.24, 15.16] |
Those intervals overlap, between 11.24 and 11.96. Draw them as error bars and every instinct says the arms are indistinguishable.
Now test the difference directly. The difference is 3.2, and its standard error is not the two standard errors added:
SEdiff = √(1² + 1²) = √2 = 1.414, not 2
which gives a 95% interval for the difference of 3.2 ± 1.96 × 1.414 = [0.43, 5.97]. That excludes zero, and the corresponding two-tailed p is 0.024. The difference is significant while the individual intervals overlap.
The reason is in that square root. Two independent errors do not both go the wrong way at once as often as they each go wrong, so they partly cancel, and the difference is pinned down more tightly than either mean is. Adding the errors assumes the worst case every time.
The consequence is a whole band where the eye and the arithmetic disagree. With equal standard errors, a difference becomes significant once it exceeds 1.96√2 = 2.77 standard errors, but the two intervals keep touching until it exceeds 2 × 1.96 = 3.92. Any gap between those two numbers overlaps *and* is significant.
| gap between the means | bars overlap? | significant? |
|---|---|---|
| 2.0 SE | yes | no |
| 3.2 SE | yes | yes |
| 4.5 SE | no | yes |
The middle row is the one that catches people, and it is why the comparison you report has to be a test of the difference rather than a picture of two intervals side by side. The picture is answering a question nobody asked.
Where it goes wrong
Saying "95% chance the mean is in here." The parameter is fixed; the interval is random.
Using z at small n. Use t, and this page shows you the cost of not.
Assuming independence. Clustered or time-series data has a smaller effective sample size, and the interval will be too narrow.
Reporting an interval for a biased estimator. Coverage is about sampling variability. A systematically wrong measurement produces a tight interval around the wrong answer.
What "95% confident" actually counts
Build a confidence interval a thousand times from a known population and count how often it contains the true mean. The number the interval describes is the procedure, not your one result.
Check yourself
0 of …Answer without scrolling back up.
What does the 95% in a 95% confidence interval describe?
Why does a nominal 95% interval cover only about 89% at n = 5 here?
Two 95% intervals overlap. What follows?
Confidence Intervals
That is wrong, and the visualisation shows why. The population mean is fixed at 10 — the dashed line. It is not random. Your interval is either side of it or it is not; there is no probability left once both are determined.