Sampling Distributions and Standard Error
A statistic computed from a sample is itself random. Its spread has a name, a formula, and a square root that governs every survey ever run.
Overview
The statistic is random too
You draw a sample and compute its mean. Draw a different sample and you get a different mean. The statistic is itself a random quantity, and it has a distribution: the sampling distribution.
This is the conceptual step everything in inference rests on. It is also the one that is easy to skip, because in practice you only ever draw one sample and never see the distribution at all — you reason about what *would* happen across the samples you did not take.
The left panel makes it visible by actually doing it: 800 samples drawn, each mean plotted. That histogram is the sampling distribution of the mean, and it is the object confidence intervals describe.
Sampling Distributions and Standard Error
Worth knowing
σ/√n.Sampling Distributions and Standard Error
The idea that makes every confidence interval and every p-value mean something.
Standard error
The standard deviation of a sampling distribution is the standard error. For a sample mean:
SE = sigma / sqrt(n)This follows directly from the [rules for variance](expectation_and_variance.html): the variance of a sum of n independent draws is n * sigma^2, dividing by n to make the mean scales variance by 1/n^2, leaving sigma^2 / n, and the square root gives the formula.
Watch the readout as you drag n: the observed spread of the 800 means tracks the predicted sigma / sqrt(n) closely. That is not a coincidence being illustrated — it is an identity being checked.
Standard deviation is not standard error
These get confused constantly, and the distinction is not subtle.
Standard deviation describes the spread of the data. It is a property of the population, and collecting more data does not reduce it — more careful measurement might, but more measurements will not.
Standard error describes the spread of an estimate. It shrinks as n grows, because a larger sample pins the estimate down better.
A useful test: if the number would change when you collect more data *of the same kind*, it is a standard error. If it would not, it is a standard deviation.
An error bar on a chart could be either, and the two say completely different things. Charts that do not label which are unreadable.
The square root, again
The right-hand panel plots sigma / sqrt(n) against n, and its shape is the practical content of this page.
It falls steeply at first — going from 4 samples to 16 halves the error. Then it flattens. Going from 100 to 400 halves it again, at the cost of three hundred more observations.
Every halving of uncertainty costs four times the data. That is why national polls settle around 1,000 respondents: the marginal precision per person collapses past that point, and the remaining error is dominated by sampling bias rather than sampling noise anyway.
More data cannot fix a biased sample. A poll of 100,000 people who all answer the same badly-worded question is precisely wrong.
Beyond the mean
Every statistic has a sampling distribution, not only the mean. Medians, variances, correlations and regression coefficients all have one, and the formulas are usually harder or unavailable.
The bootstrap exists for exactly this. Resample your data with replacement, recompute the statistic, repeat thousands of times, and the spread of the results estimates the sampling distribution — with no formula and almost no assumptions. It is the same operation this page performs, run on your own sample instead of on a known population.
Where it shows up
Confidence intervals are built directly from the standard error.
t-tests and z-tests compare an observed difference against the standard error of that difference.
Cross-validation variance is the sampling distribution of a performance estimate, which is why a single fold's accuracy is a point with error bars around it.
A/B test duration is a standard error calculation before it is anything else.
The cost of precision, tabulated
The sigma / √n formula has a consequence worth seeing as a table, because it governs how big every survey and every A/B test has to be. With a population standard deviation of 10:
| sample size n | standard error |
|---|---|
| 4 | 5.00 |
| 16 | 2.50 |
| 64 | 1.25 |
| 256 | 0.625 |
Each row quadruples n and the standard error halves — the √n in the denominator made literal. Going from 4 to 16 buys a lot; going from 64 to 256 buys the same *factor* of improvement for four times the absolute cost. Precision has diminishing returns, and they diminish predictably: to add one more decimal place of accuracy you need a hundred times the data. This is the whole reason national polls settle near a thousand respondents rather than ten thousand — past that point each extra person shrinks the error by too little to be worth collecting, and any remaining error is bias, which more data cannot touch.
The table also draws the line between the two quantities people confuse. The population's own spread — its standard deviation — is 10 in every row; collecting more data never moves it. The standard error is the column that shrinks, because it describes the *estimate*, not the data. A useful test: if a number would change when you collect more observations of the same kind, it is a standard error; if it would not, it is a standard deviation. An error bar that does not say which it is could mean either, and the two tell completely different stories.
Where it goes wrong
Reporting SD when you mean SE, or the reverse. Label your error bars.
Assuming independence. The sqrt(n) assumes independent draws. Clustered or time-series data has a smaller effective n.
Treating a large sample as accurate. It is precise. Accuracy is a separate question, and bias does not shrink with n.
Using the formula for statistics that do not have one. Bootstrap instead.
The distribution of a statistic, not of the data
Draw a sample, compute its mean, write it down, repeat ten thousand times. What you get is a different distribution from the one you sampled.
Check yourself
0 of …Answer without scrolling back up.
What is a sampling distribution?
Which shrinks as you collect more data?
Why do national polls settle around 1,000 respondents?
Sampling Distributions and Standard Error
A statistic computed from a sample is itself random. Its spread has a name, a formula, and a square root that governs every survey ever run.