Hypothesis Testing and p-values
The p-value is a shaded area under an assumption. Move the observation and watch the area, and the assumption, stay exactly where they were.
Overview
The machinery
The null hypothesis is a specific, boring claim: no effect, no difference, nothing happening. It is specific on purpose, because a specific claim implies a distribution for the test statistic.
That distribution is the curve. It is what the statistic would look like across repeated samples if the null were true. Every use of a p-value inherits that conditional.
The p-value is the shaded tail: the probability of a statistic at least as extreme as the one observed, under the null.
Drag the observed statistic and watch the area shrink as it moves outward. A large statistic is unusual under the null, so its tail is small — and a small tail is what "unlikely, if the null were true" means.
Hypothesis Testing and p-values
Worth knowing
Hypothesis Testing and p-values
The most used and most misread number in statistics, defined precisely enough to stop misreading it.
One test, worked by hand
The simulation below computes a p-value by brute force. It is worth doing one the ordinary way first, on numbers small enough to check, so the quantity stops being whatever the library returned.
Five measurements per group:
| group | values | mean |
|---|---|---|
| control | 12, 14, 11, 13, 15 | 13 |
| variant | 15, 17, 14, 16, 18 | 16 |
The deviations from the mean are the same five numbers in both groups — −1, 1, −2, 0, 2 — so each group's sum of squared deviations is 10, and each sample variance is 10 / 4 = 2.5. Dividing by n − 1 rather than n is the correction covered in [expectation and variance](expectation_and_variance.html).
The difference between the means is 3. What decides whether 3 is a lot is the standard error of that difference:
SE = √(2.5/5 + 2.5/5) = √1 = 1
t = 3 / 1 = 3, on 5 + 5 − 2 = 8 degrees of freedom, which gives a two-tailed p = 0.0171.
Read that last number as the definition demands: *if the two groups were really the same*, a gap of 3 or more in either direction would turn up in about one sample in 59. It says nothing about how likely it is that they are the same, and nothing about whether a gap of 3 matters to anyone.
Notice which quantity did the work. The difference of 3 was never enough on its own — it had to be measured against the SE, and the SE fell as the spread fell and as the sample grew. Quadruple the sample and the SE halves, because it divides by √n rather than n. That single square root is why the fourth hundred observations buy so much less than the first hundred.
What it is not
Three misreadings, all common enough to have names.
Not the probability that the null is true. The p-value is computed *assuming* the null. It cannot then tell you how likely the assumption was. Getting from P(data | null) to P(null | data) needs [Bayes' theorem](bayes_theorem.html) and a prior, and the two quantities can differ by orders of magnitude.
Not the probability the result was chance. Same error, differently worded.
Not a measure of effect size. With enough data, a difference of no practical consequence produces a tiny p-value. The p-value answers "is there evidence of *any* effect", never "is the effect large enough to matter". This is why [confidence intervals](confidence_intervals.html) are increasingly preferred: they carry the size and the uncertainty in one object.
Failing to reject is not accepting
Set the observed statistic near zero. The p-value is large, and the conclusion is that the data is consistent with the null.
It is not that the null is true. A large p-value is produced both by a real absence of effect and by a study too small to detect one. Distinguishing those requires [power](type_i_and_type_ii_errors.html), and it is why "no significant difference" in an underpowered study means almost nothing.
One tail or two
Switch the alternative. The two-tailed p-value is exactly double the one-tailed one for the same statistic, because the same area is being counted on both sides.
The choice must be made before seeing the data. Deciding afterwards that you only cared about one direction halves your p-value for free, and that is a recognised form of cheating rather than a modelling choice.
Why one in twenty, exactly
The claim that a 0.05 threshold lets one test in twenty through is not a rough estimate. It is exact, and the reason is a fact about p-values that is rarely stated: when the null is true, the p-value is uniformly distributed between 0 and 1. Every value is equally likely. A p of 0.03 is exactly as probable as a p of 0.87.
Once that is granted, everything else follows arithmetically. The probability that a uniform value falls below α is α. So a 0.05 cutoff admits 5% of true nulls by construction — not as a regrettable side effect, but as the definition of what the cutoff is doing. Twenty thousand tests with no effect in them at all:
| threshold | fraction of nulls below it | uniform predicts |
|---|---|---|
| p < 0.01 | 0.0111 | 0.01 |
| p < 0.05 | 0.0520 | 0.05 |
| p < 0.10 | 0.1003 | 0.10 |
| p < 0.25 | 0.2492 | 0.25 |
| p < 0.50 | 0.4953 | 0.50 |
| p < 0.90 | 0.8968 | 0.90 |
The deciles come out 0.10, 0.20, 0.30, 0.40, 0.50, 0.61, 0.71, 0.81, 0.90 — the identity line, which is what a flat distribution looks like from the inside.
This is also the mechanism behind p-hacking, and it makes the size of the problem computable rather than vague. Running k independent tests on nothing means drawing k uniforms and keeping the smallest, and the smallest of k uniforms has an average of 1/(k + 1):
| tests run | average smallest p | chance at least one clears 0.05 |
|---|---|---|
| 5 | 0.167 | 1 − 0.95⁵ = 22.6% |
| 20 | 0.048 | 1 − 0.95²⁰ = 64.2% |
| 100 | 0.010 | 1 − 0.95¹⁰⁰ = 99.4% |
Twenty metrics on an experiment that changed nothing, and the odds are roughly two in three that one of them looks significant. At a hundred it is a near certainty. Nobody has to be dishonest for this to happen — it is what the arithmetic does on its own, which is why the correction has to be decided before the tests are run.
Why 0.05 is under attack
The threshold is a convention. Fisher suggested it as a rough guide and explicitly did not intend it as a decision rule.
The problems are structural.
p-hacking. Test enough hypotheses and something crosses 0.05 by chance. At 0.05, one test in twenty does so with no effect present at all.
Publication bias. Significant results get published, so the literature over-represents them, and the effect sizes it reports are inflated.
The cliff. p = 0.049 and p = 0.051 are indistinguishable as evidence and are treated as opposites.
The responses in circulation: report exact p-values rather than thresholds, report effect sizes and intervals alongside, pre-register the analysis, and correct for multiple comparisons. Some fields have moved the threshold to 0.005; others argue for abandoning the dichotomy entirely.
Where it goes wrong
Reading p as the probability the null is true. The central error.
Treating "not significant" as "no effect." Check the power.
Choosing the tail after seeing the data.
Running many tests without correction. Bonferroni is crude and better than nothing; false discovery rate control is usually the better tool.
Where p-values come from, and what breaks them
A p-value computed by simulation rather than looked up in a table, then the same test run two hundred times on data with no effect in it at all.
Check yourself
0 of …Answer without scrolling back up.
A p-value is the probability of:
A study reports p = 0.4. What can you conclude?
Why must the choice of one or two tails be made before seeing the data?
Hypothesis Testing and p-values
The p-value is a shaded area under an assumption. Move the observation and watch the area, and the assumption, stay exactly where they were.