Expectation and Variance
Load a die face by face and watch the two numbers that summarise any random quantity move.
Overview
Expectation
The expectation of a random variable is each value it can take, weighted by how likely that value is:
E[X] = sum over x of x * P(X = x)The [mean, mode and median module](mean_mode_and_median.html) covers the mean of a dataset you have. This is the mean of a distribution you have not sampled — what the average would settle to given enough draws.
Load the die above and watch E[X] move. Put all the weight on 6 and it becomes 6. Spread it evenly and it becomes 3.5.
3.5 is the case worth pausing on: a fair die has expectation 3.5, and no die has ever rolled a 3.5. The expectation is a balance point, not a prediction. The distribution would balance on a pin at that value, and that is all the word means.
Expectation and Variance
Worth knowing
E[X] = Σ x · P(x).E[(X − μ)²].Expectation and Variance
The two numbers that summarise a random quantity, and the rules for combining them.
Variance
Expectation alone says nothing about spread. Two very different dice can share one. Variance measures the spread:
Var(X) = E[(X - mu)^2]The expected squared distance from the mean. Squaring is what stops the positive and negative deviations cancelling, and it is why the units come out squared — variance of a height in metres is in metres squared, which is meaningless as a length. Taking the square root gives the standard deviation, back in the original units, which is why that is the number usually reported.
The identity worth memorising is:
Var(X) = E[X^2] - (E[X])^2The mean of the squares minus the square of the mean. It is how variance is computed in one pass over data, and it is also the source of a classic numerical bug: with large values and small variance, the two terms are nearly equal and subtracting them destroys precision. Welford's algorithm exists to avoid exactly this.
Combining them
The rules for expectation are as good as rules get:
E[aX + b] = a E[X] + b
E[X + Y] = E[X] + E[Y] alwaysThat second line holds whether or not X and Y are independent. Linearity of expectation is unconditional, and it is the reason a surprising number of counting arguments are one line long.
Variance is stricter:
Var(aX + b) = a^2 Var(X) note: b vanishes, a is squared
Var(X + Y) = Var(X) + Var(Y) only if X and Y are independentShifting a distribution does not change its spread, which is why b disappears. Scaling by a scales the variance by a², because variance is in squared units.
The independence condition on the sum is where mistakes live. If X and Y move together, the covariance term reappears:
Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y)The [covariance module](covariance_and_correlation.html) covers that term.
Where it shows up
The standard error. The mean of *n* independent draws has variance σ²/n, straight from the rules above. Its square root is σ/√n, which is the whole content of [the sampling distribution](sampling_distributions.html).
Bias-variance decomposition. Expected squared error splits into bias squared plus variance plus irreducible noise, using the identity above.
Portfolio diversification. Combining assets reduces variance only when they are not perfectly correlated — the covariance term again.
Bagging. Averaging *n* models reduces variance by *n* only if their errors are independent, which is why random forests work to decorrelate their trees.
Where it goes wrong
Treating the expectation as a likely value. It need not be attainable at all.
Adding variances of correlated variables. The commonest error in this material.
Forgetting the square on the scale factor. Doubling a variable quadruples its variance.
Assuming a variance exists. Some distributions have none — the Cauchy has neither a finite variance nor a finite mean, and sample averages of Cauchy draws never settle.
Both numbers, worked by hand
Five cards, drawn one at a time with every card equally likely. They are marked 1, 1, 2, 5 and 11.
Build the table. P(x) is how often that value appears out of five, and the last two columns are what the two formulas need:
| x | P(x) | x P(x) | x² P(x) |
|---|---|---|---|
| 1 | 2/5 | 0.4 | 0.4 |
| 2 | 1/5 | 0.4 | 0.8 |
| 5 | 1/5 | 1.0 | 5.0 |
| 11 | 1/5 | 2.2 | 24.2 |
| total | 1 | 4.0 | 30.4 |
The third column sums to the expectation: E[X] = 4.0. Note that no card is marked 4. The fourth sums to E[X²] = 30.4, and the one-pass identity gives the variance straight away:
Var(X) = E[X^2] - (E[X])^2 = 30.4 - 16 = 14.4Now the same number from the definition, E[(X - mu)^2], squaring each distance from 4 and weighting it:
(1 - 4)^2 = 9 weight 2/5 -> 3.6
(2 - 4)^2 = 4 weight 1/5 -> 0.8
(5 - 4)^2 = 1 weight 1/5 -> 0.2
(11 - 4)^2 = 49 weight 1/5 -> 9.8
-------
14.4The same 14.4, by a different route. The standard deviation is √14.4 = 3.79 — almost as large as the mean itself, which is the honest summary of this deck: the average is 4, and the single 11 does most of the work.
The two routes cost differently. The definition needs the mean before it can start, so it is two passes over the data. The identity needs only running totals of x and x², so it is one — which is why it is the one that gets implemented, and why the precision trap above is worth knowing about.
Questions people ask
Why square the deviations instead of taking absolute values? You can, and that gives the mean absolute deviation, which is a perfectly good measure of spread. Squaring wins on grounds of convenience rather than truth: it is differentiable everywhere, it makes the variance of a sum decompose into a sum of variances, and it produces the identity above. Absolute values give none of those, which is why almost every result in this material is stated for variance.
Does the expectation have to be a value the variable can take? No, and usually it is not. A fair die averages 3.5; the deck above averages 4 without containing a 4. Expectation is a balance point of the distribution, not a forecast of any single draw.
Why does sample variance divide by n − 1? Because the sample mean is itself estimated from the same data, and it sits closer to the data than the true mean does — it is the value that *minimises* the sum of squared deviations, so squared distances measured from it are systematically too small. Dividing by n - 1 instead of n corrects that bias exactly. With the true mean known, n would be right.
When can I add variances? Only when the variables are independent, or at least uncorrelated. Otherwise the covariance term is genuinely there and dropping it will mislead you, almost always in the direction of thinking you have less risk than you do.
Variance or standard deviation in a report? Standard deviation, nearly always, because it is in the same units as the data and can be compared to the mean. Variance is the quantity the algebra is easier in, which is why it appears in the derivations and rarely in the conclusions.
A bet worth taking that usually loses
Expectation and variance computed from the definition, then measured over 400,000 plays. The two numbers say different things, and both matter.
Check yourself
0 of …Answer without scrolling back up.
A fair die has E[X] = 3.5. What does that tell you?
When does Var(X + Y) = Var(X) + Var(Y)?
Why is Var(X) = E[X^2] - (E[X])^2 numerically dangerous?
Expectation and Variance
The [mean, mode and median module](mean_mode_and_median.html) covers the mean of a dataset you have. This is the mean of a distribution you have not sampled — what the average would settle to given enough draws.