Jensen's Inequality
The average of a curve is not the curve of the average. The gap has a direction, a size, and a great many consequences.
Jensen's Inequality
Worth knowing
E[f(X)] ≥ f(E[X]). For concave, the inequality reverses.Jensen's Inequality
Why averaging before and after a curve gives different answers, and what that difference is used for.
The statement
For a convex function f and a random variable X:
E[f(X)] >= f(E[X])Transform first and average, and you get at least as much as averaging first and transforming. For a concave function the inequality reverses.
Why it is obvious once seen
Take a distribution that puts half its weight at each of two points — the two grey dots in the visualisation.
f(E[X]) is the curve evaluated at the midpoint: the orange dot on the curve.
E[f(X)] is the average of the two heights: the midpoint of the chord, the lighter dot above.
Convex means the chord lies above the curve. So the chord's midpoint is above the curve's point, which is the inequality, drawn.
Two controls change the gap. Curvature: flatten the function toward a straight line and the chord lies on the curve, so the gap vanishes — for a linear f, expectation passes straight through. Spread: shrink the distribution to a point and the two dots merge, and the gap vanishes again.
The gap is zero exactly when the function is linear or the variable is constant, and grows with both.
Consequences worth having
The mean of ratios is not the ratio of means. 1/x is convex on the positives, so E[1/X] >= 1/E[X]. Averaging speeds in miles per hour to get average pace is wrong in a specific, predictable direction, and the same trap appears in averaging rates, ratios and per-unit costs throughout applied work.
AM-GM. Applying Jensen to the concave logarithm gives that the arithmetic mean is at least the geometric mean, immediately.
Log-loss and calibration. -log is convex, so averaging log-losses penalises confident mistakes far more than the raw error rate does. That is a design choice, and Jensen is why it works.
Portfolio returns. Compounding is multiplicative, so the geometric mean is what you keep and the arithmetic mean is what gets advertised. Volatility drag is Jensen's gap.
The one that matters most
Jensen is why EM and variational inference exist.
Both want to maximise a log-likelihood containing a sum inside a logarithm — log sum_z p(x, z) — which does not decompose and cannot be optimised directly.
The move is to write that sum as an expectation, then use Jensen on the concave log in the other direction:
log E[ ... ] >= E[ log ... ]The right-hand side is a lower bound on the thing you wanted, it decomposes into terms you can differentiate, and maximising it cannot decrease the true objective.
That bound is the ELBO — the evidence lower bound — and it is the objective a variational autoencoder trains on. EM alternates between tightening the bound and maximising it, which is exactly the two steps in [the Gaussian mixture module](gaussian_mixture_models.html).
The gap between the bound and the truth is the KL divergence between the approximate posterior and the real one, which ties Jensen directly to [cross-entropy and KL](cross_entropy_and_kl_divergence.html): non-negativity of KL *is* Jensen applied to the log.
Where it goes wrong
Averaging a transformed quantity and reporting it as the transform of the average. Log-scale averages, rates and ratios all bite here.
Getting the direction wrong. Convex up, concave down. log and sqrt are concave; exp, x^2 and 1/x (on positives) are convex.
Assuming the bound is tight. The ELBO can sit far below the true likelihood, and a rising ELBO does not prove the likelihood rose by as much.
Forgetting that equality needs linearity or a constant. Nothing else gives it.
The average speed trap, worked
Drive 60 km at 30 km/h, then another 60 km at 60 km/h. What was the average speed?
Almost everyone answers 45. Work it out instead:
first leg 60 km at 30 km/h -> 2 hours
second leg 60 km at 60 km/h -> 1 hour
--------
120 km in 3 hours -> 40 km/h40, not 45. The answer is dragged toward the slower speed, because you spend twice as long travelling at it. The correct average here is the harmonic mean, 2 / (1/30 + 1/60) = 40.
Jensen says this had to happen, and says which direction. Time per kilometre is 1/X, and 1/x is convex on the positives, so:
E[1/X] >= 1/E[X]
E[1/X] = (1/30 + 1/60)/2 = 0.02500 hours per km
1/E[X] = 1/45 = 0.02222 hours per kmThe time you actually spend is governed by the left-hand side, and it is bigger. Over 120 km: 3.00 hours actual against 2.67 predicted by the naive average. The trip is always at least as slow as the arithmetic mean suggests, and exactly as slow only when both speeds are equal — the equality case, where the distribution collapses to a point.
The same shape appears wherever a rate is averaged: cost per unit, latency per request, price-to-earnings across a portfolio. Averaging a ratio and taking the ratio of averages are different operations, and Jensen tells you in advance which one comes out larger, without computing either.
Questions people ask
How do I remember which way it points? Picture the chord. Convex means the curve bends upward and every chord lies above it, so the average of the heights (the chord) beats the height of the average (the curve): E[f(X)] >= f(E[X]). Concave flips the picture and the inequality. If you can recall that x² is convex, one example settles it — the mean of squares exceeds the square of the mean, which is just the variance being non-negative.
When is it an equality? In exactly two cases: f is linear over the range the variable actually reaches, or the variable is a constant. Anything else and the gap is strictly positive. This is why linear transformations pass cleanly through an expectation and nothing else does.
Is AM-GM really the same statement? Yes. Apply Jensen to log, which is concave, and you get log of the mean at least the mean of the logs; exponentiate and that is the arithmetic mean dominating the geometric mean. The inequality you may have proved by induction at school is one substitution away.
Does it apply to medians too? No. Jensen is a statement about expectations, and the median passes through any *monotone* function untouched — median(f(X)) = f(median(X)) for increasing f, convex or not. That difference is one practical reason to report a median when a quantity has been through a nonlinear transformation.
Where will this actually bite me? Wherever you average something that was already a rate or a log. Averaging log-probabilities and exponentiating gives the geometric mean, not the arithmetic one — which is what perplexity is, and why it is not the average probability.
The average of the function is not the function of the average
Three concrete cases where swapping those two operations changes the answer, including the one that quietly costs money.
Check yourself
0 of …Answer without scrolling back up.
For a convex f, which is larger?
When is the gap exactly zero?
How does Jensen make variational inference possible?
Jensen's Inequality
Transform first and average, and you get at least as much as averaging first and transforming. For a concave function the inequality reverses.