Outliers and Influence
Drag one point around and watch a least-squares line follow it. Twenty-four rows say one thing; the twenty-fifth overrules them.
Overview
Drag it
The chart holds twenty-four points along a clear upward line, plus one highlighted point you can move. The dashed line is the least-squares fit without it; the solid line is the fit with it.
Drag the point straight up. The solid line rotates. Drag it to the far right and up, and the line follows it much further. The readout reports both slopes and the percentage difference between them, and it does not take much to change the answer by tens of per cent.
Twenty-four rows are saying one thing. The twenty-fifth overrules them.
Outliers and Influence
Drag on the chart to move the highlighted point.
Worth knowing
Outliers and Influence
One row out of twenty-five, and the reason it can decide your answer.
Why squaring does this
Least squares chooses the line minimising the sum of squared residuals. Squaring is what makes a single point so powerful: a point twice as far from the line contributes four times the penalty, and one ten times as far contributes a hundred times.
The fit is therefore willing to move a long way to reduce one large residual, even at the cost of small increases across every other point. That is not a bug in the implementation; it is what the objective function asks for.
Leverage: where it sits matters as much as how far off it is
Drag the point up while keeping it near the middle of the x range, then drag it up while keeping it at the far right. The second position moves the line much more.
That difference is leverage. A regression line pivots roughly about the mean of x, so a point near that mean is close to the pivot and has little turning effect no matter how far off it is vertically. A point at the extreme of x sits at the end of a long lever.
The distinction is worth having in three words:
- Outlier — unusual in y, a large residual.
- High leverage — unusual in x, far from the pivot.
- Influential — both, so removing it changes the model. This is the one that matters, and Cook's distance is the standard way to measure it.
A high-leverage point that happens to lie on the trend is harmless. A moderate outlier in the middle is harmless. The combination is not.
Before you remove anything
The temptation is to delete the point. Resist it long enough to ask which of three things it is.
An error. A misplaced decimal, a sensor fault, a unit mix-up, a placeholder like 999 or −1 meaning "unknown". Fix it or remove it, and record that you did.
A different population. The row is correct but does not belong to the question — a wholesale order in a table of retail purchases. Exclude it and say so, or model the groups separately.
A real, extreme member of the population. The row is correct and belongs. This one you keep. Removing genuine extremes because they are inconvenient is how a result stops describing the world, and it is much more common than deliberate fraud.
If you cannot tell which, report the analysis both ways. A conclusion that survives the point and a conclusion that depends on it are different findings, and the reader is entitled to know which they have.
Ways to be less fragile
Robust loss. Huber loss is squared for small residuals and linear for large ones, so a distant point pulls hard but not quadratically hard. RANSAC fits on random subsets and keeps the consensus.
Quantile regression. Models the median rather than the mean, and the median does not move when one point is dragged to infinity.
Winsorising. Clip values at, say, the 1st and 99th percentiles. Keeps the row, caps its influence.
Use a model that does not care. Trees split on order, not on distance, so a value of a million and a value of a thousand are the same split if nothing lies between them.
Transform. A log transform on a skewed positive column often turns "outliers" into ordinary members of a normal-shaped distribution, which is usually a sign the column was on the wrong scale.
Why position matters as much as distance
The reason one far-off point wrecks a fit and another leaves it alone is leverage, and squared error is what makes it bite. Least squares minimises the *sum of squared* residuals, so a point twice as far off contributes four times the penalty and one ten times as far contributes a hundred times — the line will travel a long way to reduce a single large residual. But a regression line pivots roughly about the mean of x, so how far a point sits along x decides how much turning force its vertical error applies:
| the odd point | effect on the slope |
|---|---|
| large residual, near the middle of x | small — close to the pivot |
| large residual, at the extreme of x | large — the end of a long lever |
That gives three distinct things worth separating: an outlier is unusual in y, high leverage is unusual in x, and only their combination — influential — actually moves the model, which is what Cook's distance measures. A high-leverage point sitting on the trend is harmless; a mid-range outlier is harmless; the far-right outlier is the one that quietly rewrites the answer. Which is also why "delete anything beyond three standard deviations" is self-defeating: the outlier has already inflated the standard deviation the rule depends on.
Where it goes wrong
Deleting by rule. "Beyond three standard deviations" uses the standard deviation, which the outlier itself has already inflated.
Treating placeholders as data. 999 for unknown age, −1 for missing. These are not outliers; they are missing values wearing a number.
Removing points until the model fits. At that stage the model describes the selection, not the phenomenon.
Checking only y. High-leverage points can be unremarkable in y and still dominate the fit.
An outlier and an influential point are not the same thing
One far-out row can leave a fit untouched while another moves it completely. The difference is leverage, and it is computable.
Check yourself
0 of …Answer without scrolling back up.
Why does least squares react so strongly to a single distant point?
What makes a point 'influential' rather than merely an outlier?
What is wrong with removing anything beyond three standard deviations?
Outliers and Influence
The chart holds twenty-four points along a clear upward line, plus one highlighted point you can move. The dashed line is the least-squares fit without it; the solid line is the fit with it.