Partial Dependence, ICE and SHAP
What the model does as one feature moves, and why the average of many individuals can describe none of them.
Overview
Partial dependence
A partial dependence plot answers: as this feature moves across its range, what happens to the prediction on average?
The procedure is mechanical. Take the dataset. Set the feature to some value for *every* row, leaving all other columns untouched. Predict, and average. Repeat across the feature's range, and plot.
It is model-agnostic and easy to read, which is why it is everywhere.
Partial Dependence, ICE and SHAP
Worth knowing
Partial Dependence, ICE and SHAP
Three ways to ask what a model is doing, and the trap in the most popular one.
The trap
Leave the interaction control on and look at the thick line. It is nearly flat, and the readout gives its slope as close to zero. Read on its own, it says this feature does not matter.
Now the ICE curves underneath tell a completely different story. Every individual curve has a steep slope — the feature matters enormously to every single row. It is just that it slopes up for one group and down for the other, and averaging them cancels.
This is not a contrived case. Any feature whose effect reverses by subgroup behaves like this: a treatment that helps one population and harms another, a price change that attracts some customers and repels others.
A flat partial dependence curve does not mean the feature is unimportant. It means the average effect is zero, which is a much weaker statement.
Switch the interaction off and the curves stack into a single shape. Now the average describes each individual well, and the partial dependence line is trustworthy.
ICE curves
Individual Conditional Expectation curves are partial dependence without the averaging: one line per row, showing what the model predicts for *that* row as the feature sweeps.
They cost nothing extra, since partial dependence computes them and then throws them away. Plotting them is the standard defence against the trap above, and the diagnostic is simple: if the ICE curves are parallel, the partial dependence line is a fair summary. If they fan out or cross, it is not, and there is an interaction worth finding.
The other assumption
Partial dependence sets a feature to a value for every row regardless of whether that combination is possible. Sweeping age from 18 to 80 across a dataset creates rows with age 18 and forty years of work experience.
The model is asked to predict for combinations that never occur, and its answer there is extrapolation with nothing to anchor it. When features are strongly correlated, part of a partial dependence curve is describing regions of input space the model has never seen.
Accumulated Local Effects plots address this by only varying a feature within the range of values that actually co-occur with the other features' values.
SHAP
SHAP answers a different question. Partial dependence and ICE describe the model across a range; SHAP explains one prediction.
For a given row, it assigns each feature a number saying how much that feature pushed the prediction away from the base value — the average prediction over the dataset. The numbers sum exactly to the difference:
prediction = base value + sum of every feature's SHAP valueThat additivity is what makes it useful for an individual explanation: it accounts for the whole prediction, with nothing unattributed.
The values come from Shapley values in cooperative game theory, which distribute credit for a joint outcome fairly among contributors. Computing them exactly requires evaluating every subset of features, which is exponential, so implementations approximate — TreeSHAP exactly and quickly for trees, KernelSHAP by sampling for anything else.
| Answers | Scope | |
|---|---|---|
| Partial dependence | how the prediction moves with a feature, on average | global |
| ICE | the same, per row | per row |
| SHAP | how much each feature contributed here | one prediction |
| [Permutation importance](permutation_importance.html) | how much accuracy depends on a feature | global |
SHAP adds up, exactly
What makes a SHAP explanation trustworthy for one prediction is that the parts sum to the whole. Each feature gets a value saying how far it pushed this prediction from the base value — the average prediction over the dataset — and those pushes reconstruct the prediction exactly:
| term | value |
|---|---|
| base value | 0.30 |
| age | +0.15 |
| income | +0.10 |
| region | −0.05 |
| prediction | 0.50 |
0.30 + 0.15 + 0.10 - 0.05 = 0.50, with nothing unattributed — that additivity is the property partial dependence and ICE do not have, and it is what lets SHAP explain a single row rather than the model's average behaviour. The values come from Shapley values in cooperative game theory, which distribute credit for a joint outcome fairly, and computing them exactly means evaluating every subset of features — exponential, which is why TreeSHAP (exact, fast, for trees) and KernelSHAP (sampled, for anything) exist. The caution is that additivity is not causality: a model using postcode as a proxy for income will show postcode contributing, which is a true statement about the model and not about the world.
The flat-curve trap is worth restating because it is silent: a partial dependence line near zero can hide individual effects that are large but cancel — a feature that slopes up for one subgroup and down for another averages to nothing, while mattering enormously to every row. Plotting the ICE curves underneath is the standard defence: parallel curves mean the average is a fair summary, curves that fan out or cross mean there is an interaction and the average is lying. Partial dependence also quietly assumes the feature can be swept independently of the others, so with strongly correlated features part of the curve is the model extrapolating into combinations that never occur — which is what Accumulated Local Effects plots were designed to avoid.
Where it goes wrong
Reading a flat PD curve as "unimportant". Plot the ICE curves.
Ignoring correlated features. Part of the curve may be extrapolation.
Treating SHAP as causal. It attributes a prediction, not an outcome. A model that uses postcode as a proxy for income will show postcode contributing, and that is a fact about the model.
Averaging SHAP values into a global importance and stopping there. The mean absolute SHAP value is a reasonable global summary, and it discards the direction and the interactions that made SHAP worth computing.
One chart for the model, one number per row
Partial dependence describes the model's average behaviour; a Shapley value explains a single prediction. Both are computed from scratch here, so neither is a black box.
Check yourself
0 of …Answer without scrolling back up.
A partial dependence curve is flat. What can you conclude?
What does the additivity of SHAP values give you?
Why can part of a partial dependence curve be untrustworthy when features are correlated?
Partial Dependence, ICE and SHAP
The procedure is mechanical. Take the dataset. Set the feature to some value for *every* row, leaving all other columns untouched. Predict, and average. Repeat across the feature's range, and plot.