Stacking and Voting Ensembles
Combining different models rather than many copies of one, and the case where a majority vote makes things worse.
Overview
Different from bagging and boosting
[Bagging](random_forest.html) and [boosting](gradient_boosting.html) build many copies of one kind of model — hundreds of trees — and combine them.
Voting and stacking combine models of *different* kinds: a logistic regression, a gradient-boosted tree and a neural network, say. The hope is that models with different inductive biases fail on different rows, so their errors partly cancel.
That hope is a condition, not a guarantee, and this page is largely about the condition failing.
Stacking and Voting Ensembles
Worth knowing
Stacking and Voting Ensembles
Combining unlike models, when it helps, and a live demonstration of it hurting.
Voting
Hard voting takes the majority label. Soft voting averages predicted probabilities and thresholds the mean, which is generally better because it uses confidence rather than discarding it — and which requires [calibrated](probability_calibration.html) probabilities to be meaningful.
Every member counts equally. That is the strength when the members are comparable and the weakness when they are not.
Look at the readout with the default settings. Three learners score roughly 62%, 66% and 90%, and the majority vote scores about 77% — worse than the best member alone. Two weak learners outvote a strong one whenever they agree, and they agree often enough to drag the result down.
This is not a contrived arrangement. It is what happens whenever an ensemble is assembled without checking whether the members are of comparable quality, and it is the reason "just ensemble it" is bad advice.
Stacking
Stacking replaces the fixed rule with a learned one. Train the base models, take their predictions as features, and fit a second model — the meta-learner — on those.
The meta-learner discovers what a vote cannot: which model to trust, and when. The learned weights are in the readout, and the strong learner attracts most of the weight while a weak one can go slightly negative — meaning the meta-learner has found that model to be worse than uninformative in some regions.
With the defaults, stacking matches the best single learner rather than beating it. That is an honest and common outcome, and it is still a gain over the vote: stacking recovered the best model automatically, where voting destroyed it.
Move the thresholds so that the three learners are closer in quality, and the picture changes — the vote catches up, because its assumption is finally true.
Getting stacking right
The essential detail is how the meta-learner's training data is produced.
Training the base models and then feeding their predictions on the same data to the meta-learner leaks badly. A model that overfits its training set produces suspiciously good predictions there, so the meta-learner learns to trust the most overfitted member.
The fix is cross-validated predictions: split the data into folds, and for each fold predict with base models trained on the other folds. Every prediction the meta-learner sees is then out-of-sample. This is what scikit-learn's StackingClassifier does, and it is why stacking costs roughly *k* times the training of its members.
A second convention: keep the meta-learner simple. Logistic regression is the standard choice. It has few base predictions to work with, and a flexible meta-learner overfits them readily.
When an ensemble is worth it
Members must be comparable in quality. Otherwise voting hurts, as above.
Members must make different mistakes. Three models that fail on the same rows combine into one model that fails on those rows. Checking the correlation of their errors is the diagnostic, and it is worth doing before building anything.
The gain must justify the cost. An ensemble multiplies training time, inference time, memory and the number of things that can break in production. A one-point gain on a leaderboard is worth it; a one-point gain in a service usually is not.
When the majority is wrong, in numbers
Majority voting only helps when the members are comparable, and the arithmetic shows why. For three independent classifiers the vote is correct when at least two are, P = Σ over the majority outcomes:
| three members | majority-vote accuracy |
|---|---|
| 0.62, 0.66, 0.90 | 0.83 — below the 0.90 member |
| 0.80, 0.80, 0.80 | 0.90 — above any member |
Three equal 80% models vote to 90%, which is the ensemble win everyone hopes for: independent errors partly cancel. But bolt a 90% model to two mediocre ones and the vote drops it to 83%, because whenever the two weak members happen to agree they outvote the strong one, and two coin-ish models agree wrongly often enough to matter. That is voting's built-in assumption — equal competence — failing out loud, and it is why "just ensemble it" is bad advice without checking the members are comparable. Stacking fixes exactly this by learning the weights instead of fixing them: the meta-learner hands most of the weight to the strong member and can drive a weak one negative, recovering the best model where the vote destroyed it. (Real classifiers are not independent, so the true numbers shift with error correlation — which is itself the thing to measure before combining anything.)
Getting stacking right turns on one detail: the meta-learner must be trained on *out-of-sample* base predictions. Feed it predictions the base models made on their own training data and it learns to trust whichever member overfitted most, because an overfitted model looks suspiciously accurate there. The fix is cross-validated predictions — for each fold, predict with base models trained on the other folds — which is why stacking costs roughly k times the training of its members. Keep the meta-learner simple, too; logistic regression is the standard choice, because it has few base predictions to work from and a flexible meta-learner overfits them readily.
Where it goes wrong
Ensembling a strong model with weak ones by vote. Demonstrated above.
Fitting the meta-learner on in-sample predictions. It learns to trust the most overfitted base model.
A complex meta-learner. Few features, plenty of opportunity to overfit.
Assuming diversity. Two gradient-boosted models with different seeds are not diverse; check whether their errors actually differ.
Ensembling models that make different mistakes
Combining models only helps when they are wrong about different rows. This measures that disagreement first, then shows what voting and stacking do with it.
Check yourself
0 of …Answer without scrolling back up.
Why can a majority vote score worse than its best member?
Why must the meta-learner be trained on cross-validated predictions?
What condition must hold for an ensemble to help at all?
Stacking and Voting Ensembles
[Bagging](random_forest.html) and [boosting](gradient_boosting.html) build many copies of one kind of model — hundreds of trees — and combine them.