A column that was computed from the answer
The purest form: a feature that would not exist at prediction time.
The same row on both sides
Random data, nothing to learn, and a perfect score - because every test row has a twin in the training set.
One entity, several rows, both sides of the split
The label is random per person. Anything above chance is the model recognising people rather than learning anything.
Shuffling a series that has an order
Interpolating between known points is a much easier job than the one the model will actually be given.
An encoding that used the target
Target encoding is powerful and leaks by default, because each row helps compute the number that describes it.
Noticing it
There is no detector. There is a baseline, a suspicion, and two questions.
Cross-validation will not save you
Worth stating plainly, because cross-validation is otherwise the answer to almost everything on this track.
Cross-validation protects against one specific thing: a score that depends on which rows happened to be held out. If the leak is in the *data* — a column derived from the answer — then every fold contains it, every fold is inflated, and the mean of five inflated numbers is an inflated number with a reassuring standard deviation.
The first editor shows it: an honest 0.643 becomes 1.000, across all five folds, with no warning of any kind.
The five shapes
Target leakage. A feature that encodes the answer, usually because it was recorded after the outcome. A case_closed_date predicting whether a case closed. A discount_given predicting churn, when the discount was offered *because* the customer was leaving. A diagnosis code entered at the same time as the diagnosis. These are the hardest to spot because the column is legitimately in the database and nothing about its name says when it was written.
Preprocessing before the split. A scaler, imputer, encoder or feature selector fitted on all the data has seen the test rows. The cross-validation module measured this: feature selection outside the folds reported 0.825 on pure noise.
Duplicate rows. After a bad join, a re-import, or genuinely repeated records. A duplicate split across train and test means the model has the answer memorised. The second editor scores 1.000 on random data for exactly this reason.
Group leakage. Several rows describing one entity — a patient, a customer, a device, a document — landing on both sides. The model learns to recognise the entity rather than the pattern. The third editor scores 0.87 where the honest answer is chance.
Temporal leakage. Shuffling data that has an order, so the training set contains the future. The fourth editor goes from R² of 0.977 to −2.526 when the split respects time, and the second number is the one describing the job the model would actually have.
Target encoding deserves its own warning
It is the most useful high-cardinality encoding and the most reliable way to leak.
Replacing a category with the mean target for that category is compact, powerful, and computed from y. Do it over the whole dataset and every row has contributed to the number that describes it. With a hundred rows per category the contribution is small; with six, as in the fifth editor, it is a sixth — and a feature built from random labels predicts those labels at 0.71.
The fix is to compute the encoding inside the folds, with the row's own contribution excluded — which is what scikit-learn's TargetEncoder does, using an internal cross-fitting scheme. Hand-rolled target encoding is almost always leaky, and the leak is proportional to how rare the categories are, which is exactly when the encoding is most tempting.
How to notice
There is no detector. There is a set of habits.
Be suspicious of a good score. Not sceptical of a good model — suspicious of a *surprisingly* good one. If the problem is hard and the number is excellent, the first hypothesis should be a leak, not a breakthrough. Comparing against a dummy makes the size of the claim visible.
Look at feature importances. A single feature dominating, especially one nobody expected to matter, is the classic signature. Ask what it is, when it is written, and by what process.
Ask the timing question for every column. *Would this value be known, in this form, at the moment the prediction has to be made?* This is the single most effective check available, and it requires domain knowledge rather than code. A column that is only populated after the event is a leak regardless of how innocent it looks.
Ask who produced the row. If several rows can come from one entity, group the split. If rows can be duplicated, check. If the data has a timestamp, respect it.
Test on genuinely later data where possible. A model that scores well in cross-validation and badly on the next month is telling you something a random split could not.
The habits that prevent it
Split first, before anything that learns. Put every transformer in a pipeline so it is refitted inside each fold. Use GroupKFold when rows share an entity and TimeSeriesSplit when they are ordered. Check for duplicates before splitting. Hold out a final test set and touch it once.
None of these is expensive, and together they close every mechanical route. What they cannot close is target leakage, because that is a fact about the meaning of a column rather than about the code — which is why the timing question has to be asked of every feature, by someone who knows what the columns mean.
The famous cases
Leakage is not a beginner's mistake. It has invalidated published research, competition results and deployed systems, and the examples are worth knowing because they are all plausible.
A widely cited early result on detecting a disease from chest scans turned out to be partly detecting *which hospital* the scan came from — sicker patients were concentrated at one site, and the scanner left a signature in the image. The model was excellent at the task it was actually solving and useless at the one it was believed to be solving.
Competition datasets have repeatedly leaked through row ordering, where the file happened to be sorted by the target, and through id columns that correlated with when a record was created. Both give a model something real to learn that has nothing to do with the problem.
A recurring one in industry: a churn model trained on a table where the "customer contacted support" flag was updated during the cancellation call. The feature was genuinely predictive and genuinely unavailable at the moment a prediction was needed.
The common thread is that none of these look like errors in the code. Every one is a fact about how the data came to exist, which is why reading the code more carefully never finds them and asking where a column comes from usually does.
The cost of being too careful
The opposite failure exists and is worth naming, because leakage anxiety can become its own problem.
Excluding every feature that has any relationship with the target removes the signal along with the leak. A feature that is predictive because it is genuinely causally upstream is exactly what you want, and dropping it because it is "too predictive" makes the model worse for no reason.
Grouping a split when the rows are genuinely independent throws away resolution and gives an unnecessarily pessimistic estimate. Refusing to use any aggregate feature because target encoding can leak rules out a whole class of useful ones that are computed correctly.
The distinction is not how predictive a feature is. It is whether the value would exist, in that form, at the moment the prediction has to be made. A feature that passes that test is allowed to be as predictive as it likes.
Is a highly predictive feature always suspicious? No. It is worth one question - would this be known at prediction time - and if the answer is yes, use it.
How do I check for duplicates? df.duplicated().sum() before splitting, and consider near-duplicates too: rows differing only in an id or a timestamp are functionally the same row.
Does leakage matter if I only care about ranking? Yes. A leaked feature ranks well in evaluation and is absent in production, so the ranking you deploy is not the one you measured.
Can I fix a leak by removing the column? Usually, and check what else was derived from it. Aggregates, ratios and encodings built on a leaky column carry the leak with them.
A checklist before you trust a number
Six questions, in the order they are cheapest to answer. Running through them takes a few minutes and catches the large majority of leaks.
Did anything fit before the split? Scaler, imputer, encoder, selector, vectoriser. If it learned from data and it ran before train_test_split, the number is compromised. The fix is a pipeline.
Are there duplicate rows? Count them, including near-duplicates that differ only in an id or a timestamp.
Can one entity produce several rows? If so, the split must group them. This is the one people most often get wrong on data that looks tabular and independent.
Does the data have an order? Timestamps, sequence numbers, anything where later rows could not have informed earlier ones. Then the split is forward-only.
For every feature: would it exist at prediction time? Asked column by column, with someone who knows the source system. This is the slow one and the one that catches target leakage, which nothing else will.
Is the score plausible? Compared against a dummy, against published results on similar problems, and against what a domain expert would guess. Surprise is the signal.
A model that survives all six is not guaranteed honest, but the remaining routes are narrow and unusual. A model that has not been through them has an unknown probability of being wrong in the most expensive direction.
What if I find a leak after deploying? Retrain without the leaking feature and re-measure honestly before deciding whether the model is still worth running. The old score should be treated as never having existed rather than as a target to recover.
Is leakage always accidental? Mostly, and there is a well-known variety that is not: tuning against the test set until the number is satisfactory. That is leakage through the researcher rather than through the data, and it produces the same inflated result.
Things to try
- Run the first editor. 0.643 becomes 1.000 from one extra column, and every fold agrees.
- Vary the duplication. In the second editor, duplicate only a quarter of the rows and see how much of the inflation survives.
- Change the group size. In the third editor, set
per_person=2 and watch the gap narrow — the more rows per entity, the worse the leak. - Make the categories rarer. In the fifth editor, raise
n_cats to 150 and watch the leaked score climb.
Where this leaves you
The failure that improves your metrics. Cross-validation cannot see it, a good score is evidence for it rather than against it, and the only reliable defence is asking, of every column, whether it would exist at the moment the prediction is needed.