Home/MLOps/Concept and Data Drift

Concept and Data Drift

MLOpsBy AI.CruzUpdated 1 October 2026

A model trained on yesterday's data makes predictions for today. When the world shifts, performance degrades — silently, until it becomes a problem.

Drift Simulator

Drift Amount

PSI Score

0.000

Alert Level

No Drift

Distribution Shift

PSI Thresholds

PSI < 0.10

No significant drift

0.10 ≤ PSI < 0.20

Moderate — investigate

PSI ≥ 0.20

Significant — retrain

Concept and Data Drift

Why models fail silently — and how to measure it before it becomes a disaster.

Why models degrade over time

A deployed model is a snapshot of the world at training time. The world is not static. Customer behaviour shifts, product lines change, economic conditions fluctuate, fraud tactics evolve. When the gap between training-time reality and serving-time reality grows large enough, model performance drops — often without any code change, any engineer error, or any detectable system failure.

This category of failure is called model drift. It is the primary reason production ML systems require monitoring, not just deployment.

Data drift (covariate shift)

Data drift occurs when the distribution of input features P(X) changes between training and serving. The true relationship between features and label remains the same — the feature distributions themselves have shifted.

Examples: a credit scoring model trained on salary distributions from 2020 sees 2025 salary distributions shifted by inflation. A fraud detection model trained on transaction amounts during normal times sees pandemic-era transaction patterns. The model's learned boundaries are calibrated for a distribution it no longer sees.

Data drift is detectable without labels: you can compare serving feature distributions to training distributions at any time.

Concept drift

Concept drift occurs when the relationship between features and labels P(Y|X) changes. The inputs look the same but the correct output has changed.

Examples: the features that predicted spam email in 2015 no longer predict spam in 2025 because spammers adapted. The word patterns that indicated fraud evolved. A sentiment model trained before a major event may assign the wrong polarity to new linguistic patterns.

Concept drift cannot be detected without labels. You need ground truth outcomes (which may be delayed — a loan default takes months to manifest) to measure it.

Four types of concept drift

Sudden drift — an abrupt change in the data-generating process. A regulatory change makes previously-valid features illegal. A competitor launches and changes user behaviour overnight.
Gradual drift — slow, continuous change. Consumer preferences shift over months. Language evolves. Climate trends shift seasonal baselines.
Incremental drift — step-wise changes. A series of small policy changes, each causing a small distribution shift.
Recurring/seasonal drift — patterns that repeat. A demand forecasting model sees December every year; a model that forgets seasonal patterns will drift predictably.

Population Stability Index (PSI)

PSI is the standard industry metric for quantifying data drift. It compares a reference distribution (typically training data) to a current distribution (recent serving data) by bucketing both into bins and comparing proportions:

PSI = ∑ (p_current − p_reference) × ln(p_current / p_reference)

PSI interpretation: below 0.10 no meaningful drift; 0.10–0.20 moderate shift worth investigating; above 0.20 significant shift, consider retraining.

PSI was originally developed in credit scoring (banking) to monitor scorecard stability. It is now standard across industries.

KL divergence

KL divergence (Kullback-Leibler) measures information loss when approximating one distribution with another:

KL(P || Q) = ∑ P(x) × log(P(x) / Q(x))

PSI is actually symmetric KL divergence: PSI = KL(current || reference) + KL(reference || current). This symmetry makes PSI easier to interpret than directional KL.

Other tests: Kolmogorov-Smirnov (KS) test for continuous distributions, chi-squared for categorical features, Wasserstein distance for high-dimensional data.

Detection strategies

Strategy	Works for	Requires labels?
PSI on features	Data drift	No
Prediction distribution shift	Proxy for concept drift	No
Performance metrics (accuracy, AUC)	Concept drift	Yes (delayed)
CUSUM / ADWIN	Statistical change detection	Depends

Retraining triggers

Three trigger strategies in practice:

Schedule-based — retrain weekly or monthly regardless of drift. Simple, predictable, but wasteful when drift is slow and costly when drift is fast.
Threshold-based — retrain when PSI exceeds 0.20, or when accuracy drops below a floor. Requires monitoring infrastructure but is more responsive.
Online learning — continuously update model weights with new data. Appropriate for high-velocity, slowly-drifting systems. Requires careful regularisation to prevent catastrophic forgetting.

What to remember

Data drift = P(X) shifts. Detectable without labels. Use PSI.
Concept drift = P(Y|X) shifts. Requires labels to detect directly.
PSI < 0.10 = stable; 0.10–0.20 = investigate; > 0.20 = retrain.
Monitor both feature distributions and prediction distributions in production.

PSI Drift Detection

import numpy as np

# Population Stability Index (PSI) calculation
# PSI = sum((actual - expected) * ln(actual / expected))
np.random.seed(42)

def compute_psi(reference, current, bins=10):
    """Compute PSI between reference and current distributions."""
    combined = np.concatenate([reference, current])
    breakpoints = np.percentile(reference, np.linspace(0, 100, bins + 1))
    breakpoints[0] = -np.inf
    breakpoints[-1] = np.inf

    ref_counts = np.histogram(reference, bins=breakpoints)[0]
    cur_counts = np.histogram(current, bins=breakpoints)[0]

    # Avoid division by zero with small epsilon
    eps = 1e-6
    ref_pct = (ref_counts + eps) / (len(reference) + eps * bins)
    cur_pct = (cur_counts + eps) / (len(current) + eps * bins)

    psi_values = (cur_pct - ref_pct) * np.log(cur_pct / ref_pct)
    return np.sum(psi_values)

# Simulate reference distribution (training data)
reference = np.random.normal(loc=50, scale=10, size=5000)

print("=== PSI Drift Monitor ===")
print("Reference: N(50, 10), n=5000")
print()

# Test different drift scenarios
scenarios = [
    ("No drift",      np.random.normal(50, 10, 1000)),
    ("Minor drift",   np.random.normal(52, 10, 1000)),
    ("Moderate drift",np.random.normal(55, 11, 1000)),
    ("Major drift",   np.random.normal(60, 12, 1000)),
    ("Severe drift",  np.random.normal(70, 15, 1000)),
]

for name, current in scenarios:
    psi = compute_psi(reference, current)
    if psi < 0.10:
        status = "STABLE"
    elif psi < 0.20:
        status = "INVESTIGATE"
    else:
        status = "RETRAIN"
    print(name + ": PSI=" + str(round(psi, 4)) + " -> " + status)

Recall check

0 of …

Say the answer out loud before you reveal it.

What is the difference between data drift and concept drift?
What does PSI measure and what are the thresholds?
Why can't you detect concept drift without labels?
What are the three retraining trigger strategies in MLOps?

Cheat sheet

Concept and Data Drift

Data drift = P(X) shifts (detectable without labels). Concept drift = P(Y|X) shifts (needs labels). PSI quantifies feature drift: <0.10 stable, 0.10-0.20 investigate, ≥0.20 retrain. Monitor both feature distributions and prediction distributions in production.

MLOPS · AI.Cruz.in/mlops/concept_and_data_drift.html