Data & Concept Drift Detection
Detect when your production data diverges from training data before it hurts model performance
Two Types of Drift
Models degrade in production for two reasons, often confused with each other: **Data drift** (covariate shift): The input distribution p(X) changes but the relationship p(Y|X) stays the same. Example: Your fraud model was trained on desktop transactions but mobile transactions are now 60% of volume. **Concept drift**: The relationship p(Y|X) itself changes. Example: Fraudsters adapt their tactics, so the patterns your model learned no longer predict fraud as well. Data drift is detectable without labels. Concept drift requires ground truth labels, which may arrive with a delay. Most monitoring setups detect data drift in real time and use delayed label collection to detect concept drift.
Statistical tests for drift
- **KS test** (Kolmogorov-Smirnov): Compares two continuous distributions. Sensitive to location and shape differences. - **Chi-squared test**: For categorical features. Compares observed vs expected frequencies. - **Population Stability Index (PSI)**: Industry standard from credit scoring. PSI > 0.2 = significant drift. - **MMD** (Maximum Mean Discrepancy): Kernel-based, can detect subtle distributional differences.
Drift Detection with Evidently
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
import pandas as pd
reference = pd.read_parquet("data/training_features.parquet")
current = pd.read_parquet("data/last_7days_features.parquet")
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference, current_data=current)
# Get drift summary
result = report.as_dict()
drifted_columns = [
col for col, info in result["metrics"][0]["result"]["drift_by_columns"].items()
if info["drift_detected"]
]
print(f"Drifted features ({len(drifted_columns)}): {drifted_columns[:5]}")
# Save HTML report for the team
report.save_html("reports/drift_report.html")