Model Monitoring & Drift Detection

Five drift scenarios versus five statistical detectors.

MLOpsDrift DetectionScikit-learnStatisticsRandomForest

View code ↗

Output

Feature distributions under five drift scenarios: reference (blue) vs current (orange). No-drift overlaps; sudden, covariate, and seasonal drift visibly pull the current distribution apart.

Monitoring dashboard: drift-rate summary and performance impact per scenario. Accuracy holds at baseline under no-drift (green) and degrades with drift severity (amber → red).

Feature × detector drift grid: every feature scored by KS, PSI, JS divergence, and Wasserstein. Only the three genuinely shifted features light up as DRIFT; the rest read OK.

Page-Hinkley sequential detection on a stream whose mean shifts at t=500: the change is flagged at t=519, ~19 steps later, with no false alarms before the shift.

Detection rate per method × scenario: no-drift is clean (0/40 pairs); KS and Wasserstein catch covariate drift on every feature, while JS divergence is the most conservative.

Detector comparison on a single feature under sudden drift: three of four detectors agree it has drifted, showing why an ensemble of tests is more robust than any one.

Problem

Models silently degrade in production as data drifts. Which detection methods actually catch which kinds of drift, and how much does each cost you in accuracy?

Approach

Simulated five drift scenarios (gradual, sudden, covariate, seasonal, and a no-drift control) against a reference RandomForest model. Ran five statistical detectors (KS test, Population Stability Index, Jensen-Shannon divergence, Wasserstein distance, and Page-Hinkley) and correlated each detector’s signal against the real drop in model performance.

Result

A monitoring toolkit that maps each detection method to the drift type it catches best and quantifies the accuracy/F1 impact of drift severity: the MLOps layer that keeps deployed models honest over time.

Highlights

Five simulated drift scenarios incl. a no-drift control
Five detectors: KS, PSI, Jensen-Shannon, Wasserstein, Page-Hinkley
Correlates detector signal with real performance degradation
Practical guidance on which method fits which drift type

Next projectPPO Reinforcement Learning from Scratch →