Model Prediction Distribution Drift Detector

In production machine learning systems, one of the most critical operational challenges is detecting when a deployed model's behavior begins to deviate from its expected performance. This phenomenon, known as model drift or prediction drift, occurs when the statistical properties of model outputs change over time, often signaling degraded model quality, changing user behavior, or shifts in the underlying data distribution.

Effective MLOps (Machine Learning Operations) practices require continuous monitoring of model predictions to ensure reliability and enable timely model retraining or intervention. This problem focuses on implementing a comprehensive prediction distribution monitoring system.

Problem Statement

Given two sequences of prediction scores—a reference set representing baseline model behavior (e.g., from validation or a stable production period) and a current set representing recent model outputs—compute key statistical metrics that quantify distribution differences and detect potential drift.

Metrics to Compute

1. Mean Shift: The arithmetic difference between the mean of current predictions and the mean of reference predictions. This captures overall shifts in model confidence or scoring tendencies.

$$\text{Mean Shift} = \mu_{current} - \mu_{reference}$$

2. Standard Deviation Ratio: The ratio of the current distribution's standard deviation to the reference distribution's standard deviation. This measures changes in prediction variability—a ratio significantly different from 1.0 indicates altered model uncertainty characteristics.

$$\text{Std Ratio} = \frac{\sigma_{current}}{\sigma_{reference}}$$

3. Jensen-Shannon Divergence (JSD): A symmetric, bounded measure of similarity between two probability distributions. Unlike KL divergence, JSD always produces a finite value between 0 (identical distributions) and 1 (maximally different distributions).

For the Jensen-Shannon divergence calculation:

• Create histograms using n_bins equally spaced bins spanning the interval [0, 1]
• Apply Laplace smoothing to handle zero-count bins: $$P(bin) = \frac{count + 1}{total + n_bins}$$
• Compute JSD as the average of KL divergences from each distribution to their mixture distribution M:

$$JSD(P | Q) = \frac{1}{2} \cdot KL(P | M) + \frac{1}{2} \cdot KL(Q | M)$$

where $$M = \frac{1}{2}(P + Q)$$

4. Drift Detection Flag: A boolean indicator set to True if the Jensen-Shannon divergence exceeds the significance threshold of 0.1, signaling that the prediction distributions have diverged meaningfully.

Your Task

Implement a function that accepts two lists of prediction probabilities and a bin count, then returns a dictionary containing all four monitoring metrics.