Monitoring and Maintenance

Deploying a model to production is not the finish line—it's the starting line. Unlike traditional software that behaves consistently until code changes, ML models operate on a moving target. The data that feeds them shifts. The world that generated the training data evolves. User behavior changes. Competitors adapt. The model that performed brilliantly at launch can quietly degrade until it's actively harming the business.

Monitoring and maintenance for ML systems is fundamentally different from traditional software operations. You're not just watching for crashes and latency—you're watching for silent failures where the system continues running but predictions become worthless. You're detecting drift that accumulates gradually over weeks or months. You're making decisions about when and how to retrain, and how to validate that retraining actually helps.

This page covers the principles, techniques, and operational practices for maintaining healthy ML systems throughout their lifecycle.

ML monitoring operates across three distinct layers, each catching different types of problems. Comprehensive monitoring requires coverage of all three.

The Three Layers of ML Monitoring:

Layer 1: Infrastructure Monitoring

Traditional system monitoring—necessary but not sufficient for ML:

Layer 2: Model Performance Monitoring

ML-specific metrics that track prediction quality:

Layer 3: Business Impact Monitoring

Connect ML predictions to business outcomes:

Drift is the fundamental challenge of ML maintenance. The statistical properties of production data change over time, causing models trained on historical data to become increasingly inaccurate.

Types of Drift:

Drift Detection Methods:

Statistical Tests:

For numerical features:

• Kolmogorov-Smirnov (KS) Test: Compares two distributions; reports maximum distance between CDFs
• Population Stability Index (PSI): Measures distribution shift; commonly used threshold: PSI > 0.2 indicates significant shift
• Jensen-Shannon Divergence: Symmetric measure of distribution similarity

For categorical features:

• Chi-Square Test: Tests whether observed frequencies differ from expected
• Cramér's V: Measures association strength

Example: PSI Calculation

import numpy as np

def calculate_psi(expected, actual, buckets=10):
    """
    Calculate Population Stability Index.
    
    PSI < 0.1: No significant shift
    0.1 <= PSI < 0.2: Moderate shift (investigate)
    PSI >= 0.2: Significant shift (action required)
    """
    # Discretize into buckets
    breakpoints = np.percentile(expected, np.linspace(0, 100, buckets + 1))
    
    expected_percents = np.histogram(expected, breakpoints)[0] / len(expected)
    actual_percents = np.histogram(actual, breakpoints)[0] / len(actual)
    
    # Avoid division by zero
    expected_percents = np.maximum(expected_percents, 0.001)
    actual_percents = np.maximum(actual_percents, 0.001)
    
    psi = np.sum(
        (actual_percents - expected_percents) * 
        np.log(actual_percents / expected_percents)
    )
    
    return psi

Window-Based Monitoring:

    ┌─────────────────────────────────────────────────────────────────┐
    │                    Time Windows                                  │
    │                                                                  │
    │    Training    │  Reference  │   Current    │    Future        │
    │    (historical)│  (baseline) │  (monitored) │   (incoming)     │
    │                │             │              │                   │
    │    ═══════════ │ ─────────── │ ▓▓▓▓▓▓▓▓▓▓▓▓▓│                  │
    │                │             │              │                   │
    │  Compare reference window to current window                     │
    │  to detect shift                                                │
    └─────────────────────────────────────────────────────────────────┘

Typical window configurations:

• Reference: Last 30 days (or training data)
• Current: Last 24 hours
• Compare hourly or daily

Detecting model performance degradation is challenging because ground truth is often delayed or unavailable. Different strategies apply depending on when (or if) you receive labels.

Ground Truth Availability Scenarios:

Strategy 1: Direct Accuracy Monitoring (When Labels Are Available)

# Track accuracy metrics over time
class AccuracyMonitor:
    def __init__(self, window_size=3600):  # 1 hour window
        self.predictions = []
        self.labels = []
        self.timestamps = []
        self.window_size = window_size
    
    def record(self, prediction, label, timestamp):
        self.predictions.append(prediction)
        self.labels.append(label)
        self.timestamps.append(timestamp)
        self._trim_old_records(timestamp)
    
    def get_current_metrics(self):
        if len(self.predictions) < 100:  # Minimum sample size
            return None
        
        return {
            'accuracy': accuracy_score(self.labels, self.predictions),
            'precision': precision_score(self.labels, self.predictions),
            'recall': recall_score(self.labels, self.predictions),
            'auc': roc_auc_score(self.labels, self.predictions),
            'sample_size': len(self.predictions),
        }
    
    def check_degradation(self, baseline_metrics, threshold=0.05):
        current = self.get_current_metrics()
        if current is None:
            return None
        
        degradations = {}
        for metric, baseline_value in baseline_metrics.items():
            current_value = current.get(metric, 0)
            relative_drop = (baseline_value - current_value) / baseline_value
            
            if relative_drop > threshold:
                degradations[metric] = {
                    'baseline': baseline_value,
                    'current': current_value,
                    'drop': relative_drop,
                }
        
        return degradations

Strategy 2: Proxy Metrics (When Labels Are Delayed)

Use correlated, faster-available metrics as leading indicators:

Strategy 3: Prediction Distribution Monitoring (No Labels)

Even without labels, significant changes in prediction patterns suggest issues:

• Score distribution shift: If the model suddenly predicts many more positives (or negatives), something has changed
• Confidence calibration: If high-confidence predictions increase dramatically, the model may be overconfident
• Segment-level shifts: Predictions for specific user segments may shift while aggregate remains stable

Reference Model Comparison:

Maintain a reference model to compare production predictions:

    Production Request
          │
          ├──→ [Production Model] ──→ Production Prediction
          │                              │
          └──→ [Reference Model]  ──→ Reference Prediction
                                         │
                                         ▼
                              [Comparison Analysis]
                              
    Alert if: Production significantly diverges from Reference
              AND we believe Reference is still accurate

Reference model can be:

• Previous production version (detect regression)
• Simple baseline (detect when complex model fails)
• Ensemble of models (detect outlier predictions)

Segment-Level Monitoring:

Aggregate metrics can hide segment-level problems:

    Overall Accuracy: 92% ✓
    
    By User Segment:
    - Power Users: 95% ✓
    - New Users: 70% ✗  ← Problem hidden in aggregate!
    - Mobile Users: 85% ✓
    - Desktop Users: 94% ✓

Monitor metrics for important segments independently:

• User cohorts (new, returning, power users)
• Geographic regions
• Device types
• Feature value ranges (e.g., high vs. low price items)

Effective alerting turns monitoring data into actionable notifications. Too many alerts lead to alert fatigue; too few miss real problems. The goal is high signal-to-noise ratio with fast detection of genuine issues.

Alert Design Principles:

ML Incident Response Playbook:

Phase 1: Detection and Triage (< 15 minutes)

1. Confirm the alert is real:

   • Is this a false positive? Check multiple data sources
   • What's the scope? All traffic or specific segment?
   • When did it start? Correlate with deployments/changes

2. Assess impact:

   • How many users/requests affected?
   • What's the business impact (revenue, experience)?
   • Is it getting worse or stabilizing?

3. Decide on immediate action:

   • Rollback to previous model version?
   • Switch to fallback/baseline?
   • Disable affected feature?

Phase 2: Stabilization (< 1 hour)

1. Implement mitigation:

   • Execute rollback/fallback if decided
   • Confirm mitigation is effective
   • Communicate status to stakeholders

2. Preserve evidence:

   • Snapshot affected data and logs
   • Record timeline of events
   • Note any manual interventions

Phase 3: Investigation (hours to days)

1. Root cause analysis:

   • What caused the degradation?
   • Data issue? Model issue? Infrastructure issue?
   • Why wasn't it caught earlier?

2. Remediation:

   • Fix the underlying issue
   • Validate fix in staging
   • Deploy fix with appropriate rollout strategy

Phase 4: Postmortem and Prevention (< 1 week)

1. Document the incident:

   • Timeline, impact, root cause, resolution
   • What went well? What could improve?

2. Identify preventive actions:

   • New monitoring needed?
   • Testing gaps to address?
   • Process changes required?

3. Implement improvements:

   • Add detection for this failure mode
   • Update runbooks with lessons learned

Models degrade over time. Retraining refreshes the model with recent data, adapting to current patterns. The key decisions are when to retrain and how to retrain.

Retraining Triggers:

Retraining Approaches:

Full Retraining: Train from scratch on all available data.

    [All Historical Data] → [Training] → [New Model]

• Pros: Clean slate, captures all patterns, simple to reason about
• Cons: Computationally expensive, may lose recent patterns in large historical data
• Best for: Models that can be trained relatively quickly, when architecture might change

Incremental/Online Update: Update existing model with new data.

    [Existing Model] + [New Data] → [Update] → [Updated Model]

• Pros: Fast, emphasizes recent data, continuous adaptation
• Cons: Can drift from optimal, accumulates errors, some algorithms don't support
• Best for: High-frequency updates, when recent data is most relevant

Sliding Window: Retrain on a fixed-length window of recent data.

    [Last N Months of Data] → [Training] → [New Model]
    
    Window slides forward: discard oldest data, add newest

• Pros: Balances historical and recent, bounded training cost
• Cons: Loses long-term patterns, window size selection is critical
• Best for: Domains with strong recency effects (e.g., fashion, trending content)

Weighted/Decay Training: Weight recent data more heavily than old data.

    Sample weights: w(t) = e^(-λ * age(t))
    
    Older samples contribute less to training objective

• Pros: Uses all data, emphasizes recent, smooth transition
• Cons: Requires sample weighting support, decay rate selection
• Best for: When both historical patterns and recent trends matter

Retraining Validation:

Never deploy a retrained model without validation:

1. Offline evaluation:

   • Holdout set performance (must use temporally correct split)
   • Comparison to current production model
   • Performance on known edge cases

2. Shadow deployment:

   • Run on production traffic without serving
   • Compare predictions to production model
   • Verify latency and error rates

3. Canary deployment:

   • Small traffic percentage initially
   • Monitor closely for degradation
   • Progressive rollout if healthy

Retraining Pipeline Example:

    Trigger (scheduled/drift)
           │
           ▼
    [Data Collection] ←── Collect recent labeled data
           │
           ▼
    [Data Validation] ←── Check data quality
           │
           ▼
    [Feature Engineering] ←── Compute training features
           │
           ▼
    [Model Training] ←── Train with current hyperparameters
           │
           ▼
    [Offline Evaluation] ←── Compare to baseline
           │
           ├── Fail → Alert, investigate
           │
           ▼ Pass
    [Model Registration] ←── Version, store artifacts
           │
           ▼
    [Shadow Deployment] ←── Validate on production traffic
           │
           ├── Fail → Rollback, investigate
           │
           ▼ Pass
    [Canary Deployment] ←── Progressive rollout
           │
           ▼
    [Full Production] ←── New model is live

Manual ML operations don't scale. As models multiply and retraining becomes routine, automation is essential to maintain quality and velocity. MLOps brings DevOps principles to ML systems.

The ML Automation Maturity Model:

CI/CD for ML:

Continuous Integration (CI):

    Code Change → [Build] → [Unit Tests] → [Integration Tests] → Merge
                                │                  │
                                │                  └── Data validation tests
                                │                  └── Feature transform tests
                                │                  └── Model inference tests
                                │
                                └── Code quality, linting
                                └── Security scanning

Continuous Training (CT):

    Trigger → [Data Pipeline] → [Train] → [Evaluate] → [Register] → Staging
       │
       ├── Scheduled (daily/weekly)
       ├── Data-triggered (new data arrived)
       ├── Performance-triggered (degradation detected)
       └── Manual (on-demand)

Continuous Deployment (CD):

    Staging Model → [Shadow Test] → [Canary] → [Progressive Rollout] → Production
                          │              │               │
                    Auto-validation  Auto-validation  Auto-validation
                          │              │               │
                    Fail → Halt    Fail → Rollback  Fail → Rollback

Key Automation Infrastructure:

Infrastructure as Code for ML:

Define ML infrastructure declaratively:

# Example: ML pipeline configuration
pipeline:
  name: churn-prediction-training
  schedule: "0 2 * * *"  # Daily at 2 AM
  
  stages:
    - name: data-preparation
      image: data-prep:v1
      resources:
        cpu: "4"
        memory: "16Gi"
      inputs:
        - source: s3://data/transactions
          date_range: last_90_days
      outputs:
        - destination: s3://features/training
    
    - name: model-training
      image: training:v2
      resources:
        gpu: "1"  # NVIDIA T4
        memory: "32Gi"
      inputs:
        - source: s3://features/training
      hyperparameters:
        learning_rate: 0.001
        epochs: 100
        batch_size: 256
      outputs:
        - destination: s3://models/churn
    
    - name: model-evaluation
      image: evaluation:v1
      inputs:
        - model: s3://models/churn
        - validation_data: s3://features/validation
      thresholds:
        accuracy: 0.85
        auc: 0.90
      on_failure: alert_and_halt
    
    - name: model-registration
      image: registry:v1
      inputs:
        - model: s3://models/churn
      destination: model-registry/churn-prediction
      promote_to: staging

This configuration:

• Lives in version control
• Can be reviewed like code
• Executed consistently across environments
• Enables reproducibility

ML systems are complex, and knowledge about them often exists only in the heads of their creators. When team members leave or forget, this knowledge is lost. Systematic documentation preserves institutional knowledge and enables effective maintenance.

Model Cards:

Model cards standardize model documentation:

# Model Card: Churn Prediction v2.3

## Model Details
- **Developed by:** ML Team
- **Model date:** 2024-01-15
- **Model version:** 2.3.0
- **Model type:** Gradient Boosted Trees (LightGBM)
- **Training data:** User transactions Jan 2023 - Dec 2023

## Intended Use
- **Primary use:** Predict 30-day churn probability for subscription users
- **Out-of-scope:** Free tier users, B2B accounts, new users (<7 days)

## Metrics
| Metric | Value | Threshold |
|--------|-------|----------|
| AUC-ROC | 0.87 | > 0.85 |
| Precision@10% | 0.45 | > 0.40 |
| Recall@10% | 0.62 | > 0.50 |

## Training Data
- **Size:** 1.2M users, 14.3% churn rate
- **Features:** 47 features covering engagement, billing, support
- **Label definition:** No activity for 30+ consecutive days

## Ethical Considerations
- **Fairness:** Tested for demographic parity across age groups
- **Privacy:** No PII used directly; only behavioral features

## Limitations
- Performs poorly for users with <10 sessions (AUC drops to 0.72)
- Seasonal effects not fully captured (holiday periods)
- Does not account for promotional offers

## Caveats and Recommendations
- Retrain monthly to maintain accuracy
- Monitor for drift in session features
- High-value user predictions should be reviewed manually

Operational Runbooks:

Document procedures for common operational tasks:

# Runbook: Churn Model Degradation Alert

## Trigger
- Alert: "Churn model AUC dropped below 0.82 (threshold: 0.85)"

## Immediate Actions
1. **Verify alert is genuine:**
   - Check dashboard for sample size (< 1000 may be noise)
   - Compare to previous day's metrics
   - Check if there are infrastructure issues

2. **Assess impact:**
   - How many predictions affected?
   - Are intervention campaigns running on this model?

3. **Decide on mitigation:**
   - If degradation > 10%: Consider rollback to previous version
   - If degradation 5-10%: Monitor closely, prepare rollback
   - If degradation < 5%: Continue monitoring

## Investigation Steps
1. Check for data issues:
   - Feature freshness in feature store
   - Data pipeline success status
   - Feature distribution shifts

2. Check for model issues:
   - Recent model deployments
   - Serving infrastructure health
   - Memory/CPU utilization

3. Check for external factors:
   - Seasonal effects
   - Product changes
   - Marketing campaigns

## Escalation
- If cause unclear after 30 minutes: Page senior ML engineer
- If rollback fails: Page infrastructure on-call
- If business impact significant: Notify product stakeholders

## Recovery Validation
- Confirm metrics return to baseline
- Run smoke tests on predictions
- Document root cause for postmortem