Canary Deployment Health Analysis for ML Model Rollouts

Overview

In enterprise machine learning operations (MLOps), canary deployments represent a critical strategy for safely releasing new model versions into production environments. Rather than immediately deploying a new model to all users—which carries inherent risk—organizations route a small percentage of live traffic (typically 1-10%) to the candidate model while the majority of requests continue flowing to the battle-tested production model.

This traffic splitting approach enables real-time performance comparison between the experimental "canary" model and the established "baseline" model using actual production data and conditions. By monitoring key performance indicators (KPIs) such as prediction accuracy and response latency, engineering teams can make data-driven decisions about whether to:

1. Promote the canary to full production deployment
2. Rollback to the baseline if the canary exhibits degraded performance
3. Extend observation if results are inconclusive

The Challenge

You are tasked with implementing a deployment health analyzer that processes prediction results from both canary and baseline models to compute comprehensive comparison metrics and generate an automated promotion recommendation.

Each prediction result in both input lists is represented as a dictionary with the following structure:

Function Specification

Implement the function evaluate_canary_rollout(canary_results, baseline_results, accuracy_tolerance, latency_tolerance) that computes and returns the following metrics:

Accuracy Metrics

• canary_accuracy: Fraction of correct predictions for the canary model (range 0-1)
• baseline_accuracy: Fraction of correct predictions for the baseline model (range 0-1)
• accuracy_change_pct: Relative change in accuracy expressed as a percentage, calculated as:
  $$\text{accuracy_change_pct} = \frac{\text{canary_accuracy} - \text{baseline_accuracy}}{\text{baseline_accuracy}} \times 100$$

Latency Metrics

• canary_avg_latency: Mean response latency of the canary model (milliseconds)
• baseline_avg_latency: Mean response latency of the baseline model (milliseconds)
• latency_change_pct: Relative change in latency expressed as a percentage, calculated as:
  $$\text{latency_change_pct} = \frac{\text{canary_avg_latency} - \text{baseline_avg_latency}}{\text{baseline_avg_latency}} \times 100$$

Promotion Decision

• promote_recommended: A boolean value indicating whether the canary model should be promoted to full production. This should be True if and only if:
  1. Accuracy has not degraded beyond the specified accuracy_tolerance (i.e., accuracy_change_pct >= -accuracy_tolerance)
  2. Latency has not increased beyond the specified latency_tolerance (i.e., latency_change_pct <= latency_tolerance)

Rounding Requirements

• Accuracy values (canary_accuracy, baseline_accuracy): Round to 4 decimal places
• All other numeric values: Round to 2 decimal places

Edge Cases

• If either input list is empty, return an empty dictionary {}

Key Insights

A negative accuracy_change_pct indicates the canary model is performing worse than baseline. A negative latency_change_pct indicates the canary model is faster than baseline (improvement). The promotion decision uses tolerance thresholds to allow for minor statistical variations in real-world traffic patterns.