Serving Infrastructure

A model sitting in a notebook produces no value. Value is created when that model serves predictions to users, applications, and downstream systems—reliably, quickly, and at scale. This is the domain of serving infrastructure.

Serving infrastructure is where ML meets distributed systems engineering. The challenges are familiar to anyone who has built scalable web services—load balancing, caching, failure handling, resource management—but ML adds unique wrinkles: large model sizes, GPU acceleration, dynamic batching, and the constant evolution of models through retraining and experimentation.

This page covers the principles and practices for building serving infrastructure that transforms trained models into production prediction services.

Different use cases require different serving patterns. The right pattern depends on latency requirements, traffic patterns, and integration constraints.

The Serving Pattern Spectrum:

Pattern 1: Batch Prediction

Generate predictions for all entities on a schedule, store results:

    ┌─────────────┐      ┌─────────────┐      ┌─────────────┐
    │ All Users   │ ──→  │ Model       │ ──→  │ Predictions │
    │ (from DB)   │      │ (batch job) │      │ (to DB)     │
    └─────────────┘      └─────────────┘      └─────────────┘
                         Runs nightly
                         
    At serving time:
    Request → Database lookup → Return precomputed prediction

Advantages:

• Simplest to implement and operate
• No real-time compute required
• Can use heavyweight models
• Easy debugging (predictions are stored)

Limitations:

• Cannot handle new entities (cold start)
• Stale predictions between updates
• Storage grows with entity count
• Cannot incorporate real-time signals

Best for: Recommendations that can be a few hours old, offline scoring, regulatory reporting.

Pattern 2: Precomputed (Cache-Based)

Similar to batch but with finer-grained caching:

    Request ─┬→ Cache Hit? ──Yes──→ Return cached prediction
             │
             └→ No ──→ [Model] ──→ Cache + Return

Best for: High-traffic scenarios with significant request overlap, e.g., popular item recommendations.

Pattern 3: Online Prediction (Synchronous)

Compute predictions on-demand for each request:

    Request → [Feature Fetch] → [Model Inference] → Response
               ~10-50ms           ~10-100ms

Advantages:

• Fresh predictions with latest features
• Handles any input (no cold start)
• No storage scaling with entity count

Limitations:

• Latency includes inference time
• Requires always-on serving infrastructure
• More complex operations

Best for: Real-time personalization, fraud detection, dynamic pricing.

Pattern 4: Streaming Predictions

Process events continuously as they arrive:

    Event Stream → [Stream Processor] → [Model] → Prediction Stream
                     (Flink, Spark)     (embedded)   (Kafka, etc.)

Best for: Near-real-time analytics, continuous monitoring, event-driven predictions.

Pattern 5: Edge Deployment

Run models directly on client devices:

    [Mobile App] ←──model sync──→ [Model Server]
         │
         ▼
    [On-Device Model] → Prediction (local)

Advantages:

• Zero network latency for inference
• Works offline
• Reduced server cost

Limitations:

• Model size constraints (memory, download)
• Limited compute (no GPU)
• Update propagation delay
• Difficult to debug/monitor

Best for: Keyboard predictions, on-device image classification, privacy-sensitive applications.

For online prediction systems, the serving architecture determines how requests flow from clients to model inference and back. A well-designed architecture handles scale, failure, and evolution gracefully.

Standard Serving Architecture:

    ┌───────────────────────────────────────────────────────────────────────┐
    │                          Load Balancer                                 │
    │                    (Route, rate limit, TLS)                           │
    └───────────────────────────────┬───────────────────────────────────────┘
                                    │
            ┌───────────────────────┼───────────────────────┐
            │                       │                       │
            ▼                       ▼                       ▼
    ┌───────────────┐       ┌───────────────┐       ┌───────────────┐
    │   Gateway     │       │   Gateway     │       │   Gateway     │
    │  (API, Auth)  │       │  (API, Auth)  │       │  (API, Auth)  │
    └───────┬───────┘       └───────┬───────┘       └───────┬───────┘
            │                       │                       │
            └───────────────────────┼───────────────────────┘
                                    │
                                    ▼
    ┌───────────────────────────────────────────────────────────────────────┐
    │                      Feature Service                                   │
    │                 (Fetch features from store)                           │
    └───────────────────────────────┬───────────────────────────────────────┘
                                    │
                                    ▼
    ┌───────────────────────────────────────────────────────────────────────┐
    │                      Model Service                                     │
    │              (Run inference, return predictions)                      │
    │                                                                        │
    │   ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐                 │
    │   │Model A  │  │Model A  │  │Model B  │  │Model B  │   ...          │
    │   │Replica 1│  │Replica 2│  │Replica 1│  │Replica 2│                 │
    │   └─────────┘  └─────────┘  └─────────┘  └─────────┘                 │
    └───────────────────────────────────────────────────────────────────────┘

Embedded vs. Standalone Model Servers:

Embedded: Model runs within the application process.

# Application code
from my_model import predict

def handle_request(request):
    features = extract_features(request)
    prediction = predict(features)  # Model runs in same process
    return prediction

Standalone: Model runs in a separate service.

    Application → HTTP/gRPC → Model Server → Prediction

Recommendation: Use standalone model servers for:

• GPU-accelerated models
• Models shared across applications
• Large models that need dedicated resources
• When you need to update models without redeploying apps

Use embedded for:

• Simple models (logistic regression, small trees)
• Lowest possible latency requirements
• Edge/mobile deployment

ML serving systems must scale to handle traffic variability—from baseline load to peak events. Scaling strategy depends on the nature of the workload and the cost of additional capacity.

Horizontal vs. Vertical Scaling:

Horizontal scaling: Add more instances of the model server.

• Pros: Linear scaling, fault tolerance (no single point of failure)
• Cons: Requires load balancing, state management complexity
• Best for: Stateless inference, high throughput requirements

Vertical scaling: Use larger/more powerful instances.

• Pros: Simpler architecture, can handle larger models
• Cons: Upper limits on instance size, single point of failure
• Best for: Large models that don't fit in smaller instances, GPU upgrade paths

Auto-Scaling Patterns:

Reactive auto-scaling: Scale based on current metrics.

    [Metrics: CPU, GPU, latency, queue depth]
              │
              ▼
    [Scaling Policy: if avg_latency > 100ms for 2min, add 2 instances]
              │
              ▼
    [Instance Pool: 4 → 6 instances]

Predictive auto-scaling: Scale based on predicted future load.

    [Historical Traffic] → [Prediction Model] → [Future Load Estimate]
                                                          │
                                                          ▼
    [Pre-scale 30 minutes before predicted spike]

Schedule-based scaling: Pre-defined scaling for known patterns.

    9:00 AM weekdays: 10 instances (morning traffic spike)
    6:00 PM weekdays:  5 instances (evening decline)
    Weekends:          3 instances (lower traffic)

Scaling Metrics for ML:

Dynamic Batching:

Batching multiple requests into a single inference call dramatically improves GPU efficiency:

    Without batching:
    Request 1 → [GPU] → Response      (20% GPU utilization)
    Request 2 → [GPU] → Response      (20% GPU utilization)
    ...
    
    With batching:
    Request 1 ─┐
    Request 2 ─┼──→ [GPU: batch of 8] → Responses (80% GPU utilization)
    ...       ─┘
    Request 8 ─┘

Batching tradeoffs:

• Larger batch → Higher throughput, higher latency (waiting to fill batch)
• Smaller batch → Lower latency, lower throughput
• Dynamic batching adjusts batch size based on traffic

Batching parameters:

• max_batch_size: Maximum requests in a batch
• max_wait_time: Maximum time to wait for batch to fill
• Balance these against latency SLAs

Example configuration:

# Triton Inference Server batching config
dynamic_batching {
  max_queue_delay_microseconds: 10000  # 10ms max wait
  preferred_batch_size: [4, 8, 16]      # Preferred batch sizes
}

Model inference has specific hardware requirements that differ from general web services. Understanding these requirements enables cost-effective infrastructure selection.

CPU vs. GPU vs. Specialized Accelerators:

Hardware Selection Decision Tree:

    Is the model a neural network?
    ├── No (tree-based, linear)
    │   └── Use CPU (optimized libraries like XGBoost, ONNX Runtime)
    │
    └── Yes (neural network)
        │
        └── What's the model size?
            ├── Small (<100MB)
            │   └── CPU may suffice; GPU if high throughput needed
            │
            ├── Medium (100MB-1GB)
            │   └── GPU recommended; consider T4 or Inferentia
            │
            └── Large (>1GB)
                └── GPU required; A10G, A100, or multi-GPU

Memory Considerations:

Models must fit in memory (RAM for CPU, VRAM for GPU):

Memory optimization techniques:

• Quantization: Reduce precision (FP32 → INT8) to shrink model size 4x
• Pruning: Remove unnecessary weights to reduce memory
• Model sharing: Load model once, share across requests
• Streaming: For huge models, stream weights as needed

Cost Optimization:

Spot/Preemptible instances: For batch inference, use spot instances at 60-90% discount. Requires handling interruptions gracefully.

Right-sizing: Match instance type to model needs. Overprovisioned GPUs waste money; underprovisioned CPUs add latency.

Inference-optimized instances: AWS Inf1 (Inferentia), GCP TPU inference, Azure ML instances are often more cost-effective than general-purpose GPUs.

Cost comparison example:

Costs are illustrative; actual costs vary by model and workload.

Deploying a new model version to production carries risk. A regression in model quality or a bug in serving code can have immediate business impact. Deployment strategies manage this risk through controlled rollout.

Deployment Strategy Comparison:

Shadow Deployment:

New model receives real traffic but predictions aren't used:

    Request → [Current Model] → Production Response
                │
                └──→ [New Model] → Log only (compare offline)

Benefits:

• Zero production risk
• Test with real traffic patterns
• Compare predictions between versions
• Validate latency and error rates

Canary Deployment:

New model receives a small percentage of traffic:

    Request → [Router: 95% / 5%]
                │        │
                ▼        ▼
        [Model v1.0]  [Model v1.1]
        (production)   (canary)

Progression:

1. Deploy canary (5% traffic)
2. Monitor for 1-2 hours
3. If metrics healthy, increase to 25%
4. More monitoring
5. Increase to 50%, then 100%
6. If any issues, instant rollback to 0% canary

Blue-Green Deployment:

Maintain two complete environments, switch traffic atomically:

    Before:  Traffic → [Blue: v1.0] ← Active
                      [Green: v1.1] ← Standby (new version)
    
    Switch:  Traffic → [Blue: v1.0] ← Standby
                      [Green: v1.1] ← Active

Benefits:

• Instant rollback (just switch back)
• Full testing on idle environment
• No mixed-version serving during deployment

Costs:

• Double infrastructure cost during deployment
• More complex routing configuration

Deployment Validation Checklist:

Before promoting any model version:

1. Offline validation passed:

   • Evaluation metrics meet thresholds
   • No data quality issues in training/validation
   • Model artifact integrity verified

2. Shadow/staging validation passed:

   • Latency within SLA (P50, P95, P99)
   • No errors on production traffic patterns
   • Prediction distributions are reasonable

3. Canary validation passed:

   • Business metrics stable (conversions, engagement)
   • Error rates not increased
   • No increase in user complaints/feedback

4. Rollback plan ready:

   • Previous version available for instant rollback
   • Rollback trigger criteria defined
   • On-call team notified of deployment

Production serving systems must be reliable—available when needed, returning correct results, and degrading gracefully when issues occur.

Availability Targets:

Each additional "9" requires significantly more investment in redundancy, monitoring, and incident response.

Health Check Design:

Shallow vs. deep health checks:

# Shallow: Just checks if process is running
@app.route('/health/live')
def liveness():
    return {'status': 'alive'}

# Deep: Actually runs inference
@app.route('/health/ready')
def readiness():
    try:
        # Run a lightweight inference
        test_prediction = model.predict(CANARY_INPUT)
        
        # Validate prediction is sensible
        assert 0 <= test_prediction <= 1
        
        # Check dependencies
        feature_store_healthy = feature_store.ping()
        
        return {
            'status': 'ready',
            'model_version': model.version,
            'feature_store': feature_store_healthy
        }
    except Exception as e:
        return {'status': 'not_ready', 'error': str(e)}, 503

Use liveness for: Kubernetes restarts when process is stuck. Use readiness for: Load balancer routing; exclude unhealthy instances from rotation.

Failure Domain Isolation:

Isolate failures to prevent cascade:

    ┌─────────────────────────────────────────────────────────────────┐
    │                     Region: us-east-1                           │
    │  ┌──────────────────────┐    ┌──────────────────────┐           │
    │  │     Zone A           │    │     Zone B           │           │
    │  │  ┌────────────────┐  │    │  ┌────────────────┐  │           │
    │  │  │Model Service   │  │    │  │Model Service   │  │           │
    │  │  │Replicas: 3     │  │    │  │Replicas: 3     │  │           │
    │  │  └────────────────┘  │    │  └────────────────┘  │           │
    │  │  ┌────────────────┐  │    │  ┌────────────────┐  │           │
    │  │  │Feature Store   │  │    │  │Feature Store   │  │           │
    │  │  │(Read Replica)  │  │    │  │(Read Replica)  │  │           │
    │  │  └────────────────┘  │    │  └────────────────┘  │           │
    │  └──────────────────────┘    └──────────────────────┘           │
    └─────────────────────────────────────────────────────────────────┘

Zone failure affects only that zone; other zones continue serving.

Running ML serving in production requires ongoing operational attention. Beyond the initial deployment, teams must handle incidents, perform maintenance, and continuously improve.

Essential Operational Capabilities:

On-Call Best Practices:

1. Runbooks for common issues: Document steps for frequent alerts. On-call shouldn't require deep investigation at 3 AM.

2. Escalation paths: Clear escalation from on-call → team lead → domain expert → management.

3. Blameless postmortems: Focus on system improvements, not individual blame. What can we change to prevent recurrence?

4. Alert hygiene: Every alert should be actionable. If alerts are frequently noise, they get ignored.

5. Rotation and coverage: Fair on-call rotation with adequate rest between shifts. Cover for holidays and time zones.

Logging and Debugging:

Production debugging requires comprehensive logging:

def predict(request):
    request_id = generate_request_id()
    
    logger.info("Prediction request received", extra={
        'request_id': request_id,
        'user_id': request.user_id,
        'model_version': model.version,
        'feature_count': len(request.features),
    })
    
    start_time = time.time()
    
    # Feature retrieval
    features = feature_store.get_features(request)
    feature_time = time.time() - start_time
    
    # Model inference
    prediction = model.predict(features)
    inference_time = time.time() - start_time - feature_time
    
    logger.info("Prediction completed", extra={
        'request_id': request_id,
        'prediction': prediction,
        'feature_retrieval_ms': feature_time * 1000,
        'inference_ms': inference_time * 1000,
        'total_ms': (time.time() - start_time) * 1000,
    })
    
    return prediction

Log query examples:

• "Show me all predictions for user X in the last hour"
• "Show me the slowest 1% of predictions today"
• "Show me predictions that returned error in the last 5 minutes"

Cost Management:

ML serving can be expensive. Ongoing cost optimization is essential:

1. Right-size instances: Monitor actual utilization; downsize over-provisioned resources.

2. Use spot/preemptible for batch: Batch prediction doesn't need on-demand reliability.

3. Optimize model efficiency: Quantization, pruning, and distillation reduce inference cost.

4. Reduce unnecessary predictions: Cache where possible; batch where possible.

5. Multi-tenancy: Share infrastructure across models and teams where feasible.

Cost visibility:

• Track cost per model, per environment
• Alert on unexpected cost increases
• Attribute costs to business use cases

Example cost dashboard metrics:

• Cost per 1M predictions (by model)
• Monthly cost trend (by service)
• Cost per user/customer (attribution)
• Cost efficiency (predictions per dollar)