System Design (HLD)Configuration Management

Configuration Management: Infrastructure Automation at Scale

LevelAdvanced

Duration120 mins

TopicConfiguration Management

2 / 5

Immutable vs Mutable Infrastructure: Paradigms and Trade-offs

The Fundamental Infrastructure Paradigm Choice

How should infrastructure change over time? This seemingly simple question has profound implications for reliability, security, debugging, and operational complexity. At its core, this is a choice between two fundamentally different philosophies:

Mutable Infrastructure: Servers are long-lived. When changes are needed, you modify existing servers in place—updating packages, changing configurations, deploying new code. Servers accumulate state over their lifetime.

Immutable Infrastructure: Servers are ephemeral. When changes are needed, you build new servers with the desired state and replace the old ones. Servers are never modified after creation—they're either running as originally built or replaced entirely.

This isn't merely a technical preference. It represents a fundamental shift in thinking about systems—from pets (unique, carefully maintained, irreplaceable) to cattle (identical, interchangeable, disposable).

What You Will Learn

This page provides a rigorous exploration of mutable and immutable infrastructure paradigms. You'll understand the operational, security, and reliability implications of each approach, learn implementation strategies for immutable infrastructure, and develop a framework for choosing the right paradigm for different workloads.

The choice between mutable and immutable infrastructure affects everything: how you build, how you deploy, how you troubleshoot, how you scale, and how you recover from failures. It's one of the most consequential architectural decisions in modern infrastructure design.

Mutable Infrastructure: The Traditional Model

Mutable infrastructure is the traditional approach, born from the era of physical servers and predating cloud computing. In this model, servers are provisioned once and modified continuously over their lifetime.

The Mutable Mindset

In mutable infrastructure:

Servers are long-lived: A server might run for months or years, accumulating updates, patches, and modifications.
Changes happen in-place: Need to update nginx? Run apt-get upgrade nginx on the running server. Need new configuration? Modify /etc/nginx/nginx.conf directly or via configuration management.
State accumulates: Log files grow, temporary files persist, manual fixes leave traces. The server's current state is the sum of all changes since provisioning.
Configuration management enforces convergence: Tools like Puppet, Chef, or Ansible run periodically, pushing servers toward the desired state while accommodating their current reality.

mutable_deployment.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# Ansible Playbook: Mutable Infrastructure Deployment
# Servers are updated in-place; state accumulates over time
 
- name: Deploy Application to Existing Servers
  hosts: webservers
  become: yes
  serial: "25%"  # Rolling update - 25% at a time
  
  tasks:
    # Package updates on existing server
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600
    
    - name: Install/upgrade application dependencies
      apt:
        name:
          - python3
          - python3-pip
          - nginx
          - redis-tools
        state: latest  # Upgrade to latest available
    
    - name: Install Python requirements
      pip:
        name: "{{ item }}"
        state: latest
      loop:
        - flask
        - gunicorn
        - redis
    
    # Configuration modification on running server
    - name: Deploy application code
      git:
        repo: "https://github.com/company/app.git"
        dest: /opt/app
        version: "{{ app_version }}"
      notify: Restart app
    
    - name: Update nginx configuration
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/app.conf
      notify: Reload nginx
    
    # Service management
    - name: Ensure app service is running
      systemd:
        name: app
        state: started
        enabled: yes
    
    # Verification on modified server
    - name: Wait for app to be healthy
      uri:
        url: "http://localhost:8080/health"
        status_code: 200
      retries: 30
      delay: 2
    
    # Database migration on running system
    - name: Run database migrations
      command: /opt/app/manage.py migrate
      run_once: true  # Only on first host
      when: run_migrations | default(false)
  
  handlers:
    - name: Restart app
      systemd:
        name: app
        state: restarted
    
    - name: Reload nginx
      systemd:
        name: nginx
        state: reloaded

The Advantages of Mutable Infrastructure

Despite criticisms, mutable infrastructure offers genuine advantages:

Mutable Infrastructure Strengths

•Resource Efficiency — No need to provision new servers for every change. Updates consume minimal network bandwidth and compute resources.
•Rapid Small Changes — A one-line configuration change doesn't require rebuilding and redeploying entire server images.
•Stateful Workloads — Databases, message queues, and other stateful systems naturally fit the mutable model. You can't simply replace a PostgreSQL server.
•Traditional Tooling Compatibility — Decades of operational tooling assumes mutable servers. SSH, monitoring agents, log aggregation—all designed for long-lived hosts.
•Lower Initial Learning Curve — The mental model matches how most engineers learned computing: install, configure, modify.
•Emergency Hotfixes — When production is down, SSHing in and making a targeted fix can be faster than a full deployment cycle.

Mutable Infrastructure Challenges

•Configuration Drift — Over time, servers diverge from their declared state. Manual fixes, failed updates, and environmental differences create snowflakes.
•Reproducibility Challenges — 'Works on my machine' extends to 'worked on server-1 but not server-2.' Subtle state differences cause mysterious failures.
•Audit Trail Complexity — What's actually running on a server? The answer requires reconciling CM state, manual changes, failed deployments, and accumulated cruft.
•Rollback Difficulty — Reversing a change requires knowing what the previous state was and how to restore it—not always straightforward.
•Security Concerns — Long-lived servers accumulate vulnerabilities. Patching in-place risks introducing new issues while addressing old ones.
•Disaster Recovery Complexity — Rebuilding a drifted server from scratch is error-prone. Recovery often means restoring from backup rather than rebuilding.

The Snowflake Server Problem

The worst outcome of mutable infrastructure is the 'snowflake server'—a unique, irreplaceable system that nobody fully understands. It's been modified so many times, by so many people, that its actual state is unknowable. When it fails, recovery becomes archaeology. Preventing snowflakes requires disciplined CM and strong operational practices.

Immutable Infrastructure: The Modern Paradigm

Immutable infrastructure represents a paradigm shift: instead of modifying running systems, you build new ones. Every change—code updates, configuration changes, security patches—produces a new image that replaces the old.

The Immutable Mindset

Servers are ephemeral: Running instances have minimal expected lifetime. They're created from images and destroyed without concern.
Changes create new artifacts: Want to update nginx? Build a new image with the new nginx version, deploy it, and destroy old instances.
State is externalized: Persistent data lives in managed services (RDS, S3, EBS volumes) that outlive compute instances.
Images are versioned artifacts: Each build produces a versioned, immutable artifact. Version 47 is identical whether deployed today or next month.
Consistency by construction: All servers running image version 47 are identical because they came from the same build process.

packer_build.pkr.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
# Packer Template: Building Immutable Machine Images
# This produces versioned, reproducible infrastructure artifacts
 
packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.0"
      source  = "github.com/hashicorp/amazon"
    }
    ansible = {
      version = ">= 1.1.0"
      source  = "github.com/hashicorp/ansible"
    }
  }
}
 
# Variables for parameterized builds
variable "aws_region" {
  type    = string
  default = "us-east-1"
}
 
variable "app_version" {
  type        = string
  description = "Application version to bake into the image"
}
 
variable "base_ami" {
  type        = string
  description = "Base AMI ID (Ubuntu 22.04 LTS)"
}
 
variable "instance_type" {
  type    = string
  default = "t3.medium"
}
 
# Data source: Get latest base AMI if not specified
data "amazon-ami" "ubuntu" {
  filters = {
    name                = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04*"
    virtualization-type = "hvm"
    architecture        = "x86_64"
  }
  most_recent = true
  owners      = ["099720109477"]  # Canonical
}
 
# Local variables for image metadata
locals {
  timestamp   = formatdate("YYYYMMDD-hhmm", timestamp())
  source_ami  = var.base_ami != "" ? var.base_ami : data.amazon-ami.ubuntu.id
  ami_name    = "app-${var.app_version}-${local.timestamp}"
}
 
# Source: AWS AMI Builder
source "amazon-ebs" "app" {
  ami_name        = local.ami_name
  ami_description = "Application v${var.app_version} - Built ${local.timestamp}"
  instance_type   = var.instance_type
  region          = var.aws_region
  source_ami      = local.source_ami
  ssh_username    = "ubuntu"
  
  # Networking for build
  associate_public_ip_address = true
  
  # Encrypt the resulting AMI
  encrypt_boot = true
  
  # Tag the resulting AMI and snapshots
  tags = {
    Name         = local.ami_name
    Application  = "myapp"
    Version      = var.app_version
    BuildTime    = local.timestamp
    ManagedBy    = "packer"
    Environment  = "production"
  }
  
  snapshot_tags = {
    Name         = "${local.ami_name}-root"
    Application  = "myapp"
    Version      = var.app_version
  }
  
  # Launch block device configuration
  launch_block_device_mappings {
    device_name           = "/dev/sda1"
    volume_size           = 50
    volume_type           = "gp3"
    delete_on_termination = true
    encrypted             = true
  }
}
 
# Build definition
build {
  name    = "application-server"
  sources = ["source.amazon-ebs.app"]
  
  # Wait for cloud-init to complete
  provisioner "shell" {
    inline = [
      "echo 'Waiting for cloud-init...'",
      "cloud-init status --wait"
    ]
  }
  
  # System updates and dependencies
  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get upgrade -y",
      "sudo apt-get install -y python3 python3-pip nginx"
    ]
  }
  
  # Ansible provisioning for complex configuration
  provisioner "ansible" {
    playbook_file = "./ansible/image-provision.yml"
    extra_arguments = [
      "--extra-vars", "app_version=${var.app_version}"
    ]
    user = "ubuntu"
  }
  
  # Download and install application
  provisioner "shell" {
    inline = [
      "wget https://releases.company.com/app/${var.app_version}/app.tar.gz",
      "sudo tar -xzf app.tar.gz -C /opt/",
      "rm app.tar.gz",
      "sudo /opt/app/install.sh"
    ]
  }
  
  # Security hardening
  provisioner "shell" {
    scripts = [
      "./scripts/security-hardening.sh",
      "./scripts/cis-benchmark.sh"
    ]
  }
  
  # Cleanup - remove build artifacts and temporary files
  provisioner "shell" {
    inline = [
      "sudo apt-get clean",
      "sudo rm -rf /var/lib/apt/lists/*",
      "sudo rm -rf /tmp/*",
      "sudo rm -rf /var/tmp/*",
      "sudo rm -f /var/log/wtmp /var/log/btmp",
      "sudo truncate -s 0 /var/log/lastlog",
      "sudo rm -rf /home/ubuntu/.ssh/authorized_keys",
      "sudo sync"
    ]
  }
  
  # Verify the image before completion
  provisioner "shell" {
    inline = [
      "echo 'Running image verification...'",
      "/opt/app/bin/healthcheck || exit 1",
      "nginx -t || exit 1",
      "echo 'Image verification passed!'"
    ]
  }
  
  # Post-processor: Create manifest for tracking
  post-processor "manifest" {
    output     = "manifest.json"
    strip_path = true
    custom_data = {
      app_version = var.app_version
      build_time  = local.timestamp
    }
  }
}

The Deployment Flow

With immutable infrastructure, deployment follows a fundamentally different pattern:

deployment.tf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
# Terraform: Deploying Immutable Infrastructure
# New AMI versions replace running instances
 
# Data source: Find the latest application AMI
data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
 
  filter {
    name   = "name"
    values = ["app-${var.app_version}-*"]
  }
 
  filter {
    name   = "state"
    values = ["available"]
  }
}
 
# Launch Template: Defines instance configuration
resource "aws_launch_template" "app" {
  name_prefix   = "app-"
  image_id      = data.aws_ami.app.id
  instance_type = var.instance_type
  key_name      = var.key_pair_name
 
  vpc_security_group_ids = [aws_security_group.app.id]
 
  iam_instance_profile {
    name = aws_iam_instance_profile.app.name
  }
 
  # Minimal user-data - config baked into AMI
  user_data = base64encode(<<-EOF
    #!/bin/bash
    # Only runtime-specific configuration
    echo "ENVIRONMENT=${var.environment}" >> /etc/app/env
    echo "LOG_LEVEL=${var.log_level}" >> /etc/app/env
    systemctl start app
  EOF
  )
 
  tag_specifications {
    resource_type = "instance"
    tags = {
      Name        = "app-${var.environment}"
      Application = "myapp"
      Version     = var.app_version
      Environment = var.environment
    }
  }
 
  lifecycle {
    create_before_destroy = true
  }
}
 
# Auto Scaling Group: Manages instance lifecycle
resource "aws_autoscaling_group" "app" {
  name                = "app-${var.environment}"
  desired_capacity    = var.desired_capacity
  min_size            = var.min_size
  max_size            = var.max_size
  target_group_arns   = [aws_lb_target_group.app.arn]
  vpc_zone_identifier = var.private_subnet_ids
  health_check_type   = "ELB"
  health_check_grace_period = 300
 
  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }
 
  # Rolling update configuration
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 75
      instance_warmup        = 120
    }
    triggers = ["tag"]
  }
 
  tag {
    key                 = "Name"
    value               = "app-${var.environment}"
    propagate_at_launch = true
  }
 
  lifecycle {
    create_before_destroy = true
    ignore_changes        = [desired_capacity]  # Allow autoscaling to manage
  }
}
 
# Output the current AMI being deployed
output "deployed_ami" {
  value = data.aws_ami.app.id
}
 
output "deployed_version" {
  value = var.app_version
}

Immutable Infrastructure Strengths

•Guaranteed Consistency — All instances are identical. If it works in staging, it works in production (assuming data and dependencies match).
•Trivial Rollback — Rolling back means deploying the previous image version. No need to reverse changes or resolve state conflicts.
•Full Auditability — The image ID tells you exactly what's running. The build pipeline shows exactly how it was created.
•Elimination of Configuration Drift — Drift is impossible. Servers either match the image or they don't exist.
•Security Through Freshness — New instances get latest patches baked in. No accumulation of vulnerabilities over time.
•Simplified Disaster Recovery — Rebuilding is trivial: deploy the same image. No restoration, no reconciliation.
•Blue-Green and Canary Deployments — Running multiple versions simultaneously is natural. Route traffic, validate, then decommission old.
•Debugging Reproducibility — Issues are reproducible because you can spin up the exact same image locally or in a test environment.

Immutable Infrastructure Challenges

•Build Time Overhead — Every change requires building a new image. Small configuration changes trigger full build cycles (minutes to tens of minutes).
•Storage Costs — Image repositories accumulate versions. AMIs, container images, and their snapshots consume storage and incur costs.
•Stateful Workload Complexity — Databases, message queues, and caches don't fit the 'replace rather than modify' model. Hybrid approaches required.
•Debugging Limitations — Can't SSH in and tweak things for debugging. Read-only access only (or debugging requires different mental models).
•Build Pipeline Dependency — The build pipeline becomes critical infrastructure. If it fails, deployments stop.
•Initial Learning Curve — Teams accustomed to mutable infrastructure need mindset shifts. 'Just fix it on the server' isn't an option.
•Fast Iteration Friction — Rapid prototyping and experimentation are slower when every change requires a full build cycle.

The Golden Image Anti-Pattern

Avoid the 'golden image' anti-pattern where images accumulate changes over time (cloning running servers to create 'updated' images). This reintroduces mutability and drift. Always build images from scratch using version-controlled definitions. The image build should be deterministic: same inputs produce same outputs.

Configuration Drift: The Silent System Killer

Configuration drift is the gradual divergence of system state from its declared or intended configuration. It's the primary failure mode of mutable infrastructure and the primary problem that immutable infrastructure solves.

How Drift Happens

Drift accumulates through seemingly innocent actions:

Emergency Fixes: Production is down. An engineer SSHs in and modifies a configuration file to resolve the issue. The fix works but is never backported to configuration management.
Failed Updates: A package update fails midway, leaving the server in a partially updated state. The CM tool's next run sees 'package installed' and doesn't attempt to fix halfway states.
Manual Debugging: While investigating an issue, someone disables a service or modifies a log level. They forget to revert the change.
Security Patches: Emergency security patches are applied directly, bypassing the standard deployment pipeline.
Time-Based Divergence: Even servers configured identically at provisioning diverge over time due to log growth, temporary files, and background processes.
CM Timing Windows: Configuration management tools run periodically. Between runs, manual changes persist.

drift_detection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
#!/bin/bash
# Script: Configuration Drift Detection
# Compares expected state against actual state across fleet
 
# Define expected state (from configuration management)
declare -A EXPECTED_STATE=(
  ["nginx_version"]="1.24.0"
  ["nginx_config_hash"]="a1b2c3d4e5f6"
  ["app_version"]="2.3.1"
  ["ssl_cert_expiry"]="2025-06-15"
  ["kernel_version"]="5.15.0-91"
)
 
# List of servers to check
SERVERS=$(aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=production" \
  --query 'Reservations[].Instances[].PrivateDnsName' \
  --output text)
 
# Drift detection results
DRIFT_DETECTED=false
DRIFT_REPORT=""
 
for server in $SERVERS; do
  echo "Checking drift on $server..."
  
  # Collect actual state
  ACTUAL_NGINX=$(ssh "$server" "nginx -v 2>&1 | grep -oP '\d+\.\d+\.\d+'")
  ACTUAL_CONFIG=$(ssh "$server" "md5sum /etc/nginx/nginx.conf | cut -d' ' -f1")
  ACTUAL_APP=$(ssh "$server" "cat /opt/app/VERSION")
  ACTUAL_KERNEL=$(ssh "$server" "uname -r")
  
  # Compare states
  if [[ "$ACTUAL_NGINX" != "${EXPECTED_STATE[nginx_version]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: nginx version drift ($ACTUAL_NGINX vs ${EXPECTED_STATE[nginx_version]})
"
  fi
  
  if [[ "$ACTUAL_CONFIG" != "${EXPECTED_STATE[nginx_config_hash]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: nginx config drift (hash mismatch)
"
    
    # Capture diff for analysis
    ssh "$server" "cat /etc/nginx/nginx.conf" > /tmp/${server}_nginx.conf
    diff /tmp/expected_nginx.conf /tmp/${server}_nginx.conf > /tmp/${server}_diff.txt
  fi
  
  if [[ "$ACTUAL_APP" != "${EXPECTED_STATE[app_version]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: app version drift ($ACTUAL_APP vs ${EXPECTED_STATE[app_version]})
"
  fi
  
  if [[ "$ACTUAL_KERNEL" != *"${EXPECTED_STATE[kernel_version]}"* ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: kernel version drift ($ACTUAL_KERNEL vs ${EXPECTED_STATE[kernel_version]})
"
  fi
done
 
# Report results
if $DRIFT_DETECTED; then
  echo "=== DRIFT DETECTED ==="
  echo -e "$DRIFT_REPORT"
  
  # Send alert
  aws sns publish \
    --topic-arn "arn:aws:sns:us-east-1:123456789:alerts" \
    --message "Configuration drift detected:
$DRIFT_REPORT" \
    --subject "ALERT: Configuration Drift Detected"
  
  exit 1
else
  echo "No drift detected. All servers match expected state."
  exit 0
fi

The Cost of Drift

Drift isn't just a theoretical concern—it has real operational impact:

Configuration Drift Impact Analysis
Impact Area	Consequence	Business Cost
Reliability	Servers behave inconsistently under load	Intermittent outages, customer complaints
Security	Unpatched servers, exposed credentials	Breaches, compliance violations, fines
Debugging	Issues unreproducible between servers	Extended incident response times
Deployment	Deployments fail on some servers, not others	Partial rollouts, manual intervention
Scaling	New servers don't match old servers	Auto-scaling produces broken instances
Recovery	DR attempts fail due to unknown state	Extended downtime during disasters
Compliance	Audit failures due to undocumented changes	Regulatory penalties, lost certifications

Drift Detection vs. Drift Prevention

Drift detection (monitoring for drift) is necessary but insufficient. Detection tells you drift exists; it doesn't prevent it. Immutable infrastructure prevents drift by construction—there's no mechanism for in-place modification. If you must use mutable infrastructure, combine aggressive drift detection with rapid remediation (automated reconvergence).

Implementation Strategies for Immutable Infrastructure

Transitioning to immutable infrastructure requires careful strategy. The approach differs based on your infrastructure type: VMs, containers, or hybrid environments.

Virtual Machine-Based Immutability

For VM-based infrastructure (AWS EC2, GCP Compute Engine, Azure VMs), the immutable pattern involves machine images:

VM Immutability Pipeline

•Base Image Selection — Start with a vendor-provided, minimal base image (Ubuntu Server LTS, Amazon Linux 2, etc.)
•Image Building (Packer) — Use Packer with Ansible/Shell provisioners to install packages, deploy application, configure services
•Image Validation — Run automated tests against the built image: security scans, compliance checks, application health
•Image Registry — Store versioned images in cloud-native registries (AMI, GCP Images) with clear naming and tagging
•Deployment (Terraform) — Use Infrastructure as Code to deploy instances from the validated image
•Rolling Replacement — Auto Scaling Groups with instance refresh replace running instances with new image versions

Container-Based Immutability

Containers (Docker, containerd) are inherently immutable by design. The container image is fixed at build time; running containers are read-only (with explicit writable layers for temporary data):

Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Dockerfile: Immutable Application Container
# Multi-stage build for minimal, secure runtime image
 
# Stage 1: Build environment
FROM node:20-alpine AS builder
 
WORKDIR /app
 
# Install dependencies first (layer caching)
COPY package*.json ./
RUN npm ci --only=production
 
# Copy source and build
COPY src/ ./src/
COPY tsconfig.json ./
RUN npm run build
 
# Prune dev dependencies
RUN npm prune --production
 
# Stage 2: Runtime environment
FROM node:20-alpine AS runtime
 
# Security: Run as non-root user
RUN addgroup -g 1001 appgroup && \
    adduser -u 1001 -G appgroup -s /bin/sh -D appuser
 
# Install runtime dependencies only
RUN apk add --no-cache dumb-init
 
WORKDIR /app
 
# Copy only necessary artifacts from builder
COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/package.json ./
 
# Application configuration (non-sensitive)
ENV NODE_ENV=production
ENV PORT=8080
 
# Health check for orchestrator
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
 
# Drop to non-root user
USER appuser
 
# Expose application port
EXPOSE 8080
 
# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "dist/server.js"]

kubernetes-deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
# Kubernetes: Deploying Immutable Containers
# Immutability enforced via image tags and read-only filesystem
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: production
  labels:
    app: myapp
    version: v2.3.1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
        version: v2.3.1
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      # Security context for immutability
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      
      containers:
      - name: app
        # Immutable: Pinned image digest (not :latest)
        image: registry.company.com/app@sha256:abc123def456...
        imagePullPolicy: Always
        
        ports:
        - containerPort: 8080
          protocol: TCP
        
        # Environment from ConfigMap and Secrets
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets
        
        # Resource limits
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        
        # Enforce read-only root filesystem
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
              - ALL
        
        # Writable volumes for temporary data only
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/cache
        
        # Probes for lifecycle management
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
      
      # Ephemeral volumes for runtime data
      volumes:
      - name: tmp
        emptyDir:
          medium: Memory
          sizeLimit: "64Mi"
      - name: cache
        emptyDir:
          sizeLimit: "256Mi"

Image Tagging Best Practices

Never deploy ':latest' in production—it's mutable. Use immutable identifiers: semantic versions (v2.3.1), git commit SHAs (abc123def456), or full image digests (sha256:...). Digests provide absolute immutability; even if a tag is overwritten, the digest points to the original build.

Hybrid Approaches: Practical Reality

In practice, most organizations operate hybrid environments. Pure immutability isn't always feasible or optimal. The key is being intentional about which components are mutable and which are immutable.

The Spectrum of Mutability

Rather than a binary choice, think of mutability as a spectrum:

Infrastructure Mutability Spectrum
Category	Mutability	Examples	Rationale
Stateless Compute	Fully Immutable	Web servers, API servers, workers	No persistent state; replacement is trivial
Application Config	Semi-Immutable	Feature flags, log levels	Runtime toggles via ConfigMaps; pod unchanged
Databases (Primary)	Mutable with Controls	PostgreSQL, MySQL primaries	State is the point; in-place updates necessary
Databases (Replicas)	More Immutable	Read replicas, analytics DBs	Can rebuild from primary; replacement viable
Network Infrastructure	Stable/Long-lived	VPCs, subnets, routes	Foundation layer; changes rare and deliberate
Secrets/Certs	Externally Managed	Vault secrets, TLS certificates	Rotated independently; injected at runtime

Practical Hybrid Patterns

Immutable Compute, Mutable Data: The most common pattern. Compute instances are immutable; data lives in managed services (RDS, S3) or dedicated persistent volumes.
Immutable Base, Mutable Layer: The base image is immutable, but runtime configuration is injected at startup via environment variables, ConfigMaps, or parameter stores.
Immutable Container, Mutable Sidecar: The main application container is immutable; a mutable sidecar handles dynamic concerns (log shipping, metrics, config reload).
Periodic Refresh: Mutable servers are periodically terminated and replaced (weekly, monthly) to limit drift accumulation without full immutability.

hybrid-architecture.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# Hybrid Architecture: Immutable Compute + Mutable Config + Persistent Data
#
# This pattern separates concerns across the mutability spectrum
 
# IMMUTABLE LAYER: Application containers
# - Built once, deployed identically everywhere
# - Version-pinned image references
# - Read-only root filesystem
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: api
        image: registry/api:v3.2.1-sha256abc  # Immutable reference
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - name: config
          mountPath: /app/config
          readOnly: true  # Config is read-only to container
        - name: secrets
          mountPath: /app/secrets
          readOnly: true
      
      volumes:
      # SEMI-MUTABLE LAYER: ConfigMaps
      # - Changes require pod restart but not rebuild
      # - Version controlled in Git
      - name: config
        configMap:
          name: api-config  # Can be updated, pod sees new version on restart
      
      # EXTERNALLY MUTABLE: Secrets
      # - Managed by Vault/External Secrets Operator
      # - Rotated independently of application
      - name: secrets
        secret:
          secretName: api-secrets
 
---
# MUTABLE/PERSISTENT LAYER: StatefulSet for databases
# - Cannot be simply replaced; state is critical
# - In-place updates necessary
# - Strong backup/restore procedures critical
 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3  # Primary + 2 replicas
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:15.4  # Version pinned, but pod is long-lived
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-encrypted
      resources:
        requests:
          storage: 500Gi  # Mutable, persistent storage

The Data Plane vs Control Plane Distinction

A useful mental model: the control plane (how things are configured) should be immutable and version-controlled. The data plane (what the system processes) is inherently mutable. Conflating these—baking data into images or treating configuration as runtime data—leads to problems. Keep them separate.

Migrating to Immutable Infrastructure

Transitioning from mutable to immutable infrastructure is a journey, not a switch. Organizations typically move incrementally, starting with new workloads and gradually migrating existing systems.

Migration Phases

Phased Migration Approach

•Phase 0: Foundation — Establish image building pipelines (Packer/Docker), container registries, and deployment automation (Terraform/Kubernetes). Run parallel to existing infrastructure.
•Phase 1: New Workloads — Deploy all new applications as immutable from the start. Build organizational muscle without risking existing systems.
•Phase 2: Stateless Services — Migrate existing stateless services (web frontends, API servers, workers) to immutable infrastructure. These have the lowest risk.
•Phase 3: Stateless with State Dependencies — Migrate services that depend on stateful backends. Requires careful data layer planning and externalizing state.
•Phase 4: Evaluate Stateful — Assess stateful workloads. Some may remain mutable; others may migrate to managed services (RDS vs self-managed Postgres).
•Phase 5: Continuous Improvement — Refine pipelines, reduce image build times, improve rollback procedures, enhance monitoring.

Common Migration Challenges

Migration Challenges and Mitigations
Challenge	Description	Mitigation Strategy
Configuration in code	Hardcoded values in application expecting local files	Externalize to environment variables, ConfigMaps, or parameter stores
Local file dependencies	Applications writing to local filesystem	Use object storage (S3), EBS volumes, or ephemeral volumes
SSH access expectations	Teams accustomed to debugging via SSH	Implement centralized logging, tracing, and metrics. Kubectl exec for containers
Slow build times	Long image builds slow deployment velocity	Optimize Dockerfiles, use layer caching, parallelize builds
Stateful application patterns	Applications designed for persistent servers	Refactor to externalize state or accept managed mutability
Cultural resistance	Teams uncomfortable with new mental model	Training, pair programming, celebrate early wins, document runbooks

The 'Just This Once' Trap

The most dangerous moment is when something goes wrong and someone says 'Just this once, let me SSH in and fix it.' Every exception erodes the immutability discipline. Instead, invest in faster build/deploy cycles so that 'deploy a fix' becomes faster than 'manually fix a server.' If a hotfix is truly unavoidable, document it and immediately backport to the image pipeline.

Summary: Choosing Your Infrastructure Paradigm

We've explored the fundamental choice between mutable and immutable infrastructure paradigms. Let's consolidate the key insights:

Key Takeaways

•Mutable infrastructure modifies in-place — Long-lived servers accumulate changes. Configuration management maintains convergence but drift remains a constant battle.
•Immutable infrastructure replaces rather than modifies — Every change produces new artifacts. Consistency is guaranteed by construction, not enforcement.
•Configuration drift is the primary failure mode of mutable infrastructure — Servers diverge subtly, causing inconsistent behavior, failed deployments, and debugging nightmares.
•Immutable infrastructure eliminates drift by design — No modification mechanism means no drift. Images are versioned, reproducible, auditable artifacts.
•Pure immutability isn't always practical — Databases, message queues, and some legacy systems require mutable patterns. Hybrid approaches are common.
•The key is intentionality — Understand which components should be mutable and which immutable. Establish clear boundaries and workflows for each.
•Migration is incremental — Start with new workloads, progress through stateless services, evaluate stateful workloads individually.

What's Next:

With an understanding of infrastructure paradigms, we'll dive into a specific and critical challenge in configuration management: configuration drift. We'll explore detection mechanisms, prevention strategies, and remediation approaches in detail.

Page Complete

You now understand the fundamental choice between mutable and immutable infrastructure, their trade-offs, implementation strategies, and practical hybrid approaches. This foundation enables informed architectural decisions about how your infrastructure should evolve over time. Next, we'll explore configuration drift detection and prevention in depth.

2 / 5

Loading learning content...

System Design (HLD)Configuration Management

Configuration Management: Infrastructure Automation at Scale

LevelAdvanced

Duration120 mins

TopicConfiguration Management

2 / 5

Immutable vs Mutable Infrastructure: Paradigms and Trade-offs

The Fundamental Infrastructure Paradigm Choice

What You Will Learn

Mutable Infrastructure: The Traditional Model

The Mutable Mindset

In mutable infrastructure:

Servers are long-lived: A server might run for months or years, accumulating updates, patches, and modifications.
Changes happen in-place: Need to update nginx? Run apt-get upgrade nginx on the running server. Need new configuration? Modify /etc/nginx/nginx.conf directly or via configuration management.
State accumulates: Log files grow, temporary files persist, manual fixes leave traces. The server's current state is the sum of all changes since provisioning.
Configuration management enforces convergence: Tools like Puppet, Chef, or Ansible run periodically, pushing servers toward the desired state while accommodating their current reality.

mutable_deployment.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# Ansible Playbook: Mutable Infrastructure Deployment
# Servers are updated in-place; state accumulates over time
 
- name: Deploy Application to Existing Servers
  hosts: webservers
  become: yes
  serial: "25%"  # Rolling update - 25% at a time
  
  tasks:
    # Package updates on existing server
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600
    
    - name: Install/upgrade application dependencies
      apt:
        name:
          - python3
          - python3-pip
          - nginx
          - redis-tools
        state: latest  # Upgrade to latest available
    
    - name: Install Python requirements
      pip:
        name: "{{ item }}"
        state: latest
      loop:
        - flask
        - gunicorn
        - redis
    
    # Configuration modification on running server
    - name: Deploy application code
      git:
        repo: "https://github.com/company/app.git"
        dest: /opt/app
        version: "{{ app_version }}"
      notify: Restart app
    
    - name: Update nginx configuration
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/app.conf
      notify: Reload nginx
    
    # Service management
    - name: Ensure app service is running
      systemd:
        name: app
        state: started
        enabled: yes
    
    # Verification on modified server
    - name: Wait for app to be healthy
      uri:
        url: "http://localhost:8080/health"
        status_code: 200
      retries: 30
      delay: 2
    
    # Database migration on running system
    - name: Run database migrations
      command: /opt/app/manage.py migrate
      run_once: true  # Only on first host
      when: run_migrations | default(false)
  
  handlers:
    - name: Restart app
      systemd:
        name: app
        state: restarted
    
    - name: Reload nginx
      systemd:
        name: nginx
        state: reloaded

The Advantages of Mutable Infrastructure

Despite criticisms, mutable infrastructure offers genuine advantages:

Mutable Infrastructure Strengths

•Resource Efficiency — No need to provision new servers for every change. Updates consume minimal network bandwidth and compute resources.
•Rapid Small Changes — A one-line configuration change doesn't require rebuilding and redeploying entire server images.
•Stateful Workloads — Databases, message queues, and other stateful systems naturally fit the mutable model. You can't simply replace a PostgreSQL server.
•Traditional Tooling Compatibility — Decades of operational tooling assumes mutable servers. SSH, monitoring agents, log aggregation—all designed for long-lived hosts.
•Lower Initial Learning Curve — The mental model matches how most engineers learned computing: install, configure, modify.
•Emergency Hotfixes — When production is down, SSHing in and making a targeted fix can be faster than a full deployment cycle.

Mutable Infrastructure Challenges

•Configuration Drift — Over time, servers diverge from their declared state. Manual fixes, failed updates, and environmental differences create snowflakes.
•Reproducibility Challenges — 'Works on my machine' extends to 'worked on server-1 but not server-2.' Subtle state differences cause mysterious failures.
•Audit Trail Complexity — What's actually running on a server? The answer requires reconciling CM state, manual changes, failed deployments, and accumulated cruft.
•Rollback Difficulty — Reversing a change requires knowing what the previous state was and how to restore it—not always straightforward.
•Security Concerns — Long-lived servers accumulate vulnerabilities. Patching in-place risks introducing new issues while addressing old ones.
•Disaster Recovery Complexity — Rebuilding a drifted server from scratch is error-prone. Recovery often means restoring from backup rather than rebuilding.

The Snowflake Server Problem

Immutable Infrastructure: The Modern Paradigm

The Immutable Mindset

Servers are ephemeral: Running instances have minimal expected lifetime. They're created from images and destroyed without concern.
Changes create new artifacts: Want to update nginx? Build a new image with the new nginx version, deploy it, and destroy old instances.
State is externalized: Persistent data lives in managed services (RDS, S3, EBS volumes) that outlive compute instances.
Images are versioned artifacts: Each build produces a versioned, immutable artifact. Version 47 is identical whether deployed today or next month.
Consistency by construction: All servers running image version 47 are identical because they came from the same build process.

packer_build.pkr.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
# Packer Template: Building Immutable Machine Images
# This produces versioned, reproducible infrastructure artifacts
 
packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.0"
      source  = "github.com/hashicorp/amazon"
    }
    ansible = {
      version = ">= 1.1.0"
      source  = "github.com/hashicorp/ansible"
    }
  }
}
 
# Variables for parameterized builds
variable "aws_region" {
  type    = string
  default = "us-east-1"
}
 
variable "app_version" {
  type        = string
  description = "Application version to bake into the image"
}
 
variable "base_ami" {
  type        = string
  description = "Base AMI ID (Ubuntu 22.04 LTS)"
}
 
variable "instance_type" {
  type    = string
  default = "t3.medium"
}
 
# Data source: Get latest base AMI if not specified
data "amazon-ami" "ubuntu" {
  filters = {
    name                = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04*"
    virtualization-type = "hvm"
    architecture        = "x86_64"
  }
  most_recent = true
  owners      = ["099720109477"]  # Canonical
}
 
# Local variables for image metadata
locals {
  timestamp   = formatdate("YYYYMMDD-hhmm", timestamp())
  source_ami  = var.base_ami != "" ? var.base_ami : data.amazon-ami.ubuntu.id
  ami_name    = "app-${var.app_version}-${local.timestamp}"
}
 
# Source: AWS AMI Builder
source "amazon-ebs" "app" {
  ami_name        = local.ami_name
  ami_description = "Application v${var.app_version} - Built ${local.timestamp}"
  instance_type   = var.instance_type
  region          = var.aws_region
  source_ami      = local.source_ami
  ssh_username    = "ubuntu"
  
  # Networking for build
  associate_public_ip_address = true
  
  # Encrypt the resulting AMI
  encrypt_boot = true
  
  # Tag the resulting AMI and snapshots
  tags = {
    Name         = local.ami_name
    Application  = "myapp"
    Version      = var.app_version
    BuildTime    = local.timestamp
    ManagedBy    = "packer"
    Environment  = "production"
  }
  
  snapshot_tags = {
    Name         = "${local.ami_name}-root"
    Application  = "myapp"
    Version      = var.app_version
  }
  
  # Launch block device configuration
  launch_block_device_mappings {
    device_name           = "/dev/sda1"
    volume_size           = 50
    volume_type           = "gp3"
    delete_on_termination = true
    encrypted             = true
  }
}
 
# Build definition
build {
  name    = "application-server"
  sources = ["source.amazon-ebs.app"]
  
  # Wait for cloud-init to complete
  provisioner "shell" {
    inline = [
      "echo 'Waiting for cloud-init...'",
      "cloud-init status --wait"
    ]
  }
  
  # System updates and dependencies
  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get upgrade -y",
      "sudo apt-get install -y python3 python3-pip nginx"
    ]
  }
  
  # Ansible provisioning for complex configuration
  provisioner "ansible" {
    playbook_file = "./ansible/image-provision.yml"
    extra_arguments = [
      "--extra-vars", "app_version=${var.app_version}"
    ]
    user = "ubuntu"
  }
  
  # Download and install application
  provisioner "shell" {
    inline = [
      "wget https://releases.company.com/app/${var.app_version}/app.tar.gz",
      "sudo tar -xzf app.tar.gz -C /opt/",
      "rm app.tar.gz",
      "sudo /opt/app/install.sh"
    ]
  }
  
  # Security hardening
  provisioner "shell" {
    scripts = [
      "./scripts/security-hardening.sh",
      "./scripts/cis-benchmark.sh"
    ]
  }
  
  # Cleanup - remove build artifacts and temporary files
  provisioner "shell" {
    inline = [
      "sudo apt-get clean",
      "sudo rm -rf /var/lib/apt/lists/*",
      "sudo rm -rf /tmp/*",
      "sudo rm -rf /var/tmp/*",
      "sudo rm -f /var/log/wtmp /var/log/btmp",
      "sudo truncate -s 0 /var/log/lastlog",
      "sudo rm -rf /home/ubuntu/.ssh/authorized_keys",
      "sudo sync"
    ]
  }
  
  # Verify the image before completion
  provisioner "shell" {
    inline = [
      "echo 'Running image verification...'",
      "/opt/app/bin/healthcheck || exit 1",
      "nginx -t || exit 1",
      "echo 'Image verification passed!'"
    ]
  }
  
  # Post-processor: Create manifest for tracking
  post-processor "manifest" {
    output     = "manifest.json"
    strip_path = true
    custom_data = {
      app_version = var.app_version
      build_time  = local.timestamp
    }
  }
}

The Deployment Flow

With immutable infrastructure, deployment follows a fundamentally different pattern:

deployment.tf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
# Terraform: Deploying Immutable Infrastructure
# New AMI versions replace running instances
 
# Data source: Find the latest application AMI
data "aws_ami" "app" {
  most_recent = true
  owners      = ["self"]
 
  filter {
    name   = "name"
    values = ["app-${var.app_version}-*"]
  }
 
  filter {
    name   = "state"
    values = ["available"]
  }
}
 
# Launch Template: Defines instance configuration
resource "aws_launch_template" "app" {
  name_prefix   = "app-"
  image_id      = data.aws_ami.app.id
  instance_type = var.instance_type
  key_name      = var.key_pair_name
 
  vpc_security_group_ids = [aws_security_group.app.id]
 
  iam_instance_profile {
    name = aws_iam_instance_profile.app.name
  }
 
  # Minimal user-data - config baked into AMI
  user_data = base64encode(<<-EOF
    #!/bin/bash
    # Only runtime-specific configuration
    echo "ENVIRONMENT=${var.environment}" >> /etc/app/env
    echo "LOG_LEVEL=${var.log_level}" >> /etc/app/env
    systemctl start app
  EOF
  )
 
  tag_specifications {
    resource_type = "instance"
    tags = {
      Name        = "app-${var.environment}"
      Application = "myapp"
      Version     = var.app_version
      Environment = var.environment
    }
  }
 
  lifecycle {
    create_before_destroy = true
  }
}
 
# Auto Scaling Group: Manages instance lifecycle
resource "aws_autoscaling_group" "app" {
  name                = "app-${var.environment}"
  desired_capacity    = var.desired_capacity
  min_size            = var.min_size
  max_size            = var.max_size
  target_group_arns   = [aws_lb_target_group.app.arn]
  vpc_zone_identifier = var.private_subnet_ids
  health_check_type   = "ELB"
  health_check_grace_period = 300
 
  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }
 
  # Rolling update configuration
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 75
      instance_warmup        = 120
    }
    triggers = ["tag"]
  }
 
  tag {
    key                 = "Name"
    value               = "app-${var.environment}"
    propagate_at_launch = true
  }
 
  lifecycle {
    create_before_destroy = true
    ignore_changes        = [desired_capacity]  # Allow autoscaling to manage
  }
}
 
# Output the current AMI being deployed
output "deployed_ami" {
  value = data.aws_ami.app.id
}
 
output "deployed_version" {
  value = var.app_version
}

Immutable Infrastructure Strengths

•Guaranteed Consistency — All instances are identical. If it works in staging, it works in production (assuming data and dependencies match).
•Trivial Rollback — Rolling back means deploying the previous image version. No need to reverse changes or resolve state conflicts.
•Full Auditability — The image ID tells you exactly what's running. The build pipeline shows exactly how it was created.
•Elimination of Configuration Drift — Drift is impossible. Servers either match the image or they don't exist.
•Security Through Freshness — New instances get latest patches baked in. No accumulation of vulnerabilities over time.
•Simplified Disaster Recovery — Rebuilding is trivial: deploy the same image. No restoration, no reconciliation.
•Blue-Green and Canary Deployments — Running multiple versions simultaneously is natural. Route traffic, validate, then decommission old.
•Debugging Reproducibility — Issues are reproducible because you can spin up the exact same image locally or in a test environment.

Immutable Infrastructure Challenges

•Build Time Overhead — Every change requires building a new image. Small configuration changes trigger full build cycles (minutes to tens of minutes).
•Storage Costs — Image repositories accumulate versions. AMIs, container images, and their snapshots consume storage and incur costs.
•Stateful Workload Complexity — Databases, message queues, and caches don't fit the 'replace rather than modify' model. Hybrid approaches required.
•Debugging Limitations — Can't SSH in and tweak things for debugging. Read-only access only (or debugging requires different mental models).
•Build Pipeline Dependency — The build pipeline becomes critical infrastructure. If it fails, deployments stop.
•Initial Learning Curve — Teams accustomed to mutable infrastructure need mindset shifts. 'Just fix it on the server' isn't an option.
•Fast Iteration Friction — Rapid prototyping and experimentation are slower when every change requires a full build cycle.

The Golden Image Anti-Pattern

Configuration Drift: The Silent System Killer

How Drift Happens

Drift accumulates through seemingly innocent actions:

Emergency Fixes: Production is down. An engineer SSHs in and modifies a configuration file to resolve the issue. The fix works but is never backported to configuration management.
Failed Updates: A package update fails midway, leaving the server in a partially updated state. The CM tool's next run sees 'package installed' and doesn't attempt to fix halfway states.
Manual Debugging: While investigating an issue, someone disables a service or modifies a log level. They forget to revert the change.
Security Patches: Emergency security patches are applied directly, bypassing the standard deployment pipeline.
Time-Based Divergence: Even servers configured identically at provisioning diverge over time due to log growth, temporary files, and background processes.
CM Timing Windows: Configuration management tools run periodically. Between runs, manual changes persist.

drift_detection.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
#!/bin/bash
# Script: Configuration Drift Detection
# Compares expected state against actual state across fleet
 
# Define expected state (from configuration management)
declare -A EXPECTED_STATE=(
  ["nginx_version"]="1.24.0"
  ["nginx_config_hash"]="a1b2c3d4e5f6"
  ["app_version"]="2.3.1"
  ["ssl_cert_expiry"]="2025-06-15"
  ["kernel_version"]="5.15.0-91"
)
 
# List of servers to check
SERVERS=$(aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=production" \
  --query 'Reservations[].Instances[].PrivateDnsName' \
  --output text)
 
# Drift detection results
DRIFT_DETECTED=false
DRIFT_REPORT=""
 
for server in $SERVERS; do
  echo "Checking drift on $server..."
  
  # Collect actual state
  ACTUAL_NGINX=$(ssh "$server" "nginx -v 2>&1 | grep -oP '\d+\.\d+\.\d+'")
  ACTUAL_CONFIG=$(ssh "$server" "md5sum /etc/nginx/nginx.conf | cut -d' ' -f1")
  ACTUAL_APP=$(ssh "$server" "cat /opt/app/VERSION")
  ACTUAL_KERNEL=$(ssh "$server" "uname -r")
  
  # Compare states
  if [[ "$ACTUAL_NGINX" != "${EXPECTED_STATE[nginx_version]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: nginx version drift ($ACTUAL_NGINX vs ${EXPECTED_STATE[nginx_version]})
"
  fi
  
  if [[ "$ACTUAL_CONFIG" != "${EXPECTED_STATE[nginx_config_hash]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: nginx config drift (hash mismatch)
"
    
    # Capture diff for analysis
    ssh "$server" "cat /etc/nginx/nginx.conf" > /tmp/${server}_nginx.conf
    diff /tmp/expected_nginx.conf /tmp/${server}_nginx.conf > /tmp/${server}_diff.txt
  fi
  
  if [[ "$ACTUAL_APP" != "${EXPECTED_STATE[app_version]}" ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: app version drift ($ACTUAL_APP vs ${EXPECTED_STATE[app_version]})
"
  fi
  
  if [[ "$ACTUAL_KERNEL" != *"${EXPECTED_STATE[kernel_version]}"* ]]; then
    DRIFT_DETECTED=true
    DRIFT_REPORT+="$server: kernel version drift ($ACTUAL_KERNEL vs ${EXPECTED_STATE[kernel_version]})
"
  fi
done
 
# Report results
if $DRIFT_DETECTED; then
  echo "=== DRIFT DETECTED ==="
  echo -e "$DRIFT_REPORT"
  
  # Send alert
  aws sns publish \
    --topic-arn "arn:aws:sns:us-east-1:123456789:alerts" \
    --message "Configuration drift detected:
$DRIFT_REPORT" \
    --subject "ALERT: Configuration Drift Detected"
  
  exit 1
else
  echo "No drift detected. All servers match expected state."
  exit 0
fi

The Cost of Drift

Drift isn't just a theoretical concern—it has real operational impact:

Configuration Drift Impact Analysis
Impact Area	Consequence	Business Cost
Reliability	Servers behave inconsistently under load	Intermittent outages, customer complaints
Security	Unpatched servers, exposed credentials	Breaches, compliance violations, fines
Debugging	Issues unreproducible between servers	Extended incident response times
Deployment	Deployments fail on some servers, not others	Partial rollouts, manual intervention
Scaling	New servers don't match old servers	Auto-scaling produces broken instances
Recovery	DR attempts fail due to unknown state	Extended downtime during disasters
Compliance	Audit failures due to undocumented changes	Regulatory penalties, lost certifications

Drift Detection vs. Drift Prevention

Implementation Strategies for Immutable Infrastructure

Transitioning to immutable infrastructure requires careful strategy. The approach differs based on your infrastructure type: VMs, containers, or hybrid environments.

Virtual Machine-Based Immutability

For VM-based infrastructure (AWS EC2, GCP Compute Engine, Azure VMs), the immutable pattern involves machine images:

VM Immutability Pipeline

•Base Image Selection — Start with a vendor-provided, minimal base image (Ubuntu Server LTS, Amazon Linux 2, etc.)
•Image Building (Packer) — Use Packer with Ansible/Shell provisioners to install packages, deploy application, configure services
•Image Validation — Run automated tests against the built image: security scans, compliance checks, application health
•Image Registry — Store versioned images in cloud-native registries (AMI, GCP Images) with clear naming and tagging
•Deployment (Terraform) — Use Infrastructure as Code to deploy instances from the validated image
•Rolling Replacement — Auto Scaling Groups with instance refresh replace running instances with new image versions

Container-Based Immutability

Containers (Docker, containerd) are inherently immutable by design. The container image is fixed at build time; running containers are read-only (with explicit writable layers for temporary data):

Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Dockerfile: Immutable Application Container
# Multi-stage build for minimal, secure runtime image
 
# Stage 1: Build environment
FROM node:20-alpine AS builder
 
WORKDIR /app
 
# Install dependencies first (layer caching)
COPY package*.json ./
RUN npm ci --only=production
 
# Copy source and build
COPY src/ ./src/
COPY tsconfig.json ./
RUN npm run build
 
# Prune dev dependencies
RUN npm prune --production
 
# Stage 2: Runtime environment
FROM node:20-alpine AS runtime
 
# Security: Run as non-root user
RUN addgroup -g 1001 appgroup && \
    adduser -u 1001 -G appgroup -s /bin/sh -D appuser
 
# Install runtime dependencies only
RUN apk add --no-cache dumb-init
 
WORKDIR /app
 
# Copy only necessary artifacts from builder
COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/package.json ./
 
# Application configuration (non-sensitive)
ENV NODE_ENV=production
ENV PORT=8080
 
# Health check for orchestrator
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
 
# Drop to non-root user
USER appuser
 
# Expose application port
EXPOSE 8080
 
# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "dist/server.js"]

kubernetes-deployment.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
# Kubernetes: Deploying Immutable Containers
# Immutability enforced via image tags and read-only filesystem
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: production
  labels:
    app: myapp
    version: v2.3.1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
        version: v2.3.1
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      # Security context for immutability
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      
      containers:
      - name: app
        # Immutable: Pinned image digest (not :latest)
        image: registry.company.com/app@sha256:abc123def456...
        imagePullPolicy: Always
        
        ports:
        - containerPort: 8080
          protocol: TCP
        
        # Environment from ConfigMap and Secrets
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets
        
        # Resource limits
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        
        # Enforce read-only root filesystem
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
              - ALL
        
        # Writable volumes for temporary data only
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/cache
        
        # Probes for lifecycle management
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
      
      # Ephemeral volumes for runtime data
      volumes:
      - name: tmp
        emptyDir:
          medium: Memory
          sizeLimit: "64Mi"
      - name: cache
        emptyDir:
          sizeLimit: "256Mi"

Image Tagging Best Practices

Hybrid Approaches: Practical Reality

The Spectrum of Mutability

Rather than a binary choice, think of mutability as a spectrum:

Infrastructure Mutability Spectrum
Category	Mutability	Examples	Rationale
Stateless Compute	Fully Immutable	Web servers, API servers, workers	No persistent state; replacement is trivial
Application Config	Semi-Immutable	Feature flags, log levels	Runtime toggles via ConfigMaps; pod unchanged
Databases (Primary)	Mutable with Controls	PostgreSQL, MySQL primaries	State is the point; in-place updates necessary
Databases (Replicas)	More Immutable	Read replicas, analytics DBs	Can rebuild from primary; replacement viable
Network Infrastructure	Stable/Long-lived	VPCs, subnets, routes	Foundation layer; changes rare and deliberate
Secrets/Certs	Externally Managed	Vault secrets, TLS certificates	Rotated independently; injected at runtime

Practical Hybrid Patterns

Immutable Compute, Mutable Data: The most common pattern. Compute instances are immutable; data lives in managed services (RDS, S3) or dedicated persistent volumes.
Immutable Base, Mutable Layer: The base image is immutable, but runtime configuration is injected at startup via environment variables, ConfigMaps, or parameter stores.
Immutable Container, Mutable Sidecar: The main application container is immutable; a mutable sidecar handles dynamic concerns (log shipping, metrics, config reload).
Periodic Refresh: Mutable servers are periodically terminated and replaced (weekly, monthly) to limit drift accumulation without full immutability.

hybrid-architecture.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# Hybrid Architecture: Immutable Compute + Mutable Config + Persistent Data
#
# This pattern separates concerns across the mutability spectrum
 
# IMMUTABLE LAYER: Application containers
# - Built once, deployed identically everywhere
# - Version-pinned image references
# - Read-only root filesystem
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: api
        image: registry/api:v3.2.1-sha256abc  # Immutable reference
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - name: config
          mountPath: /app/config
          readOnly: true  # Config is read-only to container
        - name: secrets
          mountPath: /app/secrets
          readOnly: true
      
      volumes:
      # SEMI-MUTABLE LAYER: ConfigMaps
      # - Changes require pod restart but not rebuild
      # - Version controlled in Git
      - name: config
        configMap:
          name: api-config  # Can be updated, pod sees new version on restart
      
      # EXTERNALLY MUTABLE: Secrets
      # - Managed by Vault/External Secrets Operator
      # - Rotated independently of application
      - name: secrets
        secret:
          secretName: api-secrets
 
---
# MUTABLE/PERSISTENT LAYER: StatefulSet for databases
# - Cannot be simply replaced; state is critical
# - In-place updates necessary
# - Strong backup/restore procedures critical
 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3  # Primary + 2 replicas
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:15.4  # Version pinned, but pod is long-lived
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-encrypted
      resources:
        requests:
          storage: 500Gi  # Mutable, persistent storage

The Data Plane vs Control Plane Distinction

Migrating to Immutable Infrastructure

Transitioning from mutable to immutable infrastructure is a journey, not a switch. Organizations typically move incrementally, starting with new workloads and gradually migrating existing systems.

Migration Phases

Phased Migration Approach

•Phase 0: Foundation — Establish image building pipelines (Packer/Docker), container registries, and deployment automation (Terraform/Kubernetes). Run parallel to existing infrastructure.
•Phase 1: New Workloads — Deploy all new applications as immutable from the start. Build organizational muscle without risking existing systems.
•Phase 2: Stateless Services — Migrate existing stateless services (web frontends, API servers, workers) to immutable infrastructure. These have the lowest risk.
•Phase 3: Stateless with State Dependencies — Migrate services that depend on stateful backends. Requires careful data layer planning and externalizing state.
•Phase 4: Evaluate Stateful — Assess stateful workloads. Some may remain mutable; others may migrate to managed services (RDS vs self-managed Postgres).
•Phase 5: Continuous Improvement — Refine pipelines, reduce image build times, improve rollback procedures, enhance monitoring.

Common Migration Challenges

Migration Challenges and Mitigations
Challenge	Description	Mitigation Strategy
Configuration in code	Hardcoded values in application expecting local files	Externalize to environment variables, ConfigMaps, or parameter stores
Local file dependencies	Applications writing to local filesystem	Use object storage (S3), EBS volumes, or ephemeral volumes
SSH access expectations	Teams accustomed to debugging via SSH	Implement centralized logging, tracing, and metrics. Kubectl exec for containers
Slow build times	Long image builds slow deployment velocity	Optimize Dockerfiles, use layer caching, parallelize builds
Stateful application patterns	Applications designed for persistent servers	Refactor to externalize state or accept managed mutability
Cultural resistance	Teams uncomfortable with new mental model	Training, pair programming, celebrate early wins, document runbooks

The 'Just This Once' Trap

Summary: Choosing Your Infrastructure Paradigm

We've explored the fundamental choice between mutable and immutable infrastructure paradigms. Let's consolidate the key insights:

Key Takeaways

•Mutable infrastructure modifies in-place — Long-lived servers accumulate changes. Configuration management maintains convergence but drift remains a constant battle.
•Immutable infrastructure replaces rather than modifies — Every change produces new artifacts. Consistency is guaranteed by construction, not enforcement.
•Configuration drift is the primary failure mode of mutable infrastructure — Servers diverge subtly, causing inconsistent behavior, failed deployments, and debugging nightmares.
•Immutable infrastructure eliminates drift by design — No modification mechanism means no drift. Images are versioned, reproducible, auditable artifacts.
•Pure immutability isn't always practical — Databases, message queues, and some legacy systems require mutable patterns. Hybrid approaches are common.
•The key is intentionality — Understand which components should be mutable and which immutable. Establish clear boundaries and workflows for each.
•Migration is incremental — Start with new workloads, progress through stateless services, evaluate stateful workloads individually.

What's Next:

Page Complete

2 / 5