System Design (HLD)Kubernetes Operations

Kubernetes Operations

LevelAdvanced

Duration90 mins

TopicKubernetes Operations

3 / 5

Rolling Updates and Rollbacks

The Art of Zero-Downtime Deployments

Deploying software updates in production is one of the most critical—and potentially dangerous—operations in system administration. A misconfigured deployment can take down a live system serving millions of users. A slow rollout can leave users on inconsistent versions. A botched rollback can make a bad situation catastrophic.

Kubernetes provides sophisticated deployment machinery that, when properly configured, enables zero-downtime deployments where users experience no interruption during updates. But these tools are only as good as your understanding of them.

This page covers:

Rolling update mechanics and configuration
Rollback strategies and revision history
Advanced patterns: blue-green, canary, progressive delivery
Pod disruption budgets for safe updates
Readiness gates and startup probes
Argo Rollouts and Flagger for sophisticated deployments

What You Will Learn

By the end of this page, you'll understand how to configure rolling updates that eliminate user-facing downtime, implement instant rollbacks when deployments fail, design canary and blue-green strategies for risk mitigation, and use progressive delivery tools for production-grade release management.

Rolling Update Fundamentals

A rolling update incrementally replaces old pods with new pods, ensuring that some instances of the application are always available during the transition. This is Kubernetes' default deployment strategy.

How Rolling Updates Work:

You update the Deployment spec (typically the container image tag)
Deployment controller detects the change and creates a new ReplicaSet
Controller scales up the new ReplicaSet and scales down the old ReplicaSet
The pace is controlled by maxSurge and maxUnavailable
Process continues until all pods are running the new version
Old ReplicaSet is kept (with 0 replicas) for rollback capability

The Critical Parameters:

maxSurge: Maximum number of pods that can be created above desired replica count during update
maxUnavailable: Maximum number of pods that can be unavailable during update

These parameters work in tension: higher values mean faster updates but more resource consumption (maxSurge) or more risk (maxUnavailable).

deployment-rolling-update.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-application
  namespace: production
spec:
  replicas: 10
  
  # Rolling update strategy configuration
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Maximum pods above replica count during update
      maxSurge: 25%          # Can also be absolute: "2"
      
      # Maximum pods unavailable during update
      maxUnavailable: 25%    # Can also be absolute: "0" for zero-downtime
  
  # How many old ReplicaSets to keep for rollback
  revisionHistoryLimit: 10
  
  # Time to wait before considering pod update failed
  progressDeadlineSeconds: 600    # 10 minutes
  
  # Minimum time to wait before marking pod Ready
  minReadySeconds: 10
  
  selector:
    matchLabels:
      app: web-application
      
  template:
    metadata:
      labels:
        app: web-application
        version: v2.1.0
    spec:
      containers:
      - name: web
        image: myregistry/web-application:v2.1.0
        ports:
        - containerPort: 8080
        
        # Readiness probe is CRITICAL for rolling updates
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        
        # Liveness probe detects stuck containers
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3

maxSurge and maxUnavailable Strategy Patterns
Pattern	maxSurge	maxUnavailable	Behavior	Use Case
Conservative	1	0	One new pod at a time, never fewer than desired	Critical services, limited capacity
Balanced (default)	25%	25%	Update 25% at a time	General production workloads
Aggressive	100%	0	Double capacity, then cut old pods	Fast updates with spare capacity
Fast but risky	0	50%	Replace half immediately	Dev environments, fast iteration

Readiness Probes: The Gatekeeper of Rolling Updates

Readiness probes are the most critical component of safe rolling updates. Without proper readiness configuration, Kubernetes cannot determine when new pods are truly ready to serve traffic, leading to:

Requests routed to pods still initializing
New pods marked ready before dependencies are connected
Rolling updates completing before the application is functional

How Readiness Probes Affect Rolling Updates:

New pod is created and scheduled
Container starts and begins initialization
Kubelet starts executing readiness probe
Pod remains in NotReady state until probe succeeds
Only after readiness: pod receives traffic, counts as "available"
Deployment controller only proceeds when available pods ≥ desired - maxUnavailable

The minReadySeconds Setting:

Even after readiness probe succeeds, minReadySeconds adds an additional wait period before the pod is considered fully "available". This catches issues that only manifest after a few seconds of traffic.

spec:
  minReadySeconds: 30    # Wait 30s after ready before proceeding

comprehensive-probes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-app
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0     # Zero downtime
  minReadySeconds: 30        # Extra stabilization time
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        
        # STARTUP PROBE: For slow-starting applications
        # Only checked at startup, replaces liveness until success
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 5
          failureThreshold: 30    # Allow 2.5 minutes for startup
          
        # LIVENESS PROBE: Is the container stuck?
        # Failure = container restart
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 0   # Starts after startup probe succeeds
          periodSeconds: 10
          failureThreshold: 3      # Restart after 3 consecutive failures
          timeoutSeconds: 5
          
        # READINESS PROBE: Can the container serve traffic?
        # Failure = removed from service endpoints
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1     # One success = ready
          failureThreshold: 3     # Three failures = not ready
          timeoutSeconds: 3
          
        # Resources for predictable startup time
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "1000m"

Common Readiness Probe Mistakes

Too-fast successThreshold: Default is 1, but for critical services, consider 2-3 consecutive successes. 2. Not checking dependencies: If your app needs a database, the readiness endpoint should verify the connection. 3. Liveness = Readiness: These serve different purposes; don't use the same endpoint for both unless semantics truly match. 4. Missing startup probe: For JVM/Node apps with slow starts, startup probe prevents readiness probe from failing during initialization.

Rollback Mechanics: Recovering from Failed Deployments

When a deployment goes wrong, the ability to quickly rollback can mean the difference between a minor incident and a major outage. Kubernetes provides robust rollback capabilities through revision history.

How Revision History Works:

Every time you update a Deployment and it creates a new ReplicaSet, Kubernetes stores that ReplicaSet as a "revision". By default, the last 10 revisions are kept (revisionHistoryLimit: 10).

Rollback is not re-deployment—it's instant:

When you roll back, Kubernetes doesn't rebuild images or re-run pipelines. It simply reactivates an existing ReplicaSet that was already proven to work. This makes rollback extremely fast.

Triggering Rollback:

rollback-commands.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/bin/bash
# View rollout history
kubectl rollout history deployment/web-application
 
# Sample output:
# REVISION  CHANGE-CAUSE
# 1         Initial deployment
# 2         kubectl set image deployment/web-application web=myapp:v1.1
# 3         kubectl set image deployment/web-application web=myapp:v1.2
# 4         kubectl set image deployment/web-application web=myapp:v1.3
 
# View details of a specific revision
kubectl rollout history deployment/web-application --revision=2
 
# Rollback to the previous revision
kubectl rollout undo deployment/web-application
 
# Rollback to a specific revision
kubectl rollout undo deployment/web-application --to-revision=2
 
# Check rollout status
kubectl rollout status deployment/web-application
 
# Pause a rollout (useful if you spot problems mid-update)
kubectl rollout pause deployment/web-application
 
# Resume a paused rollout
kubectl rollout resume deployment/web-application
 
# Restart all pods (triggers rolling restart with same image)
kubectl rollout restart deployment/web-application
 
# View current ReplicaSets (each is a revision)
kubectl get replicasets -l app=web-application
 
# Pro tip: Annotate deployments for useful history
kubectl annotate deployment/web-application \
  kubernetes.io/change-cause="Deploy v1.4 - added caching feature"

Automatic Rollback with progressDeadlineSeconds:

If a deployment doesn't complete within progressDeadlineSeconds, it's marked as failed. While Kubernetes doesn't automatically roll back, this status enables external tools (Argo CD, Flux) to trigger automatic rollback.

What Constitutes Progress:

New pods becoming Ready
Old pods being terminated

If neither happens for progressDeadlineSeconds, the deployment is stuck.

Manual vs Automated Rollback:

Rollback Strategies
Strategy	Trigger	Speed	Use Case
kubectl rollout undo	Manual detection	Instant	Operator-initiated rollback
GitOps revert	Git commit revert	Minutes	Infrastructure-as-code workflows
Argo Rollouts abort	Metric analysis	Seconds	Automated canary rollback
progressDeadlineSeconds	Timeout detection	Minutes	Detect stuck deployments

Rollback Best Practices

Keep sufficient revisions: Set revisionHistoryLimit high enough (10-20) to cover your typical rollback window. 2. Use change-cause annotations: Without them, revision history is cryptic. 3. Test rollback procedures: Don't wait for a crisis to discover rollback is broken. 4. Validate after rollback: Rolling back fixes symptoms but not root cause—investigate immediately.

Blue-Green Deployments

Blue-green deployment is a release strategy where you maintain two complete production environments: "blue" (current) and "green" (new). You deploy to green, verify it works, then instantly switch all traffic from blue to green.

Advantages:

Zero downtime during switch (instantaneous)
Easy rollback (switch back to blue)
Full testing with production-identical environment
No mixed-version traffic

Disadvantages:

Requires 2x resources (both environments running)
Database migrations must be backward compatible
Session state must be externalized

Native Kubernetes Blue-Green:

Kubernetes doesn't have a built-in blue-green resource, but you can implement it using labels and Service selectors:

blue-green-native.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# Blue Deployment (currently serving production traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    app: myapp
    version: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: app
        image: myapp:v1.0.0
 
---
# Green Deployment (new version, ready to receive traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    app: myapp
    version: green
spec:
  replicas: 5     # Scaled up, fully ready
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
 
---
# Production Service (selector points to current live version)
apiVersion: v1
kind: Service
metadata:
  name: app-production
spec:
  selector:
    app: myapp
    version: blue     # <-- Change to 'green' to switch traffic
  ports:
  - port: 80
    targetPort: 8080
 
---
# Preview Service (always points to new version for testing)
apiVersion: v1
kind: Service
metadata:
  name: app-preview
spec:
  selector:
    app: myapp
    version: green    # <-- Test new version before switching
  ports:
  - port: 80
    targetPort: 8080

Switching Traffic:

# Switch production traffic from blue to green
kubectl patch service app-production 
  -p '{"spec":{"selector":{"version":"green"}}}'

# Verify the switch
kubectl get service app-production -o jsonpath='{.spec.selector.version}'

# If problems, switch back instantly
kubectl patch service app-production 
  -p '{"spec":{"selector":{"version":"blue"}}}'

# After successful switch, scale down old version
kubectl scale deployment app-blue --replicas=0

Blue-Green with Ingress:

For more sophisticated traffic management, use Ingress controllers or service mesh to route traffic between versions based on headers, weights, or other criteria.

Database Considerations for Blue-Green

The hardest part of blue-green is database schema changes. Both versions must be compatible with the same schema. Use the expand-contract pattern: first deploy schema changes that work with both versions, then deploy new code, then remove old schema elements in a future release.

Canary Deployments

Canary deployments gradually shift traffic to a new version while monitoring for problems. Unlike blue-green (all-or-nothing switch), canary allows you to catch issues with minimal user impact.

Why "Canary"?

The term comes from coal mining, where canaries were used to detect dangerous gases. If the canary died, miners knew to evacuate. Similarly, a canary deployment exposes a small percentage of traffic to the new version—if it "dies" (experiences errors), you abort before affecting most users.

Canary vs Rolling Updates:

Rolling Update	Canary
Updates pods continuously	Pauses at defined traffic percentages
No traffic split control	Precise traffic routing control
Harder to abort mid-way	Easy abort by routing all to stable
Built into Kubernetes	Requires additional tooling

Native Kubernetes Canary (Basic):

You can achieve basic canary behavior by running two Deployments with different replica counts behind the same Service:

canary-native.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Stable Deployment: 90% of traffic (9 replicas)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      track: stable
  template:
    metadata:
      labels:
        app: myapp
        track: stable
    spec:
      containers:
      - name: app
        image: myapp:v1.0.0
 
---
# Canary Deployment: 10% of traffic (1 replica)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
spec:
  replicas: 1        # 1 out of 10 total = ~10% traffic
  selector:
    matchLabels:
      app: myapp
      track: canary
  template:
    metadata:
      labels:
        app: myapp
        track: canary
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
 
---
# Service routes to BOTH based on 'app' label
# Traffic distribution follows replica ratio
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp       # Matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

Limitations of Native Canary:

Traffic split is approximate (based on pod count, not true percentage)
No automated analysis—someone must monitor and decide
No gradual traffic shifting—you manually adjust replica counts
Can't route specific users consistently to canary

For production canary deployments, use Argo Rollouts, Flagger, or service mesh (Istio/Linkerd) for:

Precise traffic splitting (5%, 10%, 25%, etc.)
Automated metric analysis and rollback
Header-based routing (send specific users to canary)
Progressive delivery with defined stages

Canary Traffic Percentages

A common canary progression: 1% → 5% → 10% → 25% → 50% → 100%. Start with 1% for critical services to catch catastrophic bugs with minimal impact. Increase faster for well-tested changes in non-critical systems.

Argo Rollouts: Progressive Delivery

Argo Rollouts is a Kubernetes controller and set of CRDs that provide advanced deployment capabilities including blue-green, canary, and experimentation features that go far beyond native Kubernetes.

Key Features:

Blue-green and canary with traffic management
Automated analysis and rollback based on metrics
Integration with Prometheus, Datadog, NewRelic, etc.
Traffic shaping with Istio, Nginx, ALB Ingress, etc.
Experimentation with weighted routing
Manual promotion gates

argo-rollout-canary.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 10
  
  # Selector and template work like Deployment
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
  
  # Canary strategy with automated analysis
  strategy:
    canary:
      # Traffic routing via Nginx Ingress
      canaryService: myapp-canary
      stableService: myapp-stable
      
      # Traffic management 
      trafficRouting:
        nginx:
          stableIngress: myapp-ingress
          
      # Canary progression steps
      steps:
      # Step 1: 5% traffic for 5 minutes
      - setWeight: 5
      - pause: {duration: 5m}
      
      # Step 2: Run automated analysis
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
            
      # Step 3: If analysis passes, 25% for 10 minutes
      - setWeight: 25
      - pause: {duration: 10m}
      
      # Step 4: More analysis
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
            
      # Step 5: 50% traffic
      - setWeight: 50
      - pause: {duration: 10m}
      
      # Step 6: Final analysis before full rollout
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
      
      # Analysis configuration
      analysis:
        # How long to wait for analysis to complete
        successfulRunHistoryLimit: 3
        unsuccessfulRunHistoryLimit: 3
 
---
# AnalysisTemplate: Defines what metrics to check
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-analysis
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    # Query Prometheus for success rate
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{
            service="{{args.service-name}}",
            status=~"2.."
          }[5m])) /
          sum(rate(http_requests_total{
            service="{{args.service-name}}"
          }[5m])) * 100
    # Success criteria
    successCondition: result[0] >= 99    # 99% success rate required
    failureCondition: result[0] < 95      # Below 95% = immediate failure
    interval: 1m
    count: 5

Argo Rollouts Commands

•kubectl argo rollouts get rollout myapp - View rollout status and steps
•kubectl argo rollouts promote myapp - Manually promote to next step
•kubectl argo rollouts abort myapp - Abort rollout and revert to stable
•kubectl argo rollouts retry rollout myapp - Retry failed rollout
•kubectl argo rollouts set image myapp app=myapp:v3.0 - Trigger new rollout
•kubectl argo rollouts dashboard - Open web dashboard (port 3100)

Pod Disruption Budgets: Protecting Availability

Pod Disruption Budgets (PDBs) limit the number of pods that can be voluntarily disrupted at once. They protect against overly aggressive updates, cluster autoscaler drains, and admin maintenance operations.

What Counts as Voluntary Disruption:

Cluster autoscaler draining nodes
kubectl drain for maintenance
Deployment rolling updates (YES, they respect PDBs!)
Node upgrades

What Doesn't Count (Involuntary):

Node crashes/failures
Kernel panics
OOM kills
kubectl delete pod (bypasses PDB)

PDB Configuration:

pod-disruption-budget.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Option 1: Minimum available pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: production
spec:
  # At least 3 pods must always be available
  minAvailable: 3
  
  # Which pods this PDB protects
  selector:
    matchLabels:
      app: myapp
 
---
# Option 2: Maximum unavailable pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb-percentage
  namespace: production
spec:
  # At most 20% of pods can be unavailable
  maxUnavailable: 20%
  
  selector:
    matchLabels:
      app: myapp
 
---
# Option 3: For single-instance stateful workloads
# "Unhealthy pod eviction" allows evicting stuck pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: database-pdb
spec:
  maxUnavailable: 0          # Never disrupt voluntarily
  selector:
    matchLabels:
      app: postgres-primary
  # As of K8s 1.27, can configure unhealthy pod eviction
  unhealthyPodEvictionPolicy: AlwaysAllow   # or IfHealthyBudget

PDB Pitfalls

PDB with minAvailable = replicas blocks all updates: If you have 3 replicas and minAvailable: 3, no pod can be evicted, ever. Updates get stuck. 2. PDB doesn't apply to involuntary disruption: Node crashes ignore PDBs. 3. Multiple PDBs can conflict: Ensure overlapping PDBs are consistent. 4. maxUnavailable: 0 requires unhealthyPodEvictionPolicy: Otherwise stuck unhealthy pods can never be evicted.

PDB Strategy Guidelines
Workload Type	Recommended PDB	Rationale
Stateless, 3+ replicas	maxUnavailable: 1 or 25%	Allows gradual updates while maintaining quorum
Database primary	maxUnavailable: 0	Never voluntarily disrupt the primary
Database replicas	maxUnavailable: 1	Maintain read capacity during maintenance
Kafka brokers	minAvailable: N-1 (where N = replication factor)	Maintain quorum for partitions
Stateful with 2 replicas	maxUnavailable: 1	Allow rolling updates one at a time

Graceful Shutdown: Completing In-Flight Requests

For truly zero-downtime deployments, pods must handle termination gracefully—finishing in-flight requests before exiting. Kubernetes provides a termination lifecycle, but your application must cooperate.

Pod Termination Sequence:

Pod marked for termination (deletion, scale-down, etc.)
Pod enters "Terminating" state
Simultaneously:
- Pod removed from Service endpoints (stops receiving new traffic)
- Container receives SIGTERM
Container is expected to shutdown gracefully
After terminationGracePeriodSeconds, SIGKILL sent (force kill)
Pod resources released

The Race Condition Problem:

Step 3's operations happen in parallel but aren't atomic. The pod might receive new requests after SIGTERM was sent but before endpoints update propagates to all load balancers. This causes errors for users connecting to a dying pod.

The PreStop Hook Solution:

graceful-shutdown.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
spec:
  template:
    spec:
      # Give pods plenty of time to drain
      terminationGracePeriodSeconds: 60
      
      containers:
      - name: web
        image: myapp:latest
        
        lifecycle:
          preStop:
            exec:
              # Wait for endpoints to update before stopping
              command:
              - /bin/sh
              - -c
              - |
                echo "Starting graceful shutdown..."
                sleep 15    # Wait for LB to stop sending traffic
                echo "Draining connections..."
                # Or call application-specific drain endpoint
                # curl -X POST localhost:8080/admin/drain
        
        # Application must handle SIGTERM
        # Most frameworks do this automatically:
        # - Node.js: process.on('SIGTERM', handler)
        # - Java: Runtime.addShutdownHook()
        # - Go: signal.Notify()
        
      # For sidecars that should stop last
      - name: sidecar
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 20"]

Complete Graceful Shutdown Checklist:

Set adequate terminationGracePeriodSeconds: Default is 30s, but long-running requests may need 60-120s
Add preStop hook with sleep: 10-15 seconds allows endpoints to update and LBs to drain
Application handles SIGTERM: Stop accepting new connections, finish in-flight requests
Return 5xx from health checks during shutdown: Some apps mark themselves unhealthy to speed up removal from load balancers
Set proper timeouts: Connection read/write timeouts should be less than terminationGracePeriodSeconds

Cloud Load Balancer Considerations:

Cloud load balancers (ALB, NLB, GCP LB) have their own health check and deregistration delays. Configure:

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=30
Coordinate this with your preStop hook delay

Testing Graceful Shutdown

Test graceful shutdown during load! Many issues only appear under real traffic. Use tools like wrk or k6 to generate load, then trigger a rolling update and watch for error spikes.

Summary: Deployment Best Practices

Safe, reliable deployments are the result of careful configuration and thorough testing. Let's consolidate the key principles:

Key Takeaways

•Use maxUnavailable: 0 for zero downtime: Ensures new pods are ready before old pods terminate
•Readiness probes are mandatory: Without them, Kubernetes can't determine when pods are truly ready
•Add startup probes for slow-starting apps: Prevents liveness failures during initialization
•Configure minReadySeconds: Adds buffer to catch post-ready failures (10-30 seconds typical)
•Keep revision history for rollbacks: Set revisionHistoryLimit to 10-20 for adequate rollback window
•Use PodDisruptionBudgets: Protect against overly aggressive updates and drains
•Implement graceful shutdown: preStop hooks + SIGTERM handling prevents dropped requests
•Consider canary for risky changes: Gradual traffic shifting catches issues with minimal impact
•Use Argo Rollouts for sophisticated deployments: Automated analysis, precise traffic control, easy rollback
•Test deployment procedures under load: Many issues only appear with real traffic

What's Next:

With deployment strategies mastered, the next page covers Monitoring and Logging—essential observability practices that let you understand what's happening in your cluster, detect problems before they become outages, and debug issues when they occur.

Page Complete

You now have comprehensive knowledge of Kubernetes deployment strategies—from rolling updates and rollbacks through blue-green, canary, and progressive delivery with Argo Rollouts. Apply these patterns to achieve reliable, zero-downtime releases in production.

3 / 5

Loading learning content...

System Design (HLD)Kubernetes Operations

Kubernetes Operations

LevelAdvanced

Duration90 mins

TopicKubernetes Operations

3 / 5

Rolling Updates and Rollbacks

The Art of Zero-Downtime Deployments

This page covers:

Rolling update mechanics and configuration
Rollback strategies and revision history
Advanced patterns: blue-green, canary, progressive delivery
Pod disruption budgets for safe updates
Readiness gates and startup probes
Argo Rollouts and Flagger for sophisticated deployments

What You Will Learn

Rolling Update Fundamentals

How Rolling Updates Work:

You update the Deployment spec (typically the container image tag)
Deployment controller detects the change and creates a new ReplicaSet
Controller scales up the new ReplicaSet and scales down the old ReplicaSet
The pace is controlled by maxSurge and maxUnavailable
Process continues until all pods are running the new version
Old ReplicaSet is kept (with 0 replicas) for rollback capability

The Critical Parameters:

maxSurge: Maximum number of pods that can be created above desired replica count during update
maxUnavailable: Maximum number of pods that can be unavailable during update

These parameters work in tension: higher values mean faster updates but more resource consumption (maxSurge) or more risk (maxUnavailable).

deployment-rolling-update.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-application
  namespace: production
spec:
  replicas: 10
  
  # Rolling update strategy configuration
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Maximum pods above replica count during update
      maxSurge: 25%          # Can also be absolute: "2"
      
      # Maximum pods unavailable during update
      maxUnavailable: 25%    # Can also be absolute: "0" for zero-downtime
  
  # How many old ReplicaSets to keep for rollback
  revisionHistoryLimit: 10
  
  # Time to wait before considering pod update failed
  progressDeadlineSeconds: 600    # 10 minutes
  
  # Minimum time to wait before marking pod Ready
  minReadySeconds: 10
  
  selector:
    matchLabels:
      app: web-application
      
  template:
    metadata:
      labels:
        app: web-application
        version: v2.1.0
    spec:
      containers:
      - name: web
        image: myregistry/web-application:v2.1.0
        ports:
        - containerPort: 8080
        
        # Readiness probe is CRITICAL for rolling updates
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        
        # Liveness probe detects stuck containers
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10
          failureThreshold: 3

maxSurge and maxUnavailable Strategy Patterns
Pattern	maxSurge	maxUnavailable	Behavior	Use Case
Conservative	1	0	One new pod at a time, never fewer than desired	Critical services, limited capacity
Balanced (default)	25%	25%	Update 25% at a time	General production workloads
Aggressive	100%	0	Double capacity, then cut old pods	Fast updates with spare capacity
Fast but risky	0	50%	Replace half immediately	Dev environments, fast iteration

Readiness Probes: The Gatekeeper of Rolling Updates

Requests routed to pods still initializing
New pods marked ready before dependencies are connected
Rolling updates completing before the application is functional

How Readiness Probes Affect Rolling Updates:

New pod is created and scheduled
Container starts and begins initialization
Kubelet starts executing readiness probe
Pod remains in NotReady state until probe succeeds
Only after readiness: pod receives traffic, counts as "available"
Deployment controller only proceeds when available pods ≥ desired - maxUnavailable

The minReadySeconds Setting:

spec:
  minReadySeconds: 30    # Wait 30s after ready before proceeding

comprehensive-probes.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-app
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0     # Zero downtime
  minReadySeconds: 30        # Extra stabilization time
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        
        # STARTUP PROBE: For slow-starting applications
        # Only checked at startup, replaces liveness until success
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 5
          failureThreshold: 30    # Allow 2.5 minutes for startup
          
        # LIVENESS PROBE: Is the container stuck?
        # Failure = container restart
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 0   # Starts after startup probe succeeds
          periodSeconds: 10
          failureThreshold: 3      # Restart after 3 consecutive failures
          timeoutSeconds: 5
          
        # READINESS PROBE: Can the container serve traffic?
        # Failure = removed from service endpoints
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1     # One success = ready
          failureThreshold: 3     # Three failures = not ready
          timeoutSeconds: 3
          
        # Resources for predictable startup time
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "1000m"

Common Readiness Probe Mistakes

Too-fast successThreshold: Default is 1, but for critical services, consider 2-3 consecutive successes. 2. Not checking dependencies: If your app needs a database, the readiness endpoint should verify the connection. 3. Liveness = Readiness: These serve different purposes; don't use the same endpoint for both unless semantics truly match. 4. Missing startup probe: For JVM/Node apps with slow starts, startup probe prevents readiness probe from failing during initialization.

Rollback Mechanics: Recovering from Failed Deployments

How Revision History Works:

Every time you update a Deployment and it creates a new ReplicaSet, Kubernetes stores that ReplicaSet as a "revision". By default, the last 10 revisions are kept (revisionHistoryLimit: 10).

Rollback is not re-deployment—it's instant:

When you roll back, Kubernetes doesn't rebuild images or re-run pipelines. It simply reactivates an existing ReplicaSet that was already proven to work. This makes rollback extremely fast.

Triggering Rollback:

rollback-commands.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/bin/bash
# View rollout history
kubectl rollout history deployment/web-application
 
# Sample output:
# REVISION  CHANGE-CAUSE
# 1         Initial deployment
# 2         kubectl set image deployment/web-application web=myapp:v1.1
# 3         kubectl set image deployment/web-application web=myapp:v1.2
# 4         kubectl set image deployment/web-application web=myapp:v1.3
 
# View details of a specific revision
kubectl rollout history deployment/web-application --revision=2
 
# Rollback to the previous revision
kubectl rollout undo deployment/web-application
 
# Rollback to a specific revision
kubectl rollout undo deployment/web-application --to-revision=2
 
# Check rollout status
kubectl rollout status deployment/web-application
 
# Pause a rollout (useful if you spot problems mid-update)
kubectl rollout pause deployment/web-application
 
# Resume a paused rollout
kubectl rollout resume deployment/web-application
 
# Restart all pods (triggers rolling restart with same image)
kubectl rollout restart deployment/web-application
 
# View current ReplicaSets (each is a revision)
kubectl get replicasets -l app=web-application
 
# Pro tip: Annotate deployments for useful history
kubectl annotate deployment/web-application \
  kubernetes.io/change-cause="Deploy v1.4 - added caching feature"

Automatic Rollback with progressDeadlineSeconds:

What Constitutes Progress:

New pods becoming Ready
Old pods being terminated

If neither happens for progressDeadlineSeconds, the deployment is stuck.

Manual vs Automated Rollback:

Rollback Strategies
Strategy	Trigger	Speed	Use Case
kubectl rollout undo	Manual detection	Instant	Operator-initiated rollback
GitOps revert	Git commit revert	Minutes	Infrastructure-as-code workflows
Argo Rollouts abort	Metric analysis	Seconds	Automated canary rollback
progressDeadlineSeconds	Timeout detection	Minutes	Detect stuck deployments

Rollback Best Practices

Keep sufficient revisions: Set revisionHistoryLimit high enough (10-20) to cover your typical rollback window. 2. Use change-cause annotations: Without them, revision history is cryptic. 3. Test rollback procedures: Don't wait for a crisis to discover rollback is broken. 4. Validate after rollback: Rolling back fixes symptoms but not root cause—investigate immediately.

Blue-Green Deployments

Advantages:

Zero downtime during switch (instantaneous)
Easy rollback (switch back to blue)
Full testing with production-identical environment
No mixed-version traffic

Disadvantages:

Requires 2x resources (both environments running)
Database migrations must be backward compatible
Session state must be externalized

Native Kubernetes Blue-Green:

Kubernetes doesn't have a built-in blue-green resource, but you can implement it using labels and Service selectors:

blue-green-native.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# Blue Deployment (currently serving production traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    app: myapp
    version: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: app
        image: myapp:v1.0.0
 
---
# Green Deployment (new version, ready to receive traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    app: myapp
    version: green
spec:
  replicas: 5     # Scaled up, fully ready
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
 
---
# Production Service (selector points to current live version)
apiVersion: v1
kind: Service
metadata:
  name: app-production
spec:
  selector:
    app: myapp
    version: blue     # <-- Change to 'green' to switch traffic
  ports:
  - port: 80
    targetPort: 8080
 
---
# Preview Service (always points to new version for testing)
apiVersion: v1
kind: Service
metadata:
  name: app-preview
spec:
  selector:
    app: myapp
    version: green    # <-- Test new version before switching
  ports:
  - port: 80
    targetPort: 8080

Switching Traffic:

# Switch production traffic from blue to green
kubectl patch service app-production 
  -p '{"spec":{"selector":{"version":"green"}}}'

# Verify the switch
kubectl get service app-production -o jsonpath='{.spec.selector.version}'

# If problems, switch back instantly
kubectl patch service app-production 
  -p '{"spec":{"selector":{"version":"blue"}}}'

# After successful switch, scale down old version
kubectl scale deployment app-blue --replicas=0

Blue-Green with Ingress:

For more sophisticated traffic management, use Ingress controllers or service mesh to route traffic between versions based on headers, weights, or other criteria.

Database Considerations for Blue-Green

Canary Deployments

Canary deployments gradually shift traffic to a new version while monitoring for problems. Unlike blue-green (all-or-nothing switch), canary allows you to catch issues with minimal user impact.

Why "Canary"?

Canary vs Rolling Updates:

Rolling Update	Canary
Updates pods continuously	Pauses at defined traffic percentages
No traffic split control	Precise traffic routing control
Harder to abort mid-way	Easy abort by routing all to stable
Built into Kubernetes	Requires additional tooling

Native Kubernetes Canary (Basic):

You can achieve basic canary behavior by running two Deployments with different replica counts behind the same Service:

canary-native.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# Stable Deployment: 90% of traffic (9 replicas)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      track: stable
  template:
    metadata:
      labels:
        app: myapp
        track: stable
    spec:
      containers:
      - name: app
        image: myapp:v1.0.0
 
---
# Canary Deployment: 10% of traffic (1 replica)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
spec:
  replicas: 1        # 1 out of 10 total = ~10% traffic
  selector:
    matchLabels:
      app: myapp
      track: canary
  template:
    metadata:
      labels:
        app: myapp
        track: canary
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
 
---
# Service routes to BOTH based on 'app' label
# Traffic distribution follows replica ratio
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp       # Matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

Limitations of Native Canary:

Traffic split is approximate (based on pod count, not true percentage)
No automated analysis—someone must monitor and decide
No gradual traffic shifting—you manually adjust replica counts
Can't route specific users consistently to canary

For production canary deployments, use Argo Rollouts, Flagger, or service mesh (Istio/Linkerd) for:

Precise traffic splitting (5%, 10%, 25%, etc.)
Automated metric analysis and rollback
Header-based routing (send specific users to canary)
Progressive delivery with defined stages

Canary Traffic Percentages

Argo Rollouts: Progressive Delivery

Key Features:

Blue-green and canary with traffic management
Automated analysis and rollback based on metrics
Integration with Prometheus, Datadog, NewRelic, etc.
Traffic shaping with Istio, Nginx, ALB Ingress, etc.
Experimentation with weighted routing
Manual promotion gates

argo-rollout-canary.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 10
  
  # Selector and template work like Deployment
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
  
  # Canary strategy with automated analysis
  strategy:
    canary:
      # Traffic routing via Nginx Ingress
      canaryService: myapp-canary
      stableService: myapp-stable
      
      # Traffic management 
      trafficRouting:
        nginx:
          stableIngress: myapp-ingress
          
      # Canary progression steps
      steps:
      # Step 1: 5% traffic for 5 minutes
      - setWeight: 5
      - pause: {duration: 5m}
      
      # Step 2: Run automated analysis
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
            
      # Step 3: If analysis passes, 25% for 10 minutes
      - setWeight: 25
      - pause: {duration: 10m}
      
      # Step 4: More analysis
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
            
      # Step 5: 50% traffic
      - setWeight: 50
      - pause: {duration: 10m}
      
      # Step 6: Final analysis before full rollout
      - analysis:
          templates:
          - templateName: success-rate-analysis
          args:
          - name: service-name
            value: myapp-canary
      
      # Analysis configuration
      analysis:
        # How long to wait for analysis to complete
        successfulRunHistoryLimit: 3
        unsuccessfulRunHistoryLimit: 3
 
---
# AnalysisTemplate: Defines what metrics to check
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-analysis
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    # Query Prometheus for success rate
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{
            service="{{args.service-name}}",
            status=~"2.."
          }[5m])) /
          sum(rate(http_requests_total{
            service="{{args.service-name}}"
          }[5m])) * 100
    # Success criteria
    successCondition: result[0] >= 99    # 99% success rate required
    failureCondition: result[0] < 95      # Below 95% = immediate failure
    interval: 1m
    count: 5

Argo Rollouts Commands

•kubectl argo rollouts get rollout myapp - View rollout status and steps
•kubectl argo rollouts promote myapp - Manually promote to next step
•kubectl argo rollouts abort myapp - Abort rollout and revert to stable
•kubectl argo rollouts retry rollout myapp - Retry failed rollout
•kubectl argo rollouts set image myapp app=myapp:v3.0 - Trigger new rollout
•kubectl argo rollouts dashboard - Open web dashboard (port 3100)

Pod Disruption Budgets: Protecting Availability

What Counts as Voluntary Disruption:

Cluster autoscaler draining nodes
kubectl drain for maintenance
Deployment rolling updates (YES, they respect PDBs!)
Node upgrades

What Doesn't Count (Involuntary):

Node crashes/failures
Kernel panics
OOM kills
kubectl delete pod (bypasses PDB)

PDB Configuration:

pod-disruption-budget.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Option 1: Minimum available pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: production
spec:
  # At least 3 pods must always be available
  minAvailable: 3
  
  # Which pods this PDB protects
  selector:
    matchLabels:
      app: myapp
 
---
# Option 2: Maximum unavailable pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb-percentage
  namespace: production
spec:
  # At most 20% of pods can be unavailable
  maxUnavailable: 20%
  
  selector:
    matchLabels:
      app: myapp
 
---
# Option 3: For single-instance stateful workloads
# "Unhealthy pod eviction" allows evicting stuck pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: database-pdb
spec:
  maxUnavailable: 0          # Never disrupt voluntarily
  selector:
    matchLabels:
      app: postgres-primary
  # As of K8s 1.27, can configure unhealthy pod eviction
  unhealthyPodEvictionPolicy: AlwaysAllow   # or IfHealthyBudget

PDB Pitfalls

PDB with minAvailable = replicas blocks all updates: If you have 3 replicas and minAvailable: 3, no pod can be evicted, ever. Updates get stuck. 2. PDB doesn't apply to involuntary disruption: Node crashes ignore PDBs. 3. Multiple PDBs can conflict: Ensure overlapping PDBs are consistent. 4. maxUnavailable: 0 requires unhealthyPodEvictionPolicy: Otherwise stuck unhealthy pods can never be evicted.

PDB Strategy Guidelines
Workload Type	Recommended PDB	Rationale
Stateless, 3+ replicas	maxUnavailable: 1 or 25%	Allows gradual updates while maintaining quorum
Database primary	maxUnavailable: 0	Never voluntarily disrupt the primary
Database replicas	maxUnavailable: 1	Maintain read capacity during maintenance
Kafka brokers	minAvailable: N-1 (where N = replication factor)	Maintain quorum for partitions
Stateful with 2 replicas	maxUnavailable: 1	Allow rolling updates one at a time

Graceful Shutdown: Completing In-Flight Requests

Pod Termination Sequence:

Pod marked for termination (deletion, scale-down, etc.)
Pod enters "Terminating" state
Simultaneously:
- Pod removed from Service endpoints (stops receiving new traffic)
- Container receives SIGTERM
Container is expected to shutdown gracefully
After terminationGracePeriodSeconds, SIGKILL sent (force kill)
Pod resources released

The Race Condition Problem:

The PreStop Hook Solution:

graceful-shutdown.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
spec:
  template:
    spec:
      # Give pods plenty of time to drain
      terminationGracePeriodSeconds: 60
      
      containers:
      - name: web
        image: myapp:latest
        
        lifecycle:
          preStop:
            exec:
              # Wait for endpoints to update before stopping
              command:
              - /bin/sh
              - -c
              - |
                echo "Starting graceful shutdown..."
                sleep 15    # Wait for LB to stop sending traffic
                echo "Draining connections..."
                # Or call application-specific drain endpoint
                # curl -X POST localhost:8080/admin/drain
        
        # Application must handle SIGTERM
        # Most frameworks do this automatically:
        # - Node.js: process.on('SIGTERM', handler)
        # - Java: Runtime.addShutdownHook()
        # - Go: signal.Notify()
        
      # For sidecars that should stop last
      - name: sidecar
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 20"]

Complete Graceful Shutdown Checklist:

Set adequate terminationGracePeriodSeconds: Default is 30s, but long-running requests may need 60-120s
Add preStop hook with sleep: 10-15 seconds allows endpoints to update and LBs to drain
Application handles SIGTERM: Stop accepting new connections, finish in-flight requests
Return 5xx from health checks during shutdown: Some apps mark themselves unhealthy to speed up removal from load balancers
Set proper timeouts: Connection read/write timeouts should be less than terminationGracePeriodSeconds

Cloud Load Balancer Considerations:

Cloud load balancers (ALB, NLB, GCP LB) have their own health check and deregistration delays. Configure:

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=30
Coordinate this with your preStop hook delay

Testing Graceful Shutdown

Test graceful shutdown during load! Many issues only appear under real traffic. Use tools like wrk or k6 to generate load, then trigger a rolling update and watch for error spikes.

Summary: Deployment Best Practices

Safe, reliable deployments are the result of careful configuration and thorough testing. Let's consolidate the key principles:

Key Takeaways

•Use maxUnavailable: 0 for zero downtime: Ensures new pods are ready before old pods terminate
•Readiness probes are mandatory: Without them, Kubernetes can't determine when pods are truly ready
•Add startup probes for slow-starting apps: Prevents liveness failures during initialization
•Configure minReadySeconds: Adds buffer to catch post-ready failures (10-30 seconds typical)
•Keep revision history for rollbacks: Set revisionHistoryLimit to 10-20 for adequate rollback window
•Use PodDisruptionBudgets: Protect against overly aggressive updates and drains
•Implement graceful shutdown: preStop hooks + SIGTERM handling prevents dropped requests
•Consider canary for risky changes: Gradual traffic shifting catches issues with minimal impact
•Use Argo Rollouts for sophisticated deployments: Automated analysis, precise traffic control, easy rollback
•Test deployment procedures under load: Many issues only appear with real traffic

What's Next:

Page Complete

3 / 5