Operating SystemsVirtualization

Containers

LevelIntermediate

Duration90 mins

TopicVirtualization

5 / 5

Container Orchestration

Managing Containers at Scale

Running a single container on your laptop is easy. Running hundreds or thousands of containers across dozens of servers, each needing to be scheduled, scaled, networked, monitored, and self-healed—that's container orchestration.

As organizations adopt microservices and containerized deployments, the complexity explodes. Which server should run each container? How do containers discover and communicate with each other? What happens when a server fails? How do you deploy updates without downtime? These questions drive the need for orchestration platforms.

Kubernetes has become the de facto standard for container orchestration, but understanding the underlying problems and concepts applies across orchestration platforms. This knowledge is essential for any engineer working with production containerized systems.

What You Will Learn

By the end of this page, you will understand why container orchestration is necessary, the core concepts and challenges it addresses, Kubernetes architecture and its key components, essential abstractions like Pods, Services, and Deployments, and common orchestration patterns used in production systems.

Why Orchestration is Needed

Container orchestration solves problems that emerge when containers are deployed at scale. While a single Docker host is manageable, production systems quickly outgrow manual management.

Problems Orchestration Solves:

Challenges of Unorchestrated Containers

•Scheduling — Which server should run each container? Manual placement doesn't scale. Orchestrators automatically schedule containers based on resource requirements and constraints.
•Scaling — How do you add more instances when load increases? Orchestrators automate horizontal scaling based on metrics like CPU or request rate.
•Self-Healing — What happens when a container crashes or a server dies? Orchestrators detect failures and automatically restart or reschedule containers.
•Networking — How do containers on different servers communicate? Orchestrators provide virtual networks, DNS, and load balancing.
•Service Discovery — How do containers find each other? IP addresses change as containers restart. Orchestrators provide stable endpoints.
•Rolling Updates — How do you deploy new versions without downtime? Orchestrators manage gradual rollouts and automatic rollbacks.
•Configuration Management — How do you distribute secrets and configuration across containers? Orchestrators provide secure config and secret management.
•Resource Management — How do you prevent containers from monopolizing resources? Orchestrators enforce resource limits and quotas.

scaling-comparison.txt
MANUAL CONTAINER MANAGEMENT (doesn't scale)
===========================================
 
You have 50 microservices across 20 servers. Need to:
 
1. Deploy new version of payment-service
   - SSH into each server running payment-service
   - Stop old containers
   - Pull new image
   - Start new containers
   - Verify health
   - Repeat for each instance
   TIME: Hours + risk of human error
 
2. Handle server failure
   - Detect failure (monitoring needed)
   - Identify which containers were running
   - SSH into replacement server
   - Manually start containers
   - Update load balancer config
   TIME: Minutes to hours of downtime
 
3. Scale for Black Friday traffic
   - Calculate capacity needed
   - Provision servers
   - Install Docker
   - Start containers on each
   - Update DNS/load balancer
   TIME: Days of preparation
 
═══════════════════════════════════════════════════════════════
 
ORCHESTRATED CONTAINER MANAGEMENT (Kubernetes)
==============================================
 
1. Deploy new version of payment-service
   $ kubectl set image deployment/payment-service \
       payment=payment:v2.1.0
   
   Kubernetes:
   - Starts new pods with v2.1.0
   - Waits for health checks
   - Gradually shifts traffic
   - Terminates old pods only when new are healthy
   TIME: 2-5 minutes, zero downtime, automatic rollback if issues
 
2. Handle server failure
   Kubernetes automatically:
   - Detects node failure within seconds
   - Marks pods as evicted
   - Schedules pods on healthy nodes
   - Updates networking
   TIME: Seconds to minutes, automatic
 
3. Scale for Black Friday traffic
   $ kubectl scale deployment/frontend --replicas=100
   
   Or with autoscaling:
   $ kubectl autoscale deployment/frontend \
       --min=10 --max=200 --cpu-percent=70
   
   TIME: Seconds for scale-out, automatic based on actual load

The Tipping Point

Most teams find that 10-20 containers across 3+ servers marks the point where manual management becomes unsustainable. Orchestration provides value immediately and becomes essential as you scale further.

Orchestration Platforms Overview

Several container orchestration platforms exist, each with different design philosophies and target use cases. Understanding the landscape helps you choose appropriately.

Major Platforms:

Container Orchestration Platforms
Platform	Provider	Key Characteristics	Best For
Kubernetes	CNCF (Cloud Native Computing Foundation)	Extensible, declarative, industry standard. Complex but comprehensive.	Production workloads at scale, multi-cloud, vendor neutrality
Docker Swarm	Docker, Inc.	Simple, integrated with Docker. Limited compared to Kubernetes.	Small deployments, Docker-native environments
Amazon ECS	AWS	AWS-native, simpler than Kubernetes. Deep AWS integration.	AWS-only deployments, simpler requirements
Nomad	HashiCorp	Simple, supports VMs and non-containerized workloads.	Mixed workloads, simpler orchestration needs
OpenShift	Red Hat	Kubernetes + enterprise features, developer platform.	Enterprise, regulated industries

Why Kubernetes Won:

Kubernetes emerged from Google's internal container management system (Borg) and was open-sourced in 2014. It became the dominant platform for several reasons:

Proven at scale — Google's experience running billions of containers informed the design
Cloud-native ecosystem — Became the foundation for a massive ecosystem (CNCF)
Vendor neutrality — Runs on any cloud or on-premises
Extensibility — Custom resources and operators extend capabilities
Community — Largest open-source community in cloud-native computing

The rest of this section focuses on Kubernetes, but the concepts (scheduling, scaling, self-healing, networking) are universal to orchestration.

Managed Kubernetes

Running Kubernetes yourself is complex. Most organizations use managed Kubernetes services: Amazon EKS, Google GKE, Azure AKS, or DigitalOcean Kubernetes. These handle the control plane so you focus on running workloads, not managing Kubernetes itself.

Kubernetes Architecture

Kubernetes follows a control plane / worker node architecture. The control plane makes global decisions about the cluster, while worker nodes run the actual containerized workloads.

High-Level Architecture:

kubernetes-architecture.txt
KUBERNETES CLUSTER ARCHITECTURE
================================
 
                    ┌───────────────────────────────────────────────────────────┐
                    │                     CONTROL PLANE                          │
                    │            (manages cluster state & decisions)             │
                    │                                                            │
                    │  ┌─────────────────────────────────────────────────────┐  │
                    │  │                 API SERVER (kube-apiserver)          │  │
                    │  │   • Central communication hub                        │  │
                    │  │   • REST API for all operations                      │  │
                    │  │   • Authentication & authorization                   │  │
                    │  └─────────────────────────────────────────────────────┘  │
                    │        │             │               │                     │
                    │        ▼             ▼               ▼                     │
                    │  ┌──────────┐  ┌──────────┐  ┌─────────────────────────┐  │
                    │  │  etcd    │  │Controller│  │      Scheduler           │  │
                    │  │          │  │ Manager  │  │                          │  │
                    │  │ Cluster  │  │          │  │ Assigns pods to nodes   │  │
                    │  │ state    │  │ Control  │  │ based on resources      │  │
                    │  │ storage  │  │ loops    │  │ and constraints         │  │
                    │  └──────────┘  └──────────┘  └─────────────────────────┘  │
                    └───────────────────────────────────────────────────────────┘
                                              │
                                              │ API calls
                                              │
    ┌────────────────────────────────────────────────────────────────────────────┐
    │                            WORKER NODES                                     │
    │                     (run containerized workloads)                          │
    │                                                                             │
    │  ┌────────────────────────────┐     ┌────────────────────────────┐        │
    │  │         NODE 1             │     │         NODE 2             │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │      kubelet        │   │     │  │      kubelet        │   │        │
    │  │  │  • Manages pods     │   │     │  │  • Reports status   │   │        │
    │  │  │  • Reports status   │   │     │  │  • Manages pods     │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │    kube-proxy       │   │     │  │    kube-proxy       │   │        │
    │  │  │  • Network routing  │   │     │  │  • Load balancing   │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │  Container Runtime  │   │     │  │  Container Runtime  │   │        │
    │  │  │  (containerd)       │   │     │  │  (containerd)       │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────┐ ┌─────┐           │     │  ┌─────┐ ┌─────┐ ┌─────┐  │        │
    │  │  │ Pod │ │ Pod │           │     │  │ Pod │ │ Pod │ │ Pod │  │        │
    │  │  │┌─┬─┐│ │┌───┐│           │     │  │┌───┐│ │┌───┐│ │┌───┐│  │        │
    │  │  ││C│C││ ││ C ││           │     │  ││ C ││ ││ C ││ ││ C ││  │        │
    │  │  │└─┴─┘│ │└───┘│           │     │  │└───┘│ │└───┘│ │└───┘│  │        │
    │  │  └─────┘ └─────┘           │     │  └─────┘ └─────┘ └─────┘  │        │
    │  └────────────────────────────┘     └────────────────────────────┘        │
    └────────────────────────────────────────────────────────────────────────────┘
 
    C = Container    Pod = Group of co-scheduled containers

Control Plane Components

•API Server (kube-apiserver) — The frontend for the Kubernetes control plane. All communication goes through the API server—kubectl commands, internal components, and external integrations.
•etcd — A consistent and highly-available key-value store used for all cluster data. The source of truth for cluster state. Never interact with it directly.
•Controller Manager — Runs controller loops that watch cluster state and make changes to move toward desired state. Includes Node Controller, Job Controller, Endpoint Controller, etc.
•Scheduler (kube-scheduler) — Watches for newly created Pods with no assigned node and selects a node for them to run on based on resource requirements, constraints, affinities, and taints.

Worker Node Components

•kubelet — An agent that runs on each node. It ensures containers described in PodSpecs are running and healthy. Communicates with the API server and container runtime.
•kube-proxy — Network proxy on each node implementing Kubernetes Service concepts. Manages network rules so pods can communicate within and outside the cluster.
•Container Runtime — Software responsible for running containers. Kubernetes supports any OCI-compliant runtime: containerd, CRI-O, etc.

Pods: The Atomic Unit

A Pod is the smallest deployable unit in Kubernetes. It represents one or more containers that share storage, network, and specifications for how to run. Pods are ephemeral—designed to be created, destroyed, and replaced.

Key Pod Concepts:

Single-container pods — The most common pattern. One container per pod.
Multi-container pods — For tightly coupled containers that must share resources (sidecar pattern).
Shared namespace — Containers in a pod share network namespace (localhost) and can share volumes.
Ephemeral — Pods can be killed and replaced at any time. Don't store data in pods; use volumes.

pod-definition.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Pod definition (declarative configuration)
apiVersion: v1
kind: Pod
metadata:
  name: web-server
  labels:
    app: frontend
    environment: production
spec:
  containers:
  - name: nginx
    image: nginx:1.25.3
    ports:
    - containerPort: 80
    resources:                    # Resource management
      requests:                   # Minimum guaranteed
        memory: "128Mi"
        cpu: "100m"              # 100 millicores = 0.1 CPU
      limits:                     # Maximum allowed
        memory: "256Mi"
        cpu: "500m"
    livenessProbe:               # Is container alive?
      httpGet:
        path: /healthz
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 10
    readinessProbe:              # Is container ready for traffic?
      httpGet:
        path: /ready
        port: 80
      initialDelaySeconds: 3
      periodSeconds: 5
    volumeMounts:
    - name: config-volume
      mountPath: /etc/nginx/conf.d
  volumes:                       # Shared storage
  - name: config-volume
    configMap:
      name: nginx-config

Multi-Container Pod Patterns:

Common Multi-Container Pod Patterns
Pattern	Description	Example Use Case
Sidecar	Helper container that augments the main container	Log collector, service mesh proxy (Envoy)
Ambassador	Proxy outgoing connections on behalf of main container	Database proxy, API gateway
Adapter	Transforms data from main container for external consumption	Log format conversion, metrics export
Init Container	Runs before main containers, sets up prerequisites	Database migration, config generation

Don't Create Pods Directly

In production, you rarely create Pods directly. Instead, you create higher-level abstractions (Deployments, StatefulSets, Jobs) that manage Pods for you. These controllers handle scaling, updates, and self-healing that bare Pods don't provide.

Deployments and ReplicaSets

A Deployment is the standard way to manage stateless applications in Kubernetes. It provides declarative updates for Pods, handling scaling, rolling updates, and rollbacks automatically.

How Deployments Work:

Deployment (manages)
    └── ReplicaSet (ensures N pods exist)
            └── Pod, Pod, Pod...

When you update a Deployment, it creates a new ReplicaSet with the updated Pod spec and gradually scales down the old ReplicaSet while scaling up the new one—a rolling update.

deployment.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 3                    # Desired number of pods
  selector:
    matchLabels:
      app: web-app               # Which pods this deployment manages
  strategy:
    type: RollingUpdate          # Update strategy
    rollingUpdate:
      maxSurge: 1               # Max extra pods during update
      maxUnavailable: 0         # Min available pods during update
  template:                      # Pod template (what gets created)
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web
        image: myapp:v1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3

deployment-operations.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Create deployment
$ kubectl apply -f deployment.yaml
deployment.apps/web-app created
 
# View deployment status
$ kubectl get deployments
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
web-app   3/3     3            3           2m
 
# View pods created by deployment
$ kubectl get pods -l app=web-app
NAME                       READY   STATUS    RESTARTS   AGE
web-app-6d889f6bcd-7xz8q   1/1     Running   0          2m
web-app-6d889f6bcd-k2n4m   1/1     Running   0          2m
web-app-6d889f6bcd-p9x3v   1/1     Running   0          2m
 
# Scale deployment
$ kubectl scale deployment/web-app --replicas=5
deployment.apps/web-app scaled
 
# Rolling update (change image version)
$ kubectl set image deployment/web-app web=myapp:v2.0.0
deployment.apps/web-app image updated
 
# Watch rollout progress
$ kubectl rollout status deployment/web-app
Waiting for deployment "web-app" rollout to finish: 1 out of 3 new replicas...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas...
deployment "web-app" successfully rolled out
 
# View rollout history
$ kubectl rollout history deployment/web-app
 
# Rollback to previous version
$ kubectl rollout undo deployment/web-app
deployment.apps/web-app rolled back

Zero-Downtime Updates

With proper liveness and readiness probes and the RollingUpdate strategy, deployments achieve zero-downtime updates. Kubernetes only shifts traffic to new pods after they pass readiness checks, and old pods continue serving until new ones are ready.

Services and Networking

Pods are ephemeral and get new IP addresses when recreated. Services provide stable endpoints for accessing pods, abstracting away the dynamic nature of pods.

Service Types:

Kubernetes Service Types
Type	Accessibility	Use Case	How It Works
`ClusterIP`	Within cluster only	Internal services	Virtual IP routable only inside cluster
`NodePort`	External via node IP	Development/testing	Exposes service on each node's IP at a static port
`LoadBalancer`	External via LB	Production traffic	Provisions cloud load balancer (AWS ELB, etc.)
`ExternalName`	DNS alias	External services	Maps service to external DNS name (no proxy)

service-definition.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# ClusterIP Service (internal access)
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  type: ClusterIP          # Default type
  selector:
    app: web-app           # Selects pods with this label
  ports:
  - protocol: TCP
    port: 80              # Service port (how other pods access)
    targetPort: 8080      # Container port (where app listens)
 
---
# LoadBalancer Service (external access)
apiVersion: v1
kind: Service
metadata:
  name: web-app-public
spec:
  type: LoadBalancer
  selector:
    app: web-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Service Discovery:

Kubernetes provides built-in service discovery through DNS. Every service gets a DNS entry:

<service-name>.<namespace>.svc.cluster.local

Pods can connect to web-app-service or web-app-service.default.svc.cluster.local and Kubernetes DNS resolves it to the service's ClusterIP. The service then load-balances across healthy pods.

networking-flow.txt
SERVICE NETWORKING FLOW
=======================
 
External Request → LoadBalancer → Service → Pod
 
┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│   CLIENT (internet)                                                          │
│       │                                                                       │
│       ▼                                                                       │
│   ┌────────────────────────────────────────────┐                             │
│   │   CLOUD LOAD BALANCER (AWS ELB, etc.)      │                             │
│   │   External IP: 54.23.145.67:80             │                             │
│   └────────────────────────────────────────────┘                             │
│       │                                                                       │
│       │ Routes to any healthy node                                           │
│       ▼                                                                       │
│   ┌────────────────────────────────────────────────────────────────────────┐ │
│   │                    KUBERNETES CLUSTER                                   │ │
│   │                                                                         │ │
│   │   ┌───────────────────────────────────────────────────────────┐       │ │
│   │   │   SERVICE: web-app-public                                  │       │ │
│   │   │   ClusterIP: 10.96.45.123:80                              │       │ │
│   │   │   Type: LoadBalancer                                       │       │ │
│   │   │   Selector: app=web-app                                    │       │ │
│   │   └───────────────────────────────────────────────────────────┘       │ │
│   │                         │                                               │ │
│   │         ┌───────────────┼───────────────┐                              │ │
│   │         │               │               │                              │ │
│   │         ▼               ▼               ▼                              │ │
│   │   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐                     │ │
│   │   │   Pod 1     │ │   Pod 2     │ │   Pod 3     │                     │ │
│   │   │ 10.244.1.5  │ │ 10.244.2.8  │ │ 10.244.1.9  │                     │ │
│   │   │    :8080    │ │    :8080    │ │    :8080    │                     │ │
│   │   │ app=web-app │ │ app=web-app │ │ app=web-app │                     │ │
│   │   └─────────────┘ └─────────────┘ └─────────────┘                     │ │
│   │                                                                         │ │
│   │   kube-proxy on each node handles routing and load balancing          │ │
│   └────────────────────────────────────────────────────────────────────────┘ │
│                                                                               │
└──────────────────────────────────────────────────────────────────────────────┘

Ingress for HTTP Routing

For HTTP/HTTPS traffic, Ingress provides URL-based routing, SSL termination, and virtual hosting. Instead of one LoadBalancer per service (expensive), a single Ingress controller routes traffic to multiple services based on hostname and path.

ConfigMaps and Secrets

Kubernetes separates configuration from container images using ConfigMaps (for non-sensitive data) and Secrets (for sensitive data). This enables using the same image across environments with different configurations.

ConfigMaps:

configmap-secret.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# ConfigMap: Non-sensitive configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres.database.svc.cluster.local"
  DATABASE_NAME: "myapp"
  LOG_LEVEL: "info"
  config.json: |
    {
      "feature_flags": {
        "new_ui": true,
        "beta_features": false
      }
    }
---
# Secret: Sensitive data (base64 encoded in manifest, encrypted at rest)
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # Values are base64 encoded
  DATABASE_PASSWORD: cGFzc3dvcmQxMjM=    # password123
  API_KEY: c2VjcmV0LWFwaS1rZXk=          # secret-api-key
---
# Using ConfigMap and Secret in a Pod
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:v1.0.0
    env:
      # Individual values from ConfigMap
      - name: DATABASE_HOST
        valueFrom:
          configMapKeyRef:
            name: app-config
            key: DATABASE_HOST
      # Individual values from Secret
      - name: DATABASE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: DATABASE_PASSWORD
    envFrom:
      # All keys from ConfigMap as env vars
      - configMapRef:
          name: app-config
    volumeMounts:
      # Mount ConfigMap as files
      - name: config-volume
        mountPath: /etc/config
        readOnly: true
  volumes:
    - name: config-volume
      configMap:
        name: app-config

Secrets Aren't Fully Secure by Default

Kubernetes Secrets are only base64 encoded, not encrypted, in etcd by default. Enable encryption at rest in your cluster configuration. For production, consider external secret management (HashiCorp Vault, AWS Secrets Manager) with operators that sync secrets to Kubernetes.

Summary: Container Orchestration

We've explored container orchestration from the problems it solves through Kubernetes architecture and key abstractions. Let's consolidate the essential concepts:

Key Takeaways

•Orchestration solves scaling challenges — Manual container management doesn't scale. Orchestrators automate scheduling, scaling, networking, and self-healing.
•Kubernetes is the industry standard — Most production container deployments use Kubernetes or managed Kubernetes services (EKS, GKE, AKS).
•Control plane / worker node architecture — The control plane (API server, scheduler, controllers, etcd) makes decisions; worker nodes (kubelet, kube-proxy, container runtime) run workloads.
•Pods are the atomic unit — Pods contain one or more containers sharing network and storage. They're ephemeral and managed by higher-level controllers.
•Deployments manage stateless apps — They handle scaling, rolling updates, and rollbacks. ReplicaSets ensure the desired number of pods exist.
•Services provide stable endpoints — Pods get dynamic IPs; Services provide consistent DNS names and load balancing. Different types for internal vs. external access.
•ConfigMaps and Secrets separate config from code — Same image, different configuration per environment. Secrets for sensitive data (with proper encryption).

Module Complete:

You've now completed the Containers module, covering:

Container concepts and the problems they solve
Containers vs VMs—understanding the trade-offs
Docker architecture—from CLI to running container
Container images—building, optimizing, and distributing
Container orchestration—managing containers at scale

This knowledge forms the foundation for working with modern cloud-native applications. The concepts apply whether you're developing on your laptop, deploying to production Kubernetes, or architecting systems at scale.

Module Complete

You now understand containers from first principles through production orchestration. This module provided the conceptual foundation for the next module on Namespaces and cgroups, where we'll dive into the Linux kernel features that make containers possible.

5 / 5

Loading learning content...

Operating SystemsVirtualization

Containers

LevelIntermediate

Duration90 mins

TopicVirtualization

5 / 5

Container Orchestration

Managing Containers at Scale

What You Will Learn

Why Orchestration is Needed

Container orchestration solves problems that emerge when containers are deployed at scale. While a single Docker host is manageable, production systems quickly outgrow manual management.

Problems Orchestration Solves:

Challenges of Unorchestrated Containers

•Scheduling — Which server should run each container? Manual placement doesn't scale. Orchestrators automatically schedule containers based on resource requirements and constraints.
•Scaling — How do you add more instances when load increases? Orchestrators automate horizontal scaling based on metrics like CPU or request rate.
•Self-Healing — What happens when a container crashes or a server dies? Orchestrators detect failures and automatically restart or reschedule containers.
•Networking — How do containers on different servers communicate? Orchestrators provide virtual networks, DNS, and load balancing.
•Service Discovery — How do containers find each other? IP addresses change as containers restart. Orchestrators provide stable endpoints.
•Rolling Updates — How do you deploy new versions without downtime? Orchestrators manage gradual rollouts and automatic rollbacks.
•Configuration Management — How do you distribute secrets and configuration across containers? Orchestrators provide secure config and secret management.
•Resource Management — How do you prevent containers from monopolizing resources? Orchestrators enforce resource limits and quotas.

scaling-comparison.txt
MANUAL CONTAINER MANAGEMENT (doesn't scale)
===========================================
 
You have 50 microservices across 20 servers. Need to:
 
1. Deploy new version of payment-service
   - SSH into each server running payment-service
   - Stop old containers
   - Pull new image
   - Start new containers
   - Verify health
   - Repeat for each instance
   TIME: Hours + risk of human error
 
2. Handle server failure
   - Detect failure (monitoring needed)
   - Identify which containers were running
   - SSH into replacement server
   - Manually start containers
   - Update load balancer config
   TIME: Minutes to hours of downtime
 
3. Scale for Black Friday traffic
   - Calculate capacity needed
   - Provision servers
   - Install Docker
   - Start containers on each
   - Update DNS/load balancer
   TIME: Days of preparation
 
═══════════════════════════════════════════════════════════════
 
ORCHESTRATED CONTAINER MANAGEMENT (Kubernetes)
==============================================
 
1. Deploy new version of payment-service
   $ kubectl set image deployment/payment-service \
       payment=payment:v2.1.0
   
   Kubernetes:
   - Starts new pods with v2.1.0
   - Waits for health checks
   - Gradually shifts traffic
   - Terminates old pods only when new are healthy
   TIME: 2-5 minutes, zero downtime, automatic rollback if issues
 
2. Handle server failure
   Kubernetes automatically:
   - Detects node failure within seconds
   - Marks pods as evicted
   - Schedules pods on healthy nodes
   - Updates networking
   TIME: Seconds to minutes, automatic
 
3. Scale for Black Friday traffic
   $ kubectl scale deployment/frontend --replicas=100
   
   Or with autoscaling:
   $ kubectl autoscale deployment/frontend \
       --min=10 --max=200 --cpu-percent=70
   
   TIME: Seconds for scale-out, automatic based on actual load

The Tipping Point

Orchestration Platforms Overview

Several container orchestration platforms exist, each with different design philosophies and target use cases. Understanding the landscape helps you choose appropriately.

Major Platforms:

Container Orchestration Platforms
Platform	Provider	Key Characteristics	Best For
Kubernetes	CNCF (Cloud Native Computing Foundation)	Extensible, declarative, industry standard. Complex but comprehensive.	Production workloads at scale, multi-cloud, vendor neutrality
Docker Swarm	Docker, Inc.	Simple, integrated with Docker. Limited compared to Kubernetes.	Small deployments, Docker-native environments
Amazon ECS	AWS	AWS-native, simpler than Kubernetes. Deep AWS integration.	AWS-only deployments, simpler requirements
Nomad	HashiCorp	Simple, supports VMs and non-containerized workloads.	Mixed workloads, simpler orchestration needs
OpenShift	Red Hat	Kubernetes + enterprise features, developer platform.	Enterprise, regulated industries

Why Kubernetes Won:

Kubernetes emerged from Google's internal container management system (Borg) and was open-sourced in 2014. It became the dominant platform for several reasons:

Proven at scale — Google's experience running billions of containers informed the design
Cloud-native ecosystem — Became the foundation for a massive ecosystem (CNCF)
Vendor neutrality — Runs on any cloud or on-premises
Extensibility — Custom resources and operators extend capabilities
Community — Largest open-source community in cloud-native computing

The rest of this section focuses on Kubernetes, but the concepts (scheduling, scaling, self-healing, networking) are universal to orchestration.

Managed Kubernetes

Kubernetes Architecture

Kubernetes follows a control plane / worker node architecture. The control plane makes global decisions about the cluster, while worker nodes run the actual containerized workloads.

High-Level Architecture:

kubernetes-architecture.txt
KUBERNETES CLUSTER ARCHITECTURE
================================
 
                    ┌───────────────────────────────────────────────────────────┐
                    │                     CONTROL PLANE                          │
                    │            (manages cluster state & decisions)             │
                    │                                                            │
                    │  ┌─────────────────────────────────────────────────────┐  │
                    │  │                 API SERVER (kube-apiserver)          │  │
                    │  │   • Central communication hub                        │  │
                    │  │   • REST API for all operations                      │  │
                    │  │   • Authentication & authorization                   │  │
                    │  └─────────────────────────────────────────────────────┘  │
                    │        │             │               │                     │
                    │        ▼             ▼               ▼                     │
                    │  ┌──────────┐  ┌──────────┐  ┌─────────────────────────┐  │
                    │  │  etcd    │  │Controller│  │      Scheduler           │  │
                    │  │          │  │ Manager  │  │                          │  │
                    │  │ Cluster  │  │          │  │ Assigns pods to nodes   │  │
                    │  │ state    │  │ Control  │  │ based on resources      │  │
                    │  │ storage  │  │ loops    │  │ and constraints         │  │
                    │  └──────────┘  └──────────┘  └─────────────────────────┘  │
                    └───────────────────────────────────────────────────────────┘
                                              │
                                              │ API calls
                                              │
    ┌────────────────────────────────────────────────────────────────────────────┐
    │                            WORKER NODES                                     │
    │                     (run containerized workloads)                          │
    │                                                                             │
    │  ┌────────────────────────────┐     ┌────────────────────────────┐        │
    │  │         NODE 1             │     │         NODE 2             │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │      kubelet        │   │     │  │      kubelet        │   │        │
    │  │  │  • Manages pods     │   │     │  │  • Reports status   │   │        │
    │  │  │  • Reports status   │   │     │  │  • Manages pods     │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │    kube-proxy       │   │     │  │    kube-proxy       │   │        │
    │  │  │  • Network routing  │   │     │  │  • Load balancing   │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────────────────────┐   │     │  ┌─────────────────────┐   │        │
    │  │  │  Container Runtime  │   │     │  │  Container Runtime  │   │        │
    │  │  │  (containerd)       │   │     │  │  (containerd)       │   │        │
    │  │  └─────────────────────┘   │     │  └─────────────────────┘   │        │
    │  │  ┌─────┐ ┌─────┐           │     │  ┌─────┐ ┌─────┐ ┌─────┐  │        │
    │  │  │ Pod │ │ Pod │           │     │  │ Pod │ │ Pod │ │ Pod │  │        │
    │  │  │┌─┬─┐│ │┌───┐│           │     │  │┌───┐│ │┌───┐│ │┌───┐│  │        │
    │  │  ││C│C││ ││ C ││           │     │  ││ C ││ ││ C ││ ││ C ││  │        │
    │  │  │└─┴─┘│ │└───┘│           │     │  │└───┘│ │└───┘│ │└───┘│  │        │
    │  │  └─────┘ └─────┘           │     │  └─────┘ └─────┘ └─────┘  │        │
    │  └────────────────────────────┘     └────────────────────────────┘        │
    └────────────────────────────────────────────────────────────────────────────┘
 
    C = Container    Pod = Group of co-scheduled containers

Control Plane Components

•API Server (kube-apiserver) — The frontend for the Kubernetes control plane. All communication goes through the API server—kubectl commands, internal components, and external integrations.
•etcd — A consistent and highly-available key-value store used for all cluster data. The source of truth for cluster state. Never interact with it directly.
•Controller Manager — Runs controller loops that watch cluster state and make changes to move toward desired state. Includes Node Controller, Job Controller, Endpoint Controller, etc.
•Scheduler (kube-scheduler) — Watches for newly created Pods with no assigned node and selects a node for them to run on based on resource requirements, constraints, affinities, and taints.

Worker Node Components

•kubelet — An agent that runs on each node. It ensures containers described in PodSpecs are running and healthy. Communicates with the API server and container runtime.
•kube-proxy — Network proxy on each node implementing Kubernetes Service concepts. Manages network rules so pods can communicate within and outside the cluster.
•Container Runtime — Software responsible for running containers. Kubernetes supports any OCI-compliant runtime: containerd, CRI-O, etc.

Pods: The Atomic Unit

Key Pod Concepts:

Single-container pods — The most common pattern. One container per pod.
Multi-container pods — For tightly coupled containers that must share resources (sidecar pattern).
Shared namespace — Containers in a pod share network namespace (localhost) and can share volumes.
Ephemeral — Pods can be killed and replaced at any time. Don't store data in pods; use volumes.

pod-definition.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Pod definition (declarative configuration)
apiVersion: v1
kind: Pod
metadata:
  name: web-server
  labels:
    app: frontend
    environment: production
spec:
  containers:
  - name: nginx
    image: nginx:1.25.3
    ports:
    - containerPort: 80
    resources:                    # Resource management
      requests:                   # Minimum guaranteed
        memory: "128Mi"
        cpu: "100m"              # 100 millicores = 0.1 CPU
      limits:                     # Maximum allowed
        memory: "256Mi"
        cpu: "500m"
    livenessProbe:               # Is container alive?
      httpGet:
        path: /healthz
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 10
    readinessProbe:              # Is container ready for traffic?
      httpGet:
        path: /ready
        port: 80
      initialDelaySeconds: 3
      periodSeconds: 5
    volumeMounts:
    - name: config-volume
      mountPath: /etc/nginx/conf.d
  volumes:                       # Shared storage
  - name: config-volume
    configMap:
      name: nginx-config

Multi-Container Pod Patterns:

Common Multi-Container Pod Patterns
Pattern	Description	Example Use Case
Sidecar	Helper container that augments the main container	Log collector, service mesh proxy (Envoy)
Ambassador	Proxy outgoing connections on behalf of main container	Database proxy, API gateway
Adapter	Transforms data from main container for external consumption	Log format conversion, metrics export
Init Container	Runs before main containers, sets up prerequisites	Database migration, config generation

Don't Create Pods Directly

Deployments and ReplicaSets

A Deployment is the standard way to manage stateless applications in Kubernetes. It provides declarative updates for Pods, handling scaling, rolling updates, and rollbacks automatically.

How Deployments Work:

Deployment (manages)
    └── ReplicaSet (ensures N pods exist)
            └── Pod, Pod, Pod...

When you update a Deployment, it creates a new ReplicaSet with the updated Pod spec and gradually scales down the old ReplicaSet while scaling up the new one—a rolling update.

deployment.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 3                    # Desired number of pods
  selector:
    matchLabels:
      app: web-app               # Which pods this deployment manages
  strategy:
    type: RollingUpdate          # Update strategy
    rollingUpdate:
      maxSurge: 1               # Max extra pods during update
      maxUnavailable: 0         # Min available pods during update
  template:                      # Pod template (what gets created)
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web
        image: myapp:v1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3

deployment-operations.sh
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Create deployment
$ kubectl apply -f deployment.yaml
deployment.apps/web-app created
 
# View deployment status
$ kubectl get deployments
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
web-app   3/3     3            3           2m
 
# View pods created by deployment
$ kubectl get pods -l app=web-app
NAME                       READY   STATUS    RESTARTS   AGE
web-app-6d889f6bcd-7xz8q   1/1     Running   0          2m
web-app-6d889f6bcd-k2n4m   1/1     Running   0          2m
web-app-6d889f6bcd-p9x3v   1/1     Running   0          2m
 
# Scale deployment
$ kubectl scale deployment/web-app --replicas=5
deployment.apps/web-app scaled
 
# Rolling update (change image version)
$ kubectl set image deployment/web-app web=myapp:v2.0.0
deployment.apps/web-app image updated
 
# Watch rollout progress
$ kubectl rollout status deployment/web-app
Waiting for deployment "web-app" rollout to finish: 1 out of 3 new replicas...
Waiting for deployment "web-app" rollout to finish: 2 out of 3 new replicas...
deployment "web-app" successfully rolled out
 
# View rollout history
$ kubectl rollout history deployment/web-app
 
# Rollback to previous version
$ kubectl rollout undo deployment/web-app
deployment.apps/web-app rolled back

Zero-Downtime Updates

Services and Networking

Pods are ephemeral and get new IP addresses when recreated. Services provide stable endpoints for accessing pods, abstracting away the dynamic nature of pods.

Service Types:

Kubernetes Service Types
Type	Accessibility	Use Case	How It Works
`ClusterIP`	Within cluster only	Internal services	Virtual IP routable only inside cluster
`NodePort`	External via node IP	Development/testing	Exposes service on each node's IP at a static port
`LoadBalancer`	External via LB	Production traffic	Provisions cloud load balancer (AWS ELB, etc.)
`ExternalName`	DNS alias	External services	Maps service to external DNS name (no proxy)

service-definition.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# ClusterIP Service (internal access)
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  type: ClusterIP          # Default type
  selector:
    app: web-app           # Selects pods with this label
  ports:
  - protocol: TCP
    port: 80              # Service port (how other pods access)
    targetPort: 8080      # Container port (where app listens)
 
---
# LoadBalancer Service (external access)
apiVersion: v1
kind: Service
metadata:
  name: web-app-public
spec:
  type: LoadBalancer
  selector:
    app: web-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Service Discovery:

Kubernetes provides built-in service discovery through DNS. Every service gets a DNS entry:

<service-name>.<namespace>.svc.cluster.local

Pods can connect to web-app-service or web-app-service.default.svc.cluster.local and Kubernetes DNS resolves it to the service's ClusterIP. The service then load-balances across healthy pods.

networking-flow.txt
SERVICE NETWORKING FLOW
=======================
 
External Request → LoadBalancer → Service → Pod
 
┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│   CLIENT (internet)                                                          │
│       │                                                                       │
│       ▼                                                                       │
│   ┌────────────────────────────────────────────┐                             │
│   │   CLOUD LOAD BALANCER (AWS ELB, etc.)      │                             │
│   │   External IP: 54.23.145.67:80             │                             │
│   └────────────────────────────────────────────┘                             │
│       │                                                                       │
│       │ Routes to any healthy node                                           │
│       ▼                                                                       │
│   ┌────────────────────────────────────────────────────────────────────────┐ │
│   │                    KUBERNETES CLUSTER                                   │ │
│   │                                                                         │ │
│   │   ┌───────────────────────────────────────────────────────────┐       │ │
│   │   │   SERVICE: web-app-public                                  │       │ │
│   │   │   ClusterIP: 10.96.45.123:80                              │       │ │
│   │   │   Type: LoadBalancer                                       │       │ │
│   │   │   Selector: app=web-app                                    │       │ │
│   │   └───────────────────────────────────────────────────────────┘       │ │
│   │                         │                                               │ │
│   │         ┌───────────────┼───────────────┐                              │ │
│   │         │               │               │                              │ │
│   │         ▼               ▼               ▼                              │ │
│   │   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐                     │ │
│   │   │   Pod 1     │ │   Pod 2     │ │   Pod 3     │                     │ │
│   │   │ 10.244.1.5  │ │ 10.244.2.8  │ │ 10.244.1.9  │                     │ │
│   │   │    :8080    │ │    :8080    │ │    :8080    │                     │ │
│   │   │ app=web-app │ │ app=web-app │ │ app=web-app │                     │ │
│   │   └─────────────┘ └─────────────┘ └─────────────┘                     │ │
│   │                                                                         │ │
│   │   kube-proxy on each node handles routing and load balancing          │ │
│   └────────────────────────────────────────────────────────────────────────┘ │
│                                                                               │
└──────────────────────────────────────────────────────────────────────────────┘

Ingress for HTTP Routing

ConfigMaps and Secrets

ConfigMaps:

configmap-secret.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# ConfigMap: Non-sensitive configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres.database.svc.cluster.local"
  DATABASE_NAME: "myapp"
  LOG_LEVEL: "info"
  config.json: |
    {
      "feature_flags": {
        "new_ui": true,
        "beta_features": false
      }
    }
---
# Secret: Sensitive data (base64 encoded in manifest, encrypted at rest)
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # Values are base64 encoded
  DATABASE_PASSWORD: cGFzc3dvcmQxMjM=    # password123
  API_KEY: c2VjcmV0LWFwaS1rZXk=          # secret-api-key
---
# Using ConfigMap and Secret in a Pod
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:v1.0.0
    env:
      # Individual values from ConfigMap
      - name: DATABASE_HOST
        valueFrom:
          configMapKeyRef:
            name: app-config
            key: DATABASE_HOST
      # Individual values from Secret
      - name: DATABASE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: DATABASE_PASSWORD
    envFrom:
      # All keys from ConfigMap as env vars
      - configMapRef:
          name: app-config
    volumeMounts:
      # Mount ConfigMap as files
      - name: config-volume
        mountPath: /etc/config
        readOnly: true
  volumes:
    - name: config-volume
      configMap:
        name: app-config

Secrets Aren't Fully Secure by Default

Summary: Container Orchestration

We've explored container orchestration from the problems it solves through Kubernetes architecture and key abstractions. Let's consolidate the essential concepts:

Key Takeaways

•Orchestration solves scaling challenges — Manual container management doesn't scale. Orchestrators automate scheduling, scaling, networking, and self-healing.
•Kubernetes is the industry standard — Most production container deployments use Kubernetes or managed Kubernetes services (EKS, GKE, AKS).
•Control plane / worker node architecture — The control plane (API server, scheduler, controllers, etcd) makes decisions; worker nodes (kubelet, kube-proxy, container runtime) run workloads.
•Pods are the atomic unit — Pods contain one or more containers sharing network and storage. They're ephemeral and managed by higher-level controllers.
•Deployments manage stateless apps — They handle scaling, rolling updates, and rollbacks. ReplicaSets ensure the desired number of pods exist.
•Services provide stable endpoints — Pods get dynamic IPs; Services provide consistent DNS names and load balancing. Different types for internal vs. external access.
•ConfigMaps and Secrets separate config from code — Same image, different configuration per environment. Secrets for sensitive data (with proper encryption).

Module Complete:

You've now completed the Containers module, covering:

Container concepts and the problems they solve
Containers vs VMs—understanding the trade-offs
Docker architecture—from CLI to running container
Container images—building, optimizing, and distributing
Container orchestration—managing containers at scale

Module Complete

5 / 5