System Design HLDKubernetes Architecture

Understanding Kubernetes Architecture

LevelIntermediate

Duration120 mins

TopicKubernetes Architecture

4 / 5

Declarative Configuration: Tell Kubernetes What, Not How

The Power of Declarative Infrastructure

Kubernetes fundamentally changes how we think about infrastructure management. Instead of writing scripts that tell the system how to reach a state (imperative), you declare what state you want and let Kubernetes figure out how to achieve it (declarative).

This shift may seem subtle, but it's transformative. Declarative configuration enables self-healing, GitOps workflows, reliable rollbacks, and operations at scale that would be impossible with imperative approaches. Understanding and embracing this paradigm is essential for effective Kubernetes usage.

What You Will Learn

By the end of this page, you will understand the declarative model deeply—how manifests are structured, how the reconciliation loop works, why declarative beats imperative at scale, and how to structure your configurations for maintainability and reliability.

Imperative vs. Declarative: A Paradigm Shift

Before Kubernetes, infrastructure management was typically imperative—you wrote scripts that executed step-by-step:

Check if server exists
If not, create server
Install packages
Configure web server
Start service
Verify it's running

This approach has fundamental limitations:

Fragile: Any step failure breaks the chain
Non-idempotent: Running twice may cause errors
Drift: System state can diverge from what scripts assume
Hard to scale: More servers = more script execution = more failures

Kubernetes is declarative:

Imperative Approach

•You specify HOW: Step-by-step instructions
•One-time execution: Script runs once
•Drift is silent: System changes unnoticed
•Error handling is your job: Every edge case must be coded
•State is unknown: No record of what should exist

Declarative Approach

•You specify WHAT: Desired end state
•Continuous reconciliation: Controller loops forever
•Drift is corrected: System restored to desired state
•Kubernetes handles errors: Retries, backoff, self-healing
•State is explicit: Manifests ARE the source of truth

Example: Imperative Script vs. Declarative Manifest

Imperative (shell script):

#!/bin/bash
if ! kubectl get deployment nginx &>/dev/null; then
  kubectl create deployment nginx --image=nginx:1.25
fi
kubectl scale deployment nginx --replicas=3
kubectl set image deployment/nginx nginx=nginx:1.25
# What if a pod crashes? What if the node dies? Script doesn't know.

Declarative (YAML manifest):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels: { app: nginx }
  template:
    metadata:
      labels: { app: nginx }
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
# Apply once. Kubernetes maintains this state forever.
# Pod crashes? Replaced. Node dies? Rescheduled.

Declarative Enables Self-Healing

With declarative configuration, Kubernetes controllers continuously compare desired state (your manifest) with actual state (what's running). Deviations are automatically corrected. A Pod dies? Controller creates a new one. Someone manually deletes a replica? Controller restores it. This is self-healing at infrastructure level.

Anatomy of a Kubernetes Manifest

Every Kubernetes resource follows a consistent structure. Understanding this structure helps you read, write, and debug manifests effectively.

Required Fields:

Field	Purpose
`apiVersion`	API group and version (e.g., `apps/v1`, `v1`)
`kind`	Resource type (e.g., `Deployment`, `Service`)
`metadata`	Name, namespace, labels, annotations
`spec`	Desired state specification

Anatomy of a Complete Manifest:

manifest-structure.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# API version determines available fields and behaviors
apiVersion: apps/v1
 
# Resource type
kind: Deployment
 
# Metadata: identity and organization
metadata:
  # Required: resource name (unique within namespace for this kind)
  name: web-app
  
  # Optional: namespace (defaults to 'default')
  namespace: production
  
  # Optional: key-value pairs for organization and selection
  labels:
    app: web
    environment: production
    team: platform
    version: v2.1.0
  
  # Optional: non-identifying metadata (for tools, documentation)
  annotations:
    kubernetes.io/change-cause: "Deployed v2.1.0 with performance fixes"
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
 
# Spec: the desired state (structure varies by resource type)
spec:
  replicas: 3
  
  selector:
    matchLabels:
      app: web
  
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  
  # For Deployments, spec contains a Pod template
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: myapp:v2.1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
 
# Status: current state (managed by Kubernetes, not in manifests)
# status:
#   replicas: 3
#   readyReplicas: 3
#   availableReplicas: 3
#   conditions:
#     - type: Available
#       status: "True"

Labels vs. Annotations:

Aspect	Labels	Annotations
Purpose	Identify and select objects	Attach non-identifying metadata
Selection	Used in selectors and queries	Cannot be used for selection
Constraints	Limited to 63 chars, alphanumeric	Up to 256KB, any string
Examples	`app: web`, `env: prod`	`description: "Main API"`, `last-deployed: "2024-01-15"`

Common Label Conventions:

Label	Purpose
`app.kubernetes.io/name`	Application name
`app.kubernetes.io/version`	Application version
`app.kubernetes.io/component`	Component within application
`app.kubernetes.io/managed-by`	Tool managing this resource (Helm, ArgoCD)

Never Include 'status' in Manifests

The 'status' section is managed by Kubernetes controllers, reflecting current state. Never include it in your manifests—it will be ignored or cause errors. Your manifests define 'spec' (desire); Kubernetes updates 'status' (reality).

The Reconciliation Loop: How Kubernetes Maintains State

The magic of Kubernetes lies in its reconciliation loops. Controllers continuously watch resources and work to make reality match your declared intent.

The Controller Pattern:

Every controller follows the same pattern:

Observe: Watch the API server for changes to relevant resources
Analyze: Compare current state with desired state
Act: If they differ, take actions to reconcile
Repeat: Loop forever

This is sometimes called a "level-triggered" system—controllers react to the current level (state) rather than edges (individual changes).

reconciliation-loop.txt
Reconciliation Loop Illustrated:
 
┌─────────────────────────────────────────────────────────────────┐
│                    RECONCILIATION LOOP                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  1. OBSERVE: Watch API server for Deployment changes            │
│     - User applies manifest (kubectl apply)                     │
│     - Receives event: "Deployment web-app updated"              │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  2. READ CURRENT STATE                                          │
│     - Query API: "What Pods exist with label app=web?"          │
│     - Result: 2 Pods running                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  3. READ DESIRED STATE                                          │
│     - Query API: "What's the Deployment spec?"                  │
│     - Result: replicas=3                                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  4. COMPARE                                                      │
│     - Current: 2 Pods                                           │
│     - Desired: 3 Pods                                           │
│     - Difference: Need 1 more Pod                               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  5. ACT                                                          │
│     - Create 1 new Pod with spec from template                  │
│     - Update Deployment status                                  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  6. REPEAT                                                       │
│     - Return to step 1, wait for next change                    │
│     - Loop runs forever until controller shuts down             │
└─────────────────────────────────────────────────────────────────┘

Why Level-Triggered Beats Edge-Triggered:

In an edge-triggered system, the controller would react to individual events: "Pod created", "Pod deleted". This creates problems:

What if an event is missed?
What if events arrive out of order?
How do you handle startup when state already exists?

Level-triggered systems don't care about event history—they only care about "what is current state vs. desired state?" This makes them:

Resilient: Controller restart just continues from current state
Idempotent: Running reconciliation multiple times is safe
Self-healing: Manual changes are detected and corrected

Eventual Consistency

Declarative systems are eventually consistent—there's always some lag between declaring state and achieving it. This is a feature, not a bug. It allows graceful handling of failures, retries, and concurrent operations. Design your applications to tolerate this lag.

kubectl apply vs. create vs. replace

Understanding the different kubectl commands for managing resources is crucial for declarative workflows.

kubectl create (Imperative):

Creates a new resource
Fails if resource already exists
One-time operation

kubectl apply (Declarative):

Creates if doesn't exist, updates if it does
Uses three-way merge to calculate changes
Safe to run repeatedly
Records last-applied configuration

kubectl replace (Imperative):

Replaces entire resource specification
Fails if resource doesn't exist
Overwrites any manual changes

Comparison:

kubectl Command Comparison
Command	Resource Exists	Resource Doesn't Exist	Use Case
kubectl create	Error	Creates	Initial creation scripts
kubectl apply	Updates (merge)	Creates	GitOps, declarative workflows
kubectl replace	Replaces	Error	Full manifest replacement
kubectl patch	Patches specific fields	Error	Targeted updates
kubectl delete	Deletes	No-op (or error)	Resource removal

How kubectl apply Works (Three-Way Merge):

kubectl apply compares three versions of the resource:

Local manifest (what you're applying)
Last-applied configuration (stored in annotation)
Live server state (current in cluster)

This three-way merge allows:

Detecting fields you've removed from manifest
Preserving fields set by other tools (e.g., autoscaler changing replicas)
Intelligently merging concurrent changes

apply-example.txt
Three-Way Merge Example:
 
Last Applied:
  replicas: 3
  image: myapp:v1
 
Live Server (after HPA scaled up):
  replicas: 5  ← Changed by HPA
  image: myapp:v1
 
New Local Manifest:
  replicas: 3
  image: myapp:v2  ← Your change
 
Result of kubectl apply:
  replicas: 5  ← Preserved (not in delta between last-applied and new)
  image: myapp:v2  ← Updated (you changed it)
 
This is why 'apply' is smart—it respects changes made by other actors.

Always Use Apply for GitOps

In GitOps workflows, always use 'kubectl apply -f'. The last-applied-configuration annotation is essential for proper merge behavior. Using 'create' or 'replace' breaks this tracking and can cause unexpected behavior.

Organizing Manifests for Real Projects

Real applications have dozens of resources: Deployments, Services, ConfigMaps, Secrets, Ingresses, PVCs, ServiceAccounts, Roles, and more. Organizing these manifests well is critical for maintainability.

Directory Structure Patterns:

directory-structure.txt
Pattern 1: By Resource Type (Simple Projects)
 
k8s/
├── deployments/
│   ├── web.yaml
│   └── api.yaml
├── services/
│   ├── web.yaml
│   └── api.yaml
├── configmaps/
│   └── app-config.yaml
└── ingresses/
    └── main-ingress.yaml
 
Pattern 2: By Component (Microservices)
 
k8s/
├── web/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── api/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   └── secret.yaml  (reference only, actual secret managed separately)
├── database/
│   ├── statefulset.yaml
│   ├── service.yaml
│   └── pvc.yaml
└── common/
    ├── namespace.yaml
    └── network-policies.yaml
 
Pattern 3: By Environment with Kustomize
 
k8s/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── replicas-patch.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── replicas-patch.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── replicas-patch.yaml
│       └── resources-patch.yaml

Multi-Document YAML Files:

You can combine related resources in a single file using --- separator:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web }
  ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: nginx

File Naming Conventions:

Use lowercase with hyphens: web-deployment.yaml
Include resource type for clarity: api-service.yaml
Group related resources: monitoring.yaml (Prometheus + Grafana)

Use Kustomize or Helm for Environments

Don't duplicate entire manifests for different environments. Use Kustomize overlays or Helm values to manage environment-specific differences. DRY (Don't Repeat Yourself) applies to infrastructure code too.

ConfigMaps and Secrets: Externalizing Configuration

Declarative configuration extends to application configuration itself. Instead of baking config into images, externalize it using ConfigMaps and Secrets.

ConfigMaps: Store non-sensitive configuration data Secrets: Store sensitive data (passwords, tokens, certificates)

Creating ConfigMaps:

configmap-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
# ConfigMap with key-value pairs
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres.database.svc.cluster.local"
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  
---
# ConfigMap with file content
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name localhost;
      location / {
        proxy_pass http://backend:8080;
      }
    }
 
---
# Using ConfigMaps in Pods
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:v1
      
      # Method 1: Environment variables from ConfigMap
      envFrom:
        - configMapRef:
            name: app-config
      
      # Method 2: Individual env vars
      env:
        - name: DB_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: DATABASE_HOST
      
      # Method 3: Mount as files
      volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d
  
  volumes:
    - name: config-volume
      configMap:
        name: nginx-config

Secrets:

Secrets work similarly to ConfigMaps but with a focus on sensitive data:

secrets-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Secret with encoded values (base64)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=        # base64 encoded 'admin'
  password: cGFzc3dvcmQxMjM=  # base64 encoded 'password123'
 
---
# Secret with plaintext (stringData is auto-encoded)
apiVersion: v1
kind: Secret
metadata:
  name: api-keys
type: Opaque
stringData:
  api-key: "sk-live-abc123xyz789"
  webhook-secret: "whsec_somelongsecret"
 
---
# Using Secrets in Pods
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:v1
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Secrets Aren't That Secret

By default, Kubernetes Secrets are only base64-encoded, not encrypted. Anyone with API access can read them. For real security: enable encryption at rest in etcd, use external secret managers (Vault, AWS Secrets Manager), or use tools like Sealed Secrets for GitOps workflows.

Resource Dependencies and Apply Order

Kubernetes resources often depend on each other. A Pod references a ConfigMap; a Deployment references a ServiceAccount; an Ingress references a Service. How does kubectl apply handle these dependencies?

The Short Answer: It Doesn't (Directly)

Kubernetes is eventually consistent. You can apply a Deployment that references a non-existent ConfigMap—the Pods will fail to start, but Kubernetes will retry. Once the ConfigMap is created, Pods will start successfully.

Dependency Ordering Considerations:

Common Resource Dependencies
Resource	Typical Dependencies	What Happens If Missing
Pod	ConfigMaps, Secrets, PVCs, ServiceAccount	Pod stays in Pending or container fails to start
Deployment	ConfigMaps, Secrets, ServiceAccount	Pods fail to start
Service	None (Pods selected by labels)	Service works but has no endpoints
Ingress	Service	Ingress backend unhealthy
PVC	StorageClass	PVC stays Pending

Strategies for Handling Dependencies:

1. Apply Everything Together

kubectl apply -f k8s/   # Apply entire directory

Kubernetes reconciliation will eventually resolve dependencies. This works well for most cases.

2. Apply in Order (Critical Dependencies)

kubectl apply -f k8s/namespaces.yaml
kubectl apply -f k8s/crds/          # Custom Resource Definitions first
kubectl apply -f k8s/configmaps/
kubectl apply -f k8s/secrets/
kubectl apply -f k8s/deployments/
kubectl apply -f k8s/services/
kubectl apply -f k8s/ingresses/

3. Use Init Containers for Runtime Dependencies

initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 1; done']

CRDs Are Special

Custom Resource Definitions (CRDs) must be applied and become Available before you can create Custom Resources of that type. Unlike other dependencies, this isn't eventually consistent—the API rejects unknown resource types. Always apply CRDs first.

Dry Run and Diff: Preview Before Applying

Before applying changes to production, you want to see what will change. kubectl provides tools for this.

Dry Run Modes:

dry-run-examples.txt
# Client-side dry run (no API validation)
kubectl apply -f deployment.yaml --dry-run=client
# Output: deployment.apps/web-app created (dry run)
# Only checks YAML syntax, not server-side validation
 
# Server-side dry run (full validation, no persist)
kubectl apply -f deployment.yaml --dry-run=server
# API server validates request, runs admission webhooks
# But doesn't persist the change
 
# Output the resulting manifest
kubectl apply -f deployment.yaml --dry-run=client -o yaml
# Shows what would be sent to API server
 
# Diff: See what would change
kubectl diff -f deployment.yaml
# Shows diff between local manifest and server state
# Like 'git diff' for Kubernetes resources
 
# Example diff output:
# -  replicas: 3
# +  replicas: 5
# -  image: myapp:v1
# +  image: myapp:v2

Best Practices for Safe Deployments:

Always diff first: kubectl diff -f manifest.yaml before apply
Use server-side dry run: Validates admission webhooks and quotas
Review changes in CI: Automated diff in pull requests
Start with staging: Test the exact manifest in staging first
Rollback quickly if needed: Know how to revert (rollout undo, apply previous)

GitOps Integration:

Tools like ArgoCD and Flux automate this workflow:

Detect changes in Git
Show diff in UI
Apply changes (with optional manual approval)
Track sync status

CI/CD Pipeline Example

In PRs, run 'kubectl diff' and post the output as a comment. Reviewers can see exactly what Kubernetes changes the PR introduces. This is like 'terraform plan' for Kubernetes—review before apply.

Declarative Deletion: Pruning Resources

Declarative configuration creates and updates resources—but what about deletion? If you remove a resource from your manifests, by default it stays in the cluster.

The Problem:

# Day 1: Apply all resources
kubectl apply -f k8s/
# Creates: web-deployment, api-deployment, old-deployment

# Day 2: Remove old-deployment.yaml from k8s/
kubectl apply -f k8s/
# Creates/updates: web-deployment, api-deployment
# old-deployment still exists! (orphaned)

The Solution: --prune

prune-examples.txt
# Prune: delete resources not in manifests (use carefully!)
kubectl apply -f k8s/ --prune -l app=myapp
 
# How it works:
# 1. Apply all manifests in k8s/
# 2. Find all resources with label app=myapp
# 3. Delete resources with that label not in manifests
 
# Safer: dry run first
kubectl apply -f k8s/ --prune -l app=myapp --dry-run=server
 
# Modern approach: --prune with --applyset (Kubernetes 1.27+)
kubectl apply -f k8s/ --prune --applyset=myapp-resources
# Tracks applied resources in a ConfigMap, prunes based on that
 
# Example workflow:
# 1. Always use consistent labels on all resources
# 2. Always use the same apply command with --prune -l
# 3. When you remove a manifest file, resource is deleted

GitOps Pruning:

GitOps tools handle this more elegantly:

ArgoCD: Configurable auto-prune or require manual approval
Flux: Prune by default with kustomize garbage collection

These track what was applied from Git and automatically delete orphaned resources.

Deletion Strategies:

Strategy	Behavior	Use Case
No prune	Orphaned resources remain	Legacy migration, careful control
--prune with labels	Delete matching orphans	Traditional kubectl GitOps
ApplySet pruning	Track and delete orphans	Modern kubectl (1.27+)
GitOps auto-prune	Tool tracks and deletes	ArgoCD, Flux

Pruning Can Be Dangerous

Misconfigured prune labels can delete resources you didn't intend. If your label selector is too broad, you might delete unrelated resources. Always use specific labels and test with --dry-run=server first.

Kustomize: Declarative Configuration Management

Kustomize is built into kubectl and provides a way to customize manifests without templates. It follows a declarative, overlay-based approach.

Core Concepts:

Base: Common resources shared across environments
Overlay: Environment-specific modifications
Patches: Targeted modifications to base resources
Generators: Create ConfigMaps/Secrets from files or literals

kustomize-structure.txt

Kustomize Project Structure:
 
app/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   │       └── replicas.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   ├── patches/
│   │   │   └── replicas.yaml
│   │   └── resources.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── patches/
│       │   ├── replicas.yaml
│       │   └── resources.yaml
│       └── hpa.yaml

kustomization-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - deployment.yaml
  - service.yaml
 
commonLabels:
  app: myapp
 
---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - ../../base
  - hpa.yaml
 
namespace: production
 
# Add name suffix
nameSuffix: -prod
 
# Override image tag
images:
  - name: myapp
    newTag: v2.1.0
 
# Apply patches
patches:
  - path: patches/replicas.yaml
  - path: patches/resources.yaml
 
# Generate ConfigMap from files
configMapGenerator:
  - name: app-config
    files:
      - config.properties
    options:
      labels:
        env: production
 
---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp  # matches base name
spec:
  replicas: 5  # override base value

Using Kustomize:

# Preview rendered manifests
kubectl kustomize overlays/production/

# Apply directly
kubectl apply -k overlays/production/

# Or with --dry-run
kubectl apply -k overlays/production/ --dry-run=server

Kustomize vs. Helm

Kustomize: Patch-based, no templates, built into kubectl. Best for customizing your own applications. Helm: Template-based, package manager, huge ecosystem. Best for third-party charts and complex parameterization. You can even combine them—use Helm to install base, Kustomize to customize.

Summary: Declarative Configuration Mastery

We've covered the declarative paradigm that makes Kubernetes powerful. Let's consolidate the key takeaways:

Key Takeaways

•Declarative beats imperative at scale: Specify what, not how; Kubernetes handles the rest
•Reconciliation loops enable self-healing: Controllers continuously drive toward desired state
•Manifests are structured consistently: apiVersion, kind, metadata, spec
•Use kubectl apply for declarative workflows: Three-way merge respects concurrent changes
•Organize manifests for maintainability: By component or with Kustomize overlays
•Externalize configuration: ConfigMaps for config, Secrets for sensitive data
•Preview before applying: kubectl diff and --dry-run=server prevent surprises
•Handle deletion with pruning: --prune or GitOps tools for complete sync
•Kustomize enables environment variations: Overlays and patches without duplication

What's Next:

You now understand Kubernetes' declarative model. The final page explores the broader Kubernetes ecosystem—the tools, extensions, and resources that make Kubernetes a complete platform for production workloads.

Page Complete

You now understand Kubernetes' declarative paradigm deeply—how manifests work, how reconciliation maintains state, and how to structure configurations for real projects. This knowledge is fundamental for effective Kubernetes operations. Next, we'll explore the broader Kubernetes ecosystem.

4 / 5

Loading learning content...

System Design HLDKubernetes Architecture

Understanding Kubernetes Architecture

LevelIntermediate

Duration120 mins

TopicKubernetes Architecture

4 / 5

Declarative Configuration: Tell Kubernetes What, Not How

The Power of Declarative Infrastructure

What You Will Learn

Imperative vs. Declarative: A Paradigm Shift

Before Kubernetes, infrastructure management was typically imperative—you wrote scripts that executed step-by-step:

Check if server exists
If not, create server
Install packages
Configure web server
Start service
Verify it's running

This approach has fundamental limitations:

Fragile: Any step failure breaks the chain
Non-idempotent: Running twice may cause errors
Drift: System state can diverge from what scripts assume
Hard to scale: More servers = more script execution = more failures

Kubernetes is declarative:

Imperative Approach

•You specify HOW: Step-by-step instructions
•One-time execution: Script runs once
•Drift is silent: System changes unnoticed
•Error handling is your job: Every edge case must be coded
•State is unknown: No record of what should exist

Declarative Approach

•You specify WHAT: Desired end state
•Continuous reconciliation: Controller loops forever
•Drift is corrected: System restored to desired state
•Kubernetes handles errors: Retries, backoff, self-healing
•State is explicit: Manifests ARE the source of truth

Example: Imperative Script vs. Declarative Manifest

Imperative (shell script):

#!/bin/bash
if ! kubectl get deployment nginx &>/dev/null; then
  kubectl create deployment nginx --image=nginx:1.25
fi
kubectl scale deployment nginx --replicas=3
kubectl set image deployment/nginx nginx=nginx:1.25
# What if a pod crashes? What if the node dies? Script doesn't know.

Declarative (YAML manifest):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels: { app: nginx }
  template:
    metadata:
      labels: { app: nginx }
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
# Apply once. Kubernetes maintains this state forever.
# Pod crashes? Replaced. Node dies? Rescheduled.

Declarative Enables Self-Healing

Anatomy of a Kubernetes Manifest

Every Kubernetes resource follows a consistent structure. Understanding this structure helps you read, write, and debug manifests effectively.

Required Fields:

Field	Purpose
`apiVersion`	API group and version (e.g., `apps/v1`, `v1`)
`kind`	Resource type (e.g., `Deployment`, `Service`)
`metadata`	Name, namespace, labels, annotations
`spec`	Desired state specification

Anatomy of a Complete Manifest:

manifest-structure.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# API version determines available fields and behaviors
apiVersion: apps/v1
 
# Resource type
kind: Deployment
 
# Metadata: identity and organization
metadata:
  # Required: resource name (unique within namespace for this kind)
  name: web-app
  
  # Optional: namespace (defaults to 'default')
  namespace: production
  
  # Optional: key-value pairs for organization and selection
  labels:
    app: web
    environment: production
    team: platform
    version: v2.1.0
  
  # Optional: non-identifying metadata (for tools, documentation)
  annotations:
    kubernetes.io/change-cause: "Deployed v2.1.0 with performance fixes"
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
 
# Spec: the desired state (structure varies by resource type)
spec:
  replicas: 3
  
  selector:
    matchLabels:
      app: web
  
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  
  # For Deployments, spec contains a Pod template
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: myapp:v2.1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
 
# Status: current state (managed by Kubernetes, not in manifests)
# status:
#   replicas: 3
#   readyReplicas: 3
#   availableReplicas: 3
#   conditions:
#     - type: Available
#       status: "True"

Labels vs. Annotations:

Aspect	Labels	Annotations
Purpose	Identify and select objects	Attach non-identifying metadata
Selection	Used in selectors and queries	Cannot be used for selection
Constraints	Limited to 63 chars, alphanumeric	Up to 256KB, any string
Examples	`app: web`, `env: prod`	`description: "Main API"`, `last-deployed: "2024-01-15"`

Common Label Conventions:

Label	Purpose
`app.kubernetes.io/name`	Application name
`app.kubernetes.io/version`	Application version
`app.kubernetes.io/component`	Component within application
`app.kubernetes.io/managed-by`	Tool managing this resource (Helm, ArgoCD)

Never Include 'status' in Manifests

The Reconciliation Loop: How Kubernetes Maintains State

The magic of Kubernetes lies in its reconciliation loops. Controllers continuously watch resources and work to make reality match your declared intent.

The Controller Pattern:

Every controller follows the same pattern:

Observe: Watch the API server for changes to relevant resources
Analyze: Compare current state with desired state
Act: If they differ, take actions to reconcile
Repeat: Loop forever

This is sometimes called a "level-triggered" system—controllers react to the current level (state) rather than edges (individual changes).

reconciliation-loop.txt
Reconciliation Loop Illustrated:
 
┌─────────────────────────────────────────────────────────────────┐
│                    RECONCILIATION LOOP                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  1. OBSERVE: Watch API server for Deployment changes            │
│     - User applies manifest (kubectl apply)                     │
│     - Receives event: "Deployment web-app updated"              │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  2. READ CURRENT STATE                                          │
│     - Query API: "What Pods exist with label app=web?"          │
│     - Result: 2 Pods running                                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  3. READ DESIRED STATE                                          │
│     - Query API: "What's the Deployment spec?"                  │
│     - Result: replicas=3                                        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  4. COMPARE                                                      │
│     - Current: 2 Pods                                           │
│     - Desired: 3 Pods                                           │
│     - Difference: Need 1 more Pod                               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  5. ACT                                                          │
│     - Create 1 new Pod with spec from template                  │
│     - Update Deployment status                                  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  6. REPEAT                                                       │
│     - Return to step 1, wait for next change                    │
│     - Loop runs forever until controller shuts down             │
└─────────────────────────────────────────────────────────────────┘

Why Level-Triggered Beats Edge-Triggered:

In an edge-triggered system, the controller would react to individual events: "Pod created", "Pod deleted". This creates problems:

What if an event is missed?
What if events arrive out of order?
How do you handle startup when state already exists?

Level-triggered systems don't care about event history—they only care about "what is current state vs. desired state?" This makes them:

Resilient: Controller restart just continues from current state
Idempotent: Running reconciliation multiple times is safe
Self-healing: Manual changes are detected and corrected

Eventual Consistency

kubectl apply vs. create vs. replace

Understanding the different kubectl commands for managing resources is crucial for declarative workflows.

kubectl create (Imperative):

Creates a new resource
Fails if resource already exists
One-time operation

kubectl apply (Declarative):

Creates if doesn't exist, updates if it does
Uses three-way merge to calculate changes
Safe to run repeatedly
Records last-applied configuration

kubectl replace (Imperative):

Replaces entire resource specification
Fails if resource doesn't exist
Overwrites any manual changes

Comparison:

kubectl Command Comparison
Command	Resource Exists	Resource Doesn't Exist	Use Case
kubectl create	Error	Creates	Initial creation scripts
kubectl apply	Updates (merge)	Creates	GitOps, declarative workflows
kubectl replace	Replaces	Error	Full manifest replacement
kubectl patch	Patches specific fields	Error	Targeted updates
kubectl delete	Deletes	No-op (or error)	Resource removal

How kubectl apply Works (Three-Way Merge):

kubectl apply compares three versions of the resource:

Local manifest (what you're applying)
Last-applied configuration (stored in annotation)
Live server state (current in cluster)

This three-way merge allows:

Detecting fields you've removed from manifest
Preserving fields set by other tools (e.g., autoscaler changing replicas)
Intelligently merging concurrent changes

apply-example.txt
Three-Way Merge Example:
 
Last Applied:
  replicas: 3
  image: myapp:v1
 
Live Server (after HPA scaled up):
  replicas: 5  ← Changed by HPA
  image: myapp:v1
 
New Local Manifest:
  replicas: 3
  image: myapp:v2  ← Your change
 
Result of kubectl apply:
  replicas: 5  ← Preserved (not in delta between last-applied and new)
  image: myapp:v2  ← Updated (you changed it)
 
This is why 'apply' is smart—it respects changes made by other actors.

Always Use Apply for GitOps

Organizing Manifests for Real Projects

Directory Structure Patterns:

directory-structure.txt
Pattern 1: By Resource Type (Simple Projects)
 
k8s/
├── deployments/
│   ├── web.yaml
│   └── api.yaml
├── services/
│   ├── web.yaml
│   └── api.yaml
├── configmaps/
│   └── app-config.yaml
└── ingresses/
    └── main-ingress.yaml
 
Pattern 2: By Component (Microservices)
 
k8s/
├── web/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── api/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap.yaml
│   └── secret.yaml  (reference only, actual secret managed separately)
├── database/
│   ├── statefulset.yaml
│   ├── service.yaml
│   └── pvc.yaml
└── common/
    ├── namespace.yaml
    └── network-policies.yaml
 
Pattern 3: By Environment with Kustomize
 
k8s/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── replicas-patch.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── replicas-patch.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── replicas-patch.yaml
│       └── resources-patch.yaml

Multi-Document YAML Files:

You can combine related resources in a single file using --- separator:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web }
  ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: nginx

File Naming Conventions:

Use lowercase with hyphens: web-deployment.yaml
Include resource type for clarity: api-service.yaml
Group related resources: monitoring.yaml (Prometheus + Grafana)

Use Kustomize or Helm for Environments

ConfigMaps and Secrets: Externalizing Configuration

Declarative configuration extends to application configuration itself. Instead of baking config into images, externalize it using ConfigMaps and Secrets.

ConfigMaps: Store non-sensitive configuration data Secrets: Store sensitive data (passwords, tokens, certificates)

Creating ConfigMaps:

configmap-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
# ConfigMap with key-value pairs
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres.database.svc.cluster.local"
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  
---
# ConfigMap with file content
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name localhost;
      location / {
        proxy_pass http://backend:8080;
      }
    }
 
---
# Using ConfigMaps in Pods
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:v1
      
      # Method 1: Environment variables from ConfigMap
      envFrom:
        - configMapRef:
            name: app-config
      
      # Method 2: Individual env vars
      env:
        - name: DB_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: DATABASE_HOST
      
      # Method 3: Mount as files
      volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d
  
  volumes:
    - name: config-volume
      configMap:
        name: nginx-config

Secrets:

Secrets work similarly to ConfigMaps but with a focus on sensitive data:

secrets-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Secret with encoded values (base64)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=        # base64 encoded 'admin'
  password: cGFzc3dvcmQxMjM=  # base64 encoded 'password123'
 
---
# Secret with plaintext (stringData is auto-encoded)
apiVersion: v1
kind: Secret
metadata:
  name: api-keys
type: Opaque
stringData:
  api-key: "sk-live-abc123xyz789"
  webhook-secret: "whsec_somelongsecret"
 
---
# Using Secrets in Pods
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:v1
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Secrets Aren't That Secret

Resource Dependencies and Apply Order

The Short Answer: It Doesn't (Directly)

Dependency Ordering Considerations:

Common Resource Dependencies
Resource	Typical Dependencies	What Happens If Missing
Pod	ConfigMaps, Secrets, PVCs, ServiceAccount	Pod stays in Pending or container fails to start
Deployment	ConfigMaps, Secrets, ServiceAccount	Pods fail to start
Service	None (Pods selected by labels)	Service works but has no endpoints
Ingress	Service	Ingress backend unhealthy
PVC	StorageClass	PVC stays Pending

Strategies for Handling Dependencies:

1. Apply Everything Together

kubectl apply -f k8s/   # Apply entire directory

Kubernetes reconciliation will eventually resolve dependencies. This works well for most cases.

2. Apply in Order (Critical Dependencies)

kubectl apply -f k8s/namespaces.yaml
kubectl apply -f k8s/crds/          # Custom Resource Definitions first
kubectl apply -f k8s/configmaps/
kubectl apply -f k8s/secrets/
kubectl apply -f k8s/deployments/
kubectl apply -f k8s/services/
kubectl apply -f k8s/ingresses/

3. Use Init Containers for Runtime Dependencies

initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 1; done']

CRDs Are Special

Dry Run and Diff: Preview Before Applying

Before applying changes to production, you want to see what will change. kubectl provides tools for this.

Dry Run Modes:

dry-run-examples.txt
# Client-side dry run (no API validation)
kubectl apply -f deployment.yaml --dry-run=client
# Output: deployment.apps/web-app created (dry run)
# Only checks YAML syntax, not server-side validation
 
# Server-side dry run (full validation, no persist)
kubectl apply -f deployment.yaml --dry-run=server
# API server validates request, runs admission webhooks
# But doesn't persist the change
 
# Output the resulting manifest
kubectl apply -f deployment.yaml --dry-run=client -o yaml
# Shows what would be sent to API server
 
# Diff: See what would change
kubectl diff -f deployment.yaml
# Shows diff between local manifest and server state
# Like 'git diff' for Kubernetes resources
 
# Example diff output:
# -  replicas: 3
# +  replicas: 5
# -  image: myapp:v1
# +  image: myapp:v2

Best Practices for Safe Deployments:

Always diff first: kubectl diff -f manifest.yaml before apply
Use server-side dry run: Validates admission webhooks and quotas
Review changes in CI: Automated diff in pull requests
Start with staging: Test the exact manifest in staging first
Rollback quickly if needed: Know how to revert (rollout undo, apply previous)

GitOps Integration:

Tools like ArgoCD and Flux automate this workflow:

Detect changes in Git
Show diff in UI
Apply changes (with optional manual approval)
Track sync status

CI/CD Pipeline Example

In PRs, run 'kubectl diff' and post the output as a comment. Reviewers can see exactly what Kubernetes changes the PR introduces. This is like 'terraform plan' for Kubernetes—review before apply.

Declarative Deletion: Pruning Resources

Declarative configuration creates and updates resources—but what about deletion? If you remove a resource from your manifests, by default it stays in the cluster.

The Problem:

# Day 1: Apply all resources
kubectl apply -f k8s/
# Creates: web-deployment, api-deployment, old-deployment

# Day 2: Remove old-deployment.yaml from k8s/
kubectl apply -f k8s/
# Creates/updates: web-deployment, api-deployment
# old-deployment still exists! (orphaned)

The Solution: --prune

prune-examples.txt
# Prune: delete resources not in manifests (use carefully!)
kubectl apply -f k8s/ --prune -l app=myapp
 
# How it works:
# 1. Apply all manifests in k8s/
# 2. Find all resources with label app=myapp
# 3. Delete resources with that label not in manifests
 
# Safer: dry run first
kubectl apply -f k8s/ --prune -l app=myapp --dry-run=server
 
# Modern approach: --prune with --applyset (Kubernetes 1.27+)
kubectl apply -f k8s/ --prune --applyset=myapp-resources
# Tracks applied resources in a ConfigMap, prunes based on that
 
# Example workflow:
# 1. Always use consistent labels on all resources
# 2. Always use the same apply command with --prune -l
# 3. When you remove a manifest file, resource is deleted

GitOps Pruning:

GitOps tools handle this more elegantly:

ArgoCD: Configurable auto-prune or require manual approval
Flux: Prune by default with kustomize garbage collection

These track what was applied from Git and automatically delete orphaned resources.

Deletion Strategies:

Strategy	Behavior	Use Case
No prune	Orphaned resources remain	Legacy migration, careful control
--prune with labels	Delete matching orphans	Traditional kubectl GitOps
ApplySet pruning	Track and delete orphans	Modern kubectl (1.27+)
GitOps auto-prune	Tool tracks and deletes	ArgoCD, Flux

Pruning Can Be Dangerous

Kustomize: Declarative Configuration Management

Kustomize is built into kubectl and provides a way to customize manifests without templates. It follows a declarative, overlay-based approach.

Core Concepts:

Base: Common resources shared across environments
Overlay: Environment-specific modifications
Patches: Targeted modifications to base resources
Generators: Create ConfigMaps/Secrets from files or literals

kustomize-structure.txt

Kustomize Project Structure:
 
app/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   │       └── replicas.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   ├── patches/
│   │   │   └── replicas.yaml
│   │   └── resources.yaml
│   └── production/
│       ├── kustomization.yaml
│       ├── patches/
│       │   ├── replicas.yaml
│       │   └── resources.yaml
│       └── hpa.yaml

kustomization-examples.yaml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - deployment.yaml
  - service.yaml
 
commonLabels:
  app: myapp
 
---
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - ../../base
  - hpa.yaml
 
namespace: production
 
# Add name suffix
nameSuffix: -prod
 
# Override image tag
images:
  - name: myapp
    newTag: v2.1.0
 
# Apply patches
patches:
  - path: patches/replicas.yaml
  - path: patches/resources.yaml
 
# Generate ConfigMap from files
configMapGenerator:
  - name: app-config
    files:
      - config.properties
    options:
      labels:
        env: production
 
---
# overlays/production/patches/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp  # matches base name
spec:
  replicas: 5  # override base value

Using Kustomize:

# Preview rendered manifests
kubectl kustomize overlays/production/

# Apply directly
kubectl apply -k overlays/production/

# Or with --dry-run
kubectl apply -k overlays/production/ --dry-run=server

Kustomize vs. Helm

Summary: Declarative Configuration Mastery

We've covered the declarative paradigm that makes Kubernetes powerful. Let's consolidate the key takeaways:

Key Takeaways

•Declarative beats imperative at scale: Specify what, not how; Kubernetes handles the rest
•Reconciliation loops enable self-healing: Controllers continuously drive toward desired state
•Manifests are structured consistently: apiVersion, kind, metadata, spec
•Use kubectl apply for declarative workflows: Three-way merge respects concurrent changes
•Organize manifests for maintainability: By component or with Kustomize overlays
•Externalize configuration: ConfigMaps for config, Secrets for sensitive data
•Preview before applying: kubectl diff and --dry-run=server prevent surprises
•Handle deletion with pruning: --prune or GitOps tools for complete sync
•Kustomize enables environment variations: Overlays and patches without duplication

What's Next:

Page Complete

4 / 5