System Design (HLD)Kubernetes Operations

Kubernetes Operations

LevelAdvanced

Duration90 mins

TopicKubernetes Operations

5 / 5

Security Best Practices

Security is Non-Negotiable

Kubernetes has transformed how we deploy and manage applications, but it has also introduced new security challenges. A Kubernetes cluster runs with significant privileges, orchestrates access to sensitive data, and exposes a large attack surface across nodes, networks, and control plane components.

The stakes are high. A compromised Kubernetes cluster can lead to:

Data exfiltration (customer data, credentials, secrets)
Cryptomining (attackers using your compute resources)
Lateral movement (access to other systems in your infrastructure)
Service disruption (ransomware, resource exhaustion)
Compliance violations (GDPR, HIPAA, PCI-DSS)

Kubernetes security requires defense in depth—multiple layers of protection so that a breach in one layer doesn't compromise the entire system. This page covers the essential security controls for production Kubernetes operations.

What You Will Learn

By the end of this page, you'll understand how to implement Pod Security Standards, configure RBAC for least privilege, enforce network segmentation with Network Policies, manage secrets securely, harden container images, protect the supply chain, and implement runtime security monitoring.

Kubernetes Security Layers

Kubernetes security spans multiple layers, from the cluster infrastructure through application code. Understanding these layers helps you build comprehensive protection.

The 4 C's of Cloud-Native Security:

Cloud/Datacenter: Infrastructure security (network, physical access, IAM)
Cluster: Kubernetes control plane, API server, etcd
Container: Container runtime, images, registries
Code: Application security, dependencies, secrets handling

Each layer must be secured—a vulnerability at any layer can compromise the entire stack.

Security Controls by Layer
Layer	Key Controls	Kubernetes Components
Cloud/Datacenter	Network segmentation, IAM, encryption at rest	Node VMs, storage, load balancers
Cluster	API authentication, RBAC, audit logging, admission control	API server, etcd, kubelet
Container	Pod Security, image scanning, runtime policies	Pods, containers, images
Code	Secrets management, dependency scanning, input validation	Application code, ConfigMaps, Secrets

Security is Shared Responsibility

In managed Kubernetes (EKS, GKE, AKS), the cloud provider secures the control plane. You're responsible for worker node security, pod security, network policies, RBAC, and application security. In self-managed clusters, you own everything.

Pod Security Standards (PSS)

Pod Security Standards (PSS) define three profiles that restrict what pods can do. Pod Security Admission (PSA) enforces these standards at the namespace level, replacing the deprecated PodSecurityPolicy.

The Three Profiles:

1. Privileged (Unrestricted)

No restrictions
Use only for system-level workloads that need full access (CNI, storage drivers)
Should be applied only to trusted namespaces (kube-system)

2. Baseline (Minimally Restrictive)

Prevents known privilege escalations
Allows reasonable defaults for most applications
Blocks hostNetwork, hostPID, hostIPC, privileged containers

3. Restricted (Highly Restrictive)

Current best practices for hardened pods
Requires running as non-root, read-only root filesystem
Blocks ALL host access, capabilities, and privilege escalation
Goal: all production workloads should comply with Restricted

pod-security-namespace.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Apply Pod Security Standards via namespace labels
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # Enforce restricted profile (block non-compliant pods)
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    
    # Warn on violations (log but allow)
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
    
    # Audit violations (add to audit log)
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
 
---
# Namespace for system components (privileged)
apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
 
---
# Development namespace (baseline for flexibility)
apiVersion: v1
kind: Namespace
metadata:
  name: development
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted    # Warn about restricted violations

restricted-compliant-pod.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# Pod that complies with Restricted profile
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  # No host namespaces
  hostNetwork: false
  hostPID: false
  hostIPC: false
  
  securityContext:
    # Run as non-root
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    
    # Secure sysctls only
    sysctls: []
    
    # Seccomp profile (required for restricted)
    seccompProfile:
      type: RuntimeDefault
  
  containers:
  - name: app
    image: myapp:v1.0.0
    
    securityContext:
      # Cannot become root
      allowPrivilegeEscalation: false
      
      # Run as non-root user
      runAsNonRoot: true
      runAsUser: 1000
      
      # Drop all capabilities
      capabilities:
        drop:
          - ALL
      
      # Read-only root filesystem
      readOnlyRootFilesystem: true
      
      # Privileged = false (default, but explicit)
      privileged: false
    
    # If app needs to write, use emptyDir
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /app/cache
      
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Migration to Restricted

Start with 'warn' mode to identify violations without blocking deployments. Fix applications one by one, then move to 'enforce'. Many applications need changes: run as non-root user, don't write to filesystem (use emptyDir), don't require capabilities.

RBAC: Role-Based Access Control

RBAC (Role-Based Access Control) is Kubernetes' authorization mechanism. It controls who can do what on which resources. Properly configured RBAC is essential—overly permissive RBAC is a primary attack vector.

RBAC Components:

Role: Defines permissions within a namespace
ClusterRole: Defines permissions cluster-wide
RoleBinding: Grants Role permissions to users/groups/service accounts in a namespace
ClusterRoleBinding: Grants ClusterRole permissions cluster-wide

Subjects (Who):

Users (managed externally via OIDC, certificates)
Groups (logical groupings of users)
ServiceAccounts (identities for pods)

Principle of Least Privilege:

Every identity should have only the minimum permissions needed. Start with no access, add specific permissions as needed. Never grant cluster-admin to applications.

rbac-examples.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# Role: permissions within a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: deployment-manager
rules:
# Can manage deployments
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Can view pods and logs
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
# Can view services
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch"]
 
---
# RoleBinding: grant role to a group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployment-managers-binding
  namespace: production
subjects:
# Grant to a group (managed via OIDC/IdP)
- kind: Group
  name: deployment-managers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io
 
---
# ServiceAccount for application pods
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
automountServiceAccountToken: false    # Opt-in only
 
---
# Role for the application (minimal permissions)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: myapp-role
rules:
# Only read configmaps it needs
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["myapp-config"]    # Specific resource names!
  verbs: ["get", "watch"]
# Only read its own secret
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["myapp-credentials"]
  verbs: ["get"]
 
---
# Bind role to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: production
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

RBAC Anti-Patterns to Avoid

•cluster-admin to workloads: Never give applications cluster-admin; they don't need it
•Wildcard resources (*): resources: ['*'] grants access to everything in the API group
•Wildcard verbs: verbs: ['*'] includes delete, escalate, impersonate
•ClusterRoleBinding when Role suffices: Grant namespace-scoped permissions when possible
•Default service account permissions: The 'default' SA in each namespace shouldn't have extra permissions
•bind and escalate verbs: These allow privilege escalation; restrict carefully

RBAC Auditing

Use tools like kubectl-who-can (who can delete pods?), rbac-lookup (what permissions does this user have?), and rakkess (show access matrix) to audit your RBAC configuration. Regular audits catch permission creep.

Network Policies: Microsegmentation

By default, all pods in a Kubernetes cluster can communicate with each other—no network segmentation. Network Policies implement microsegmentation, controlling which pods can talk to which other pods or external endpoints.

Why Network Policies Matter:

Limit blast radius: A compromised pod can only reach what's explicitly allowed
Enforce service boundaries: Frontend can reach backend, but not the database directly
Compliance requirements: Many regulations require network segmentation
Defense in depth: Network-level protection complements application authentication

Network Policy Behavior:

By default, all traffic is allowed
Once ANY policy selects a pod, only traffic matching a policy is allowed (default deny)
Policies are additive—multiple policies combine (union of allowed traffic)
Policies don't apply to traffic within the same pod (localhost)

network-policies.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
# Default deny all ingress (baseline security)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}     # Applies to all pods in namespace
  policyTypes:
  - Ingress           # Only affects ingress; egress still allowed
 
---
# Default deny all egress (strict environments)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
 
---
# Allow frontend to receive traffic from ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx     # Traffic from ingress namespace
      podSelector:
        matchLabels:
          app: ingress-nginx      # Specifically from ingress controller pods
    ports:
    - protocol: TCP
      port: 8080
 
---
# Allow frontend to talk to backend API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 8080
 
---
# Allow backend to reach database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
 
---
# Allow DNS resolution for all pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

CNI Requirements

Network Policies require a CNI plugin that supports them. Calico, Cilium, and WeaveNet support Network Policies. The default kubenet and AWS VPC CNI (without add-ons) do NOT. Check your CNI before relying on Network Policies!

Secrets Management: Protecting Sensitive Data

Kubernetes Secrets store sensitive data (passwords, tokens, certificates). However, native Kubernetes Secrets are only base64 encoded, not encrypted. Additional protection is essential.

Kubernetes Secrets Limitations:

Stored in etcd (must be encrypted at rest)
Base64 is not encryption (trivial to decode)
Accessible to anyone with RBAC read access
Often logged or exposed in pod specs
No automatic rotation

Defense Layers for Secrets:

etcd encryption: Enable encryption at rest for etcd
RBAC: Limit who can read secrets
External secrets managers: HashiCorp Vault, AWS Secrets Manager, etc.
Sealed Secrets: Encrypt secrets for GitOps
CSI Secret Store: Mount secrets directly from external providers

secrets-security.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# Encryption configuration for etcd (control plane config)
# /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    - configmaps
    providers:
    # Use AES-GCM with 256-bit key
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-32-byte-key>
    # Fallback to identity (for reading old unencrypted secrets)
    - identity: {}
 
---
# External Secrets Operator: sync from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-credentials    # K8s secret to create
    creationPolicy: Owner
  data:
  - secretKey: password           # Key in K8s secret
    remoteRef:
      key: prod/database/master   # Path in Secrets Manager
      property: password          # JSON property to extract
 
---
# Sealed Secret (for GitOps - safe to commit to git)
# Created with: kubeseal < secret.yaml > sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-password
  namespace: production
spec:
  encryptedData:
    # Encrypted with cluster's sealing key
    password: AgBOQ...encrypted...data...
 
---
# CSI Secret Store (mount directly from Vault/AWS)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database-creds
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com:8200"
    roleName: "database-role"
    objects: |
      - objectName: "password"
        secretPath: "secret/data/production/database"
        secretKey: "password"

Secrets Best Practices

•Enable etcd encryption at rest: Prevents reading secrets from etcd backups
•Use external secrets managers: Vault, AWS Secrets Manager, etc. for production
•Limit secret access via RBAC: Only pods that need secrets should access them
•Never log secrets: Ensure application code doesn't print credentials
•Rotate regularly: Implement rotation—secrets should have limited lifetimes
•Audit secret access: Enable API server audit logging to track who accesses secrets
•Use Sealed Secrets for GitOps: Safe to commit encrypted secrets to git

Image Security: Supply Chain Protection

Container images are the artifact you deploy. If an image is compromised—through malicious base images, vulnerable dependencies, or supply chain attacks—your cluster is compromised.

Image Security Controls:

Scan for vulnerabilities: Check images for known CVEs before deployment
Sign and verify images: Ensure images haven't been tampered with
Use minimal base images: Smaller images = smaller attack surface
Pin versions: Never use :latest in production
Private registries: Don't pull from public Docker Hub in production
Admission policies: Block deployment of non-compliant images

image-security.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# ValidatingAdmissionPolicy (K8s 1.26+)
# Block images without digest or from untrusted registries
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: image-policy
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  # Require images from trusted registry
  - expression: |
      object.spec.containers.all(c, 
        c.image.startsWith('registry.example.com/') ||
        c.image.startsWith('gcr.io/my-project/')
      )
    message: "Images must be from trusted registries"
    
  # Require image digest (not just tag)
  - expression: |
      object.spec.containers.all(c,
        c.image.contains('@sha256:')
      )
    message: "Images must use digest, not tag"
 
---
# Kyverno policy (alternative to ValidatingAdmissionPolicy)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: verify-signature
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - imageReferences:
      - "registry.example.com/*"
      attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              <your cosign public key>
              -----END PUBLIC KEY-----
 
---
# Image pull secrets for private registries
apiVersion: v1
kind: Secret
metadata:
  name: registry-credentials
  namespace: production
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-docker-config>
 
---
# ServiceAccount with imagePullSecrets
apiVersion: v1
kind: ServiceAccount
metadata:
  name: production-sa
  namespace: production
imagePullSecrets:
- name: registry-credentials

Base Image Recommendations
Image Type	Example	Size	Use Case
Scratch	scratch	0 MB	Static binaries (Go)
Distroless	gcr.io/distroless/static	~2 MB	Static binaries with CA certs
Alpine	alpine:3.19	~5 MB	When shell/packages needed
Wolfi/Chainguard	cgr.dev/chainguard/static	~2 MB	Hardened, minimal, updated
Ubuntu/Debian	ubuntu:22.04	~77 MB	Legacy apps requiring glibc

Never Use :latest

The :latest tag is mutable—it can point to different images over time. This breaks reproducibility, makes rollbacks impossible, and can introduce untested changes. Always use immutable tags (v1.2.3) or digests (sha256:abc123).

Runtime Security: Detecting Threats in Real-Time

Preventive controls (RBAC, Network Policies, Pod Security) reduce attack surface, but determined attackers may still get through. Runtime security detects and responds to malicious activity as it happens.

Runtime Security Tools:

Falco: Open-source runtime detection using syscall monitoring
Sysdig Secure: Commercial runtime security with response capabilities
Aqua Security: End-to-end container security platform
Tetragon: eBPF-based security observability from Cilium

What Runtime Security Detects:

Shell spawned inside container
Unexpected network connections
File access in sensitive directories
Process spawned from unexpected parent
Privilege escalation attempts
Cryptomining indicators

falco-rules.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
# Falco custom rules for Kubernetes security
# Deploy Falco as DaemonSet with host access
 
- rule: Shell Spawned in Container
  desc: Detect shell spawned inside a container (potential breakout attempt)
  condition: >
    spawned_process and
    container and
    shell_procs and
    not shell_allowed_container
  output: >
    Shell spawned in container 
    (user=%user.name container=%container.name shell=%proc.name 
     parent=%proc.pname cmdline=%proc.cmdline container_id=%container.id 
     image=%container.image.repository)
  priority: WARNING
  tags: [container, shell, mitre_execution]
 
- rule: Unexpected Outbound Connection
  desc: Detect container connecting to unexpected IP
  condition: >
    outbound and
    container and
    not allowed_outbound_ip and
    not k8s_known_connections
  output: >
    Unexpected outbound connection 
    (command=%proc.cmdline connection=%fd.name container=%container.name 
     image=%container.image.repository)
  priority: NOTICE
  
- rule: Sensitive File Read
  desc: Detect read of sensitive files (credentials, keys)
  condition: >
    open_read and
    container and
    (
      fd.name startswith /etc/shadow or
      fd.name startswith /etc/passwd or
      fd.name contains id_rsa or
      fd.name contains .kube/config or
      fd.name contains .aws/credentials
    )
  output: >
    Sensitive file read 
    (file=%fd.name command=%proc.cmdline container=%container.name)
  priority: WARNING
 
- rule: Kubernetes Secret Mounted and Read
  desc: Detect when mounted K8s secrets are read unexpectedly
  condition: >
    open_read and
    container and
    fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount and
    not k8s_trusted_processes
  output: >
    Kubernetes secret read 
    (file=%fd.name proc=%proc.name container=%container.name 
     pod=%k8s.pod.name namespace=%k8s.ns.name)
  priority: WARNING
 
- rule: Cryptomining Detected
  desc: Detect cryptomining activity
  condition: >
    spawned_process and
    container and
    (
      proc.name in (cryptominer_processes) or
      proc.cmdline contains "stratum+tcp://" or
      proc.cmdline contains "-o pool." or
      proc.cmdline contains "xmrig" or
      proc.cmdline contains "minerd"
    )
  output: >
    Cryptomining process detected 
    (command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: CRITICAL

Runtime Security Response

Detection without response is just logging. Integrate Falco with: 1) Alerting (PagerDuty, Slack) for human response 2) Automated response (kill pod, isolate network) for critical threats 3) Forensics (capture state before cleanup). Note: Automated kill should be used carefully to avoid self-denial-of-service.

Audit Logging: Who Did What When

Kubernetes audit logs record all requests to the API server. They're essential for security investigations, compliance, and detecting anomalous behavior.

Audit Log Stages:

RequestReceived: Logged as soon as request is received (before processing)
ResponseStarted: For long-running requests (watch); response headers sent
ResponseComplete: Response body completed (most common to log)
Panic: Events generated if panic occurs

Audit Policy Levels:

None: Don't log
Metadata: Log request metadata (who, what, when) but not body
Request: Log metadata + request body
RequestResponse: Log everything including response body (verbose)

Considerations:

Audit logs can be high volume—filter appropriately
Sensitive data in request/response bodies—consider omitting
Storage and retention—audit logs must be durable

audit-policy.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# Kubernetes API Server Audit Policy
apiVersion: audit.k8s.io/v1
kind: Policy
 
# Don't log requests to these URLs
omitStages:
- "RequestReceived"
 
rules:
# Don't log read-only endpoints (reduces volume)
- level: None
  users: ["system:kube-probe"]
  
# Don't log health checks
- level: None
  nonResourceURLs:
  - "/healthz*"
  - "/livez*"
  - "/readyz*"
  - "/metrics"
 
# Log secret access at Request level (who accessed what)
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
    
# Log RBAC changes (security critical)
- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
 
# Log pod creation/deletion (detect cryptominers, etc)
- level: Request
  verbs: ["create", "delete", "patch", "update"]
  resources:
  - group: ""
    resources: ["pods"]
  - group: "apps"
    resources: ["deployments", "daemonsets", "statefulsets"]
 
# Log authentication failures
- level: Metadata
  users: ["system:anonymous"]
  verbs: ["*"]
 
# Log exec into pods (security sensitive)
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods/exec", "pods/attach", "pods/portforward"]
 
# Default: log metadata for everything else
- level: Metadata
  resources:
  - group: ""
  - group: "apps"
  - group: "batch"
  - group: "networking.k8s.io"

Audit Log Analysis

Audit logs are most useful when analyzed, not just stored. Ship audit logs to your SIEM (Splunk, Elastic, etc.) and create alerts for: 1) Secret access by unexpected service accounts 2) RBAC changes 3) exec/attach to production pods 4) Anonymous authentication attempts. Regular review catches security issues before they become incidents.

Summary: Security Best Practices Checklist

Kubernetes security requires a layered approach—no single control is sufficient. Let's consolidate the essential security controls for production clusters:

Security Checklist

•Enforce Pod Security Standards: Use 'restricted' profile for production namespaces
•Implement least-privilege RBAC: No wildcards, specific resources and verbs only
•Apply Network Policies: Default deny with explicit allow rules for required traffic
•Encrypt secrets at rest: Enable etcd encryption, use external secrets managers
•Scan images for vulnerabilities: Block deployment of images with critical CVEs
•Sign and verify images: Ensure supply chain integrity with cosign/notation
•Use minimal base images: Distroless or scratch where possible
•Deploy runtime security: Falco or similar for real-time threat detection
•Enable audit logging: Log security-sensitive API operations
•Regular security reviews: Audit RBAC, network policies, and container configs periodically

Module Complete:

Congratulations! You've completed the Kubernetes Operations module. You now have comprehensive knowledge of:

Resource Management: CPU, memory, QoS, quotas, and limit ranges
Auto-Scaling: HPA, VPA, Cluster Autoscaler, and Karpenter
Deployment Strategies: Rolling updates, rollbacks, blue-green, and canary
Observability: Metrics, logging, tracing, and alerting
Security: Pod Security, RBAC, Network Policies, and runtime protection

These operational skills distinguish production-ready Kubernetes environments from development clusters.

Module Complete

You've mastered Kubernetes Operations—the essential knowledge for running secure, observable, and efficient Kubernetes clusters in production. Apply these practices systematically to build infrastructure that scales reliably and survives real-world operational challenges.

5 / 5

Loading learning content...

System Design (HLD)Kubernetes Operations

Kubernetes Operations

LevelAdvanced

Duration90 mins

TopicKubernetes Operations

5 / 5

Security Best Practices

Security is Non-Negotiable

The stakes are high. A compromised Kubernetes cluster can lead to:

Data exfiltration (customer data, credentials, secrets)
Cryptomining (attackers using your compute resources)
Lateral movement (access to other systems in your infrastructure)
Service disruption (ransomware, resource exhaustion)
Compliance violations (GDPR, HIPAA, PCI-DSS)

What You Will Learn

Kubernetes Security Layers

Kubernetes security spans multiple layers, from the cluster infrastructure through application code. Understanding these layers helps you build comprehensive protection.

The 4 C's of Cloud-Native Security:

Cloud/Datacenter: Infrastructure security (network, physical access, IAM)
Cluster: Kubernetes control plane, API server, etcd
Container: Container runtime, images, registries
Code: Application security, dependencies, secrets handling

Each layer must be secured—a vulnerability at any layer can compromise the entire stack.

Security Controls by Layer
Layer	Key Controls	Kubernetes Components
Cloud/Datacenter	Network segmentation, IAM, encryption at rest	Node VMs, storage, load balancers
Cluster	API authentication, RBAC, audit logging, admission control	API server, etcd, kubelet
Container	Pod Security, image scanning, runtime policies	Pods, containers, images
Code	Secrets management, dependency scanning, input validation	Application code, ConfigMaps, Secrets

Security is Shared Responsibility

Pod Security Standards (PSS)

The Three Profiles:

1. Privileged (Unrestricted)

No restrictions
Use only for system-level workloads that need full access (CNI, storage drivers)
Should be applied only to trusted namespaces (kube-system)

2. Baseline (Minimally Restrictive)

Prevents known privilege escalations
Allows reasonable defaults for most applications
Blocks hostNetwork, hostPID, hostIPC, privileged containers

3. Restricted (Highly Restrictive)

Current best practices for hardened pods
Requires running as non-root, read-only root filesystem
Blocks ALL host access, capabilities, and privilege escalation
Goal: all production workloads should comply with Restricted

pod-security-namespace.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Apply Pod Security Standards via namespace labels
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # Enforce restricted profile (block non-compliant pods)
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    
    # Warn on violations (log but allow)
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
    
    # Audit violations (add to audit log)
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
 
---
# Namespace for system components (privileged)
apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
 
---
# Development namespace (baseline for flexibility)
apiVersion: v1
kind: Namespace
metadata:
  name: development
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted    # Warn about restricted violations

restricted-compliant-pod.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# Pod that complies with Restricted profile
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  # No host namespaces
  hostNetwork: false
  hostPID: false
  hostIPC: false
  
  securityContext:
    # Run as non-root
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    
    # Secure sysctls only
    sysctls: []
    
    # Seccomp profile (required for restricted)
    seccompProfile:
      type: RuntimeDefault
  
  containers:
  - name: app
    image: myapp:v1.0.0
    
    securityContext:
      # Cannot become root
      allowPrivilegeEscalation: false
      
      # Run as non-root user
      runAsNonRoot: true
      runAsUser: 1000
      
      # Drop all capabilities
      capabilities:
        drop:
          - ALL
      
      # Read-only root filesystem
      readOnlyRootFilesystem: true
      
      # Privileged = false (default, but explicit)
      privileged: false
    
    # If app needs to write, use emptyDir
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /app/cache
      
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Migration to Restricted

RBAC: Role-Based Access Control

RBAC Components:

Role: Defines permissions within a namespace
ClusterRole: Defines permissions cluster-wide
RoleBinding: Grants Role permissions to users/groups/service accounts in a namespace
ClusterRoleBinding: Grants ClusterRole permissions cluster-wide

Subjects (Who):

Users (managed externally via OIDC, certificates)
Groups (logical groupings of users)
ServiceAccounts (identities for pods)

Principle of Least Privilege:

Every identity should have only the minimum permissions needed. Start with no access, add specific permissions as needed. Never grant cluster-admin to applications.

rbac-examples.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# Role: permissions within a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: deployment-manager
rules:
# Can manage deployments
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# Can view pods and logs
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
# Can view services
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch"]
 
---
# RoleBinding: grant role to a group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployment-managers-binding
  namespace: production
subjects:
# Grant to a group (managed via OIDC/IdP)
- kind: Group
  name: deployment-managers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io
 
---
# ServiceAccount for application pods
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
automountServiceAccountToken: false    # Opt-in only
 
---
# Role for the application (minimal permissions)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: myapp-role
rules:
# Only read configmaps it needs
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["myapp-config"]    # Specific resource names!
  verbs: ["get", "watch"]
# Only read its own secret
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["myapp-credentials"]
  verbs: ["get"]
 
---
# Bind role to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: production
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

RBAC Anti-Patterns to Avoid

•cluster-admin to workloads: Never give applications cluster-admin; they don't need it
•Wildcard resources (*): resources: ['*'] grants access to everything in the API group
•Wildcard verbs: verbs: ['*'] includes delete, escalate, impersonate
•ClusterRoleBinding when Role suffices: Grant namespace-scoped permissions when possible
•Default service account permissions: The 'default' SA in each namespace shouldn't have extra permissions
•bind and escalate verbs: These allow privilege escalation; restrict carefully

RBAC Auditing

Network Policies: Microsegmentation

Why Network Policies Matter:

Limit blast radius: A compromised pod can only reach what's explicitly allowed
Enforce service boundaries: Frontend can reach backend, but not the database directly
Compliance requirements: Many regulations require network segmentation
Defense in depth: Network-level protection complements application authentication

Network Policy Behavior:

By default, all traffic is allowed
Once ANY policy selects a pod, only traffic matching a policy is allowed (default deny)
Policies are additive—multiple policies combine (union of allowed traffic)
Policies don't apply to traffic within the same pod (localhost)

network-policies.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
# Default deny all ingress (baseline security)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}     # Applies to all pods in namespace
  policyTypes:
  - Ingress           # Only affects ingress; egress still allowed
 
---
# Default deny all egress (strict environments)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
 
---
# Allow frontend to receive traffic from ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx     # Traffic from ingress namespace
      podSelector:
        matchLabels:
          app: ingress-nginx      # Specifically from ingress controller pods
    ports:
    - protocol: TCP
      port: 8080
 
---
# Allow frontend to talk to backend API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 8080
 
---
# Allow backend to reach database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
 
---
# Allow DNS resolution for all pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

CNI Requirements

Secrets Management: Protecting Sensitive Data

Kubernetes Secrets store sensitive data (passwords, tokens, certificates). However, native Kubernetes Secrets are only base64 encoded, not encrypted. Additional protection is essential.

Kubernetes Secrets Limitations:

Stored in etcd (must be encrypted at rest)
Base64 is not encryption (trivial to decode)
Accessible to anyone with RBAC read access
Often logged or exposed in pod specs
No automatic rotation

Defense Layers for Secrets:

etcd encryption: Enable encryption at rest for etcd
RBAC: Limit who can read secrets
External secrets managers: HashiCorp Vault, AWS Secrets Manager, etc.
Sealed Secrets: Encrypt secrets for GitOps
CSI Secret Store: Mount secrets directly from external providers

secrets-security.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# Encryption configuration for etcd (control plane config)
# /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    - configmaps
    providers:
    # Use AES-GCM with 256-bit key
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-32-byte-key>
    # Fallback to identity (for reading old unencrypted secrets)
    - identity: {}
 
---
# External Secrets Operator: sync from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-credentials    # K8s secret to create
    creationPolicy: Owner
  data:
  - secretKey: password           # Key in K8s secret
    remoteRef:
      key: prod/database/master   # Path in Secrets Manager
      property: password          # JSON property to extract
 
---
# Sealed Secret (for GitOps - safe to commit to git)
# Created with: kubeseal < secret.yaml > sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-password
  namespace: production
spec:
  encryptedData:
    # Encrypted with cluster's sealing key
    password: AgBOQ...encrypted...data...
 
---
# CSI Secret Store (mount directly from Vault/AWS)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database-creds
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.example.com:8200"
    roleName: "database-role"
    objects: |
      - objectName: "password"
        secretPath: "secret/data/production/database"
        secretKey: "password"

Secrets Best Practices

•Enable etcd encryption at rest: Prevents reading secrets from etcd backups
•Use external secrets managers: Vault, AWS Secrets Manager, etc. for production
•Limit secret access via RBAC: Only pods that need secrets should access them
•Never log secrets: Ensure application code doesn't print credentials
•Rotate regularly: Implement rotation—secrets should have limited lifetimes
•Audit secret access: Enable API server audit logging to track who accesses secrets
•Use Sealed Secrets for GitOps: Safe to commit encrypted secrets to git

Image Security: Supply Chain Protection

Container images are the artifact you deploy. If an image is compromised—through malicious base images, vulnerable dependencies, or supply chain attacks—your cluster is compromised.

Image Security Controls:

Scan for vulnerabilities: Check images for known CVEs before deployment
Sign and verify images: Ensure images haven't been tampered with
Use minimal base images: Smaller images = smaller attack surface
Pin versions: Never use :latest in production
Private registries: Don't pull from public Docker Hub in production
Admission policies: Block deployment of non-compliant images

image-security.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
# ValidatingAdmissionPolicy (K8s 1.26+)
# Block images without digest or from untrusted registries
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: image-policy
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  # Require images from trusted registry
  - expression: |
      object.spec.containers.all(c, 
        c.image.startsWith('registry.example.com/') ||
        c.image.startsWith('gcr.io/my-project/')
      )
    message: "Images must be from trusted registries"
    
  # Require image digest (not just tag)
  - expression: |
      object.spec.containers.all(c,
        c.image.contains('@sha256:')
      )
    message: "Images must use digest, not tag"
 
---
# Kyverno policy (alternative to ValidatingAdmissionPolicy)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: verify-signature
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - imageReferences:
      - "registry.example.com/*"
      attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              <your cosign public key>
              -----END PUBLIC KEY-----
 
---
# Image pull secrets for private registries
apiVersion: v1
kind: Secret
metadata:
  name: registry-credentials
  namespace: production
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-docker-config>
 
---
# ServiceAccount with imagePullSecrets
apiVersion: v1
kind: ServiceAccount
metadata:
  name: production-sa
  namespace: production
imagePullSecrets:
- name: registry-credentials

Base Image Recommendations
Image Type	Example	Size	Use Case
Scratch	scratch	0 MB	Static binaries (Go)
Distroless	gcr.io/distroless/static	~2 MB	Static binaries with CA certs
Alpine	alpine:3.19	~5 MB	When shell/packages needed
Wolfi/Chainguard	cgr.dev/chainguard/static	~2 MB	Hardened, minimal, updated
Ubuntu/Debian	ubuntu:22.04	~77 MB	Legacy apps requiring glibc

Never Use :latest

Runtime Security: Detecting Threats in Real-Time

Runtime Security Tools:

Falco: Open-source runtime detection using syscall monitoring
Sysdig Secure: Commercial runtime security with response capabilities
Aqua Security: End-to-end container security platform
Tetragon: eBPF-based security observability from Cilium

What Runtime Security Detects:

Shell spawned inside container
Unexpected network connections
File access in sensitive directories
Process spawned from unexpected parent
Privilege escalation attempts
Cryptomining indicators

falco-rules.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
# Falco custom rules for Kubernetes security
# Deploy Falco as DaemonSet with host access
 
- rule: Shell Spawned in Container
  desc: Detect shell spawned inside a container (potential breakout attempt)
  condition: >
    spawned_process and
    container and
    shell_procs and
    not shell_allowed_container
  output: >
    Shell spawned in container 
    (user=%user.name container=%container.name shell=%proc.name 
     parent=%proc.pname cmdline=%proc.cmdline container_id=%container.id 
     image=%container.image.repository)
  priority: WARNING
  tags: [container, shell, mitre_execution]
 
- rule: Unexpected Outbound Connection
  desc: Detect container connecting to unexpected IP
  condition: >
    outbound and
    container and
    not allowed_outbound_ip and
    not k8s_known_connections
  output: >
    Unexpected outbound connection 
    (command=%proc.cmdline connection=%fd.name container=%container.name 
     image=%container.image.repository)
  priority: NOTICE
  
- rule: Sensitive File Read
  desc: Detect read of sensitive files (credentials, keys)
  condition: >
    open_read and
    container and
    (
      fd.name startswith /etc/shadow or
      fd.name startswith /etc/passwd or
      fd.name contains id_rsa or
      fd.name contains .kube/config or
      fd.name contains .aws/credentials
    )
  output: >
    Sensitive file read 
    (file=%fd.name command=%proc.cmdline container=%container.name)
  priority: WARNING
 
- rule: Kubernetes Secret Mounted and Read
  desc: Detect when mounted K8s secrets are read unexpectedly
  condition: >
    open_read and
    container and
    fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount and
    not k8s_trusted_processes
  output: >
    Kubernetes secret read 
    (file=%fd.name proc=%proc.name container=%container.name 
     pod=%k8s.pod.name namespace=%k8s.ns.name)
  priority: WARNING
 
- rule: Cryptomining Detected
  desc: Detect cryptomining activity
  condition: >
    spawned_process and
    container and
    (
      proc.name in (cryptominer_processes) or
      proc.cmdline contains "stratum+tcp://" or
      proc.cmdline contains "-o pool." or
      proc.cmdline contains "xmrig" or
      proc.cmdline contains "minerd"
    )
  output: >
    Cryptomining process detected 
    (command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: CRITICAL

Runtime Security Response

Audit Logging: Who Did What When

Kubernetes audit logs record all requests to the API server. They're essential for security investigations, compliance, and detecting anomalous behavior.

Audit Log Stages:

RequestReceived: Logged as soon as request is received (before processing)
ResponseStarted: For long-running requests (watch); response headers sent
ResponseComplete: Response body completed (most common to log)
Panic: Events generated if panic occurs

Audit Policy Levels:

None: Don't log
Metadata: Log request metadata (who, what, when) but not body
Request: Log metadata + request body
RequestResponse: Log everything including response body (verbose)

Considerations:

Audit logs can be high volume—filter appropriately
Sensitive data in request/response bodies—consider omitting
Storage and retention—audit logs must be durable

audit-policy.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# Kubernetes API Server Audit Policy
apiVersion: audit.k8s.io/v1
kind: Policy
 
# Don't log requests to these URLs
omitStages:
- "RequestReceived"
 
rules:
# Don't log read-only endpoints (reduces volume)
- level: None
  users: ["system:kube-probe"]
  
# Don't log health checks
- level: None
  nonResourceURLs:
  - "/healthz*"
  - "/livez*"
  - "/readyz*"
  - "/metrics"
 
# Log secret access at Request level (who accessed what)
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
    
# Log RBAC changes (security critical)
- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
 
# Log pod creation/deletion (detect cryptominers, etc)
- level: Request
  verbs: ["create", "delete", "patch", "update"]
  resources:
  - group: ""
    resources: ["pods"]
  - group: "apps"
    resources: ["deployments", "daemonsets", "statefulsets"]
 
# Log authentication failures
- level: Metadata
  users: ["system:anonymous"]
  verbs: ["*"]
 
# Log exec into pods (security sensitive)
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods/exec", "pods/attach", "pods/portforward"]
 
# Default: log metadata for everything else
- level: Metadata
  resources:
  - group: ""
  - group: "apps"
  - group: "batch"
  - group: "networking.k8s.io"

Audit Log Analysis

Summary: Security Best Practices Checklist

Kubernetes security requires a layered approach—no single control is sufficient. Let's consolidate the essential security controls for production clusters:

Security Checklist

•Enforce Pod Security Standards: Use 'restricted' profile for production namespaces
•Implement least-privilege RBAC: No wildcards, specific resources and verbs only
•Apply Network Policies: Default deny with explicit allow rules for required traffic
•Encrypt secrets at rest: Enable etcd encryption, use external secrets managers
•Scan images for vulnerabilities: Block deployment of images with critical CVEs
•Sign and verify images: Ensure supply chain integrity with cosign/notation
•Use minimal base images: Distroless or scratch where possible
•Deploy runtime security: Falco or similar for real-time threat detection
•Enable audit logging: Log security-sensitive API operations
•Regular security reviews: Audit RBAC, network policies, and container configs periodically

Module Complete:

Congratulations! You've completed the Kubernetes Operations module. You now have comprehensive knowledge of:

Resource Management: CPU, memory, QoS, quotas, and limit ranges
Auto-Scaling: HPA, VPA, Cluster Autoscaler, and Karpenter
Deployment Strategies: Rolling updates, rollbacks, blue-green, and canary
Observability: Metrics, logging, tracing, and alerting
Security: Pod Security, RBAC, Network Policies, and runtime protection

These operational skills distinguish production-ready Kubernetes environments from development clusters.

Module Complete

5 / 5