System Design (HLD)Secrets Management Tools

Secrets Management Tools

LevelAdvanced

Duration90 mins

TopicSecrets Management Tools

4 / 5

Kubernetes Secrets

Secrets in Container Orchestration

Kubernetes Secrets are the built-in mechanism for managing sensitive data in Kubernetes clusters. Understanding their capabilities, limitations, and the ecosystem of tools that extend them is essential for any engineer working with containerized applications.

Kubernetes provides native secrets as a first-class resource type, enabling workloads to access sensitive data without embedding credentials in container images or source code. However, native secrets come with significant security caveats that have spawned an ecosystem of tools designed to address their shortcomings.

What You Will Learn

By the end of this page, you will understand Kubernetes native secrets, their security model, encryption at rest configuration, external secrets operators for integrating with cloud secret managers, sealed secrets for GitOps workflows, and production security best practices for secrets in Kubernetes.

Native Kubernetes Secrets

Kubernetes Secrets are API objects that store sensitive data such as passwords, OAuth tokens, SSH keys, and TLS certificates. They separate sensitive configuration from container images and pod specifications.

Secret Types:

Kubernetes defines several built-in secret types, each with specific schema requirements:

Kubernetes Secret Types
Type	Purpose	Required Keys
Opaque	Arbitrary user-defined data	None (any keys)
kubernetes.io/service-account-token	Service account tokens	Auto-generated
kubernetes.io/dockerconfigjson	Docker registry credentials	.dockerconfigjson
kubernetes.io/basic-auth	Basic authentication	username, password
kubernetes.io/ssh-auth	SSH authentication	ssh-privatekey
kubernetes.io/tls	TLS certificates	tls.crt, tls.key
bootstrap.kubernetes.io/token	Bootstrap tokens	token-id, token-secret

Creating Kubernetes Secrets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Create a generic/opaque secret from literals
kubectl create secret generic db-credentials \
    --from-literal=username=admin \
    --from-literal=password='MyStr0ngP@ss!' \
    --namespace=production
 
# Create from files
kubectl create secret generic api-keys \
    --from-file=api-key=./api-key.txt \
    --from-file=api-secret=./api-secret.txt
 
# Create TLS secret from certificate files
kubectl create secret tls tls-cert \
    --cert=./tls.crt \
    --key=./tls.key \
    --namespace=production
 
# Create Docker registry secret
kubectl create secret docker-registry regcred \
    --docker-server=myregistry.azurecr.io \
    --docker-username=myuser \
    --docker-password=mypassword \
    --docker-email=user@example.com
 
# View secret (base64 encoded)
kubectl get secret db-credentials -o yaml
 
# Decode a specific value
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode
 
# List all secrets
kubectl get secrets -n production

Base64 is NOT Encryption

Base64 encoding is NOT a security measure—it's simply a text encoding format. Anyone with access to the Secret manifest can decode the values instantly. Native Kubernetes secrets are stored in etcd in plaintext (by default) and are only as secure as your cluster's etcd access controls.

Consuming Secrets in Pods

Pods can consume secrets in two ways: as environment variables or as volume mounts. Each approach has trade-offs in terms of security, update behavior, and application compatibility.

Secret Consumption Patterns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: app
      image: myapp:latest
      
      # Method 1: Individual keys as env vars
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
              optional: false  # Pod fails if secret doesn't exist
      
      # Method 2: All keys as env vars (prefix optional)
      envFrom:
        - secretRef:
            name: api-keys
          prefix: API_  # Results in API_api-key, API_api-secret
 
---
# Deployment with secret references
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: myapp:latest
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: connection-string
          # Environment variables are NOT updated when secret changes
          # Pods must be restarted to pick up new values

Environment Variables

•❌ Not updated when secret changes
•❌ Visible in kubectl describe pod
•❌ May appear in logs/crash dumps
•❌ Inherited by child processes
•✓ Simple application integration
•✓ Works with any application

Volume Mounts

•✓ Auto-updated when secret changes
•✓ Not in kubectl describe output
•✓ Configurable file permissions
•✓ Supports tmpfs (memory-backed)
•~ Requires file watching in app
•~ Application must support file-based config

Prefer Volume Mounts for Rotation

If you need secrets to update without pod restarts (e.g., credential rotation), use volume mounts. Implement file watching in your application using inotify or polling. Remember that updates may take up to kubelet sync period (default 1 minute) plus cache propagation delay.

Security Limitations of Native Secrets

Kubernetes native secrets have significant security limitations that are often misunderstood. Understanding these limitations is critical for designing secure systems.

Critical Limitation: etcd Storage

By default, Kubernetes stores secrets in etcd without encryption. This means:

Anyone with etcd access can read all secrets
etcd backups contain plaintext secrets
etcd disk storage contains plaintext secrets
Network traffic to etcd (if unencrypted) exposes secrets

Native Secrets Security Gaps

•No encryption at rest by default — Secrets stored as base64 in etcd; requires explicit configuration
•Broad API access — Any user with get secrets permission can read any secret in their namespace
•No audit trail for content access — Kubernetes audit logs show API calls, not secret content access
•No rotation support — No built-in mechanism for credential rotation or expiration
•GitOps challenges — Cannot commit encrypted secrets to git without additional tooling
•No secrets scanning — Kubernetes doesn't detect exposed secrets in logs or environment variables
•Namespace isolation only — Secrets can't be easily shared across namespaces securely

RBAC Risks:

Kubernetes RBAC for secrets operates at the resource level, not the secret content level:

Dangerous RBAC Patterns
RBAC Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# DANGEROUS: Grants access to ALL secrets in namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]  # Can read ANY secret
 
---
# BETTER: Restrict to specific secrets by name
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["myapp-secrets", "myapp-tls"]  # Only these secrets
    verbs: ["get"]
    # Note: 'list' cannot be restricted by resourceNames
 
---
# DANGEROUS: Pod creation = secret access
# Anyone who can create pods can mount any secret they can 'get'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-creator
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete"]
  # This user can create a pod that mounts secrets they have 'get' access to
  # The pod itself runs with the secret exposed
 
---
# Even with ValidatingWebhook, privilege escalation is complex
# Pod service accounts can often access more secrets than intended

The Pod Creation Loophole

Any user who can create pods AND get secrets can effectively read secret contents—even without list permission. They create a pod that mounts the secret and echos it to logs or exfiltrates it. RBAC alone cannot fully protect secrets from users with pod creation rights.

Encryption at Rest

Kubernetes supports encrypting Secret resources at rest in etcd. This is a critical security configuration for production clusters that should be enabled from day one.

Encryption Configuration:

The API server reads encryption configuration from a file specified by --encryption-provider-config. This file defines which resources to encrypt and which providers to use.

Encryption Configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# /etc/kubernetes/encryption/config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps  # Optional: encrypt configmaps too
    providers:
      # Provider order matters: first provider is used for encryption
      # All providers are tried for decryption (for rotation)
      
      # Option 1: AES-GCM (recommended for most cases)
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
              # Generate: head -c 32 /dev/urandom | base64
      
      # Option 2: Secretbox (modern, high-performance)
      - secretbox:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      
      # Option 3: KMS provider (recommended for production)
      - kms:
          apiVersion: v2
          name: aws-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
          timeout: 3s
      
      # Identity provider allows reading old unencrypted secrets
      # Remove after all secrets are re-encrypted
      - identity: {}
 
# API server flag:
# --encryption-provider-config=/etc/kubernetes/encryption/config.yaml

After Enabling Encryption:

Existing secrets remain unencrypted until re-created. Force re-encryption of all secrets:

Re-encrypt Existing Secrets
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
# Re-encrypt all secrets in all namespaces
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -
 
# Verify encryption is working
# Check etcd directly (requires etcdctl and certificates)
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
# Encrypted output should start with 'k8s:enc:aescbc:v1:key1:'
# Unencrypted output shows plaintext JSON

Key Rotation Strategy

Plan for encryption key rotation before enabling encryption. Add new keys at the beginning of the providers list (for encryption), keep old keys (for decryption), re-encrypt all secrets, then remove old keys. Loss of encryption keys means permanent loss of secrets—back up keys securely.

External Secrets Operator

The External Secrets Operator (ESO) bridges Kubernetes with external secret management systems like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager. It synchronizes secrets from external stores into native Kubernetes secrets.

Key Concepts:

SecretStore/ClusterSecretStore — Defines connection to external secret manager
ExternalSecret — Declares which secrets to sync and how to map them
Sync process — ESO periodically fetches from external store and updates K8s secrets

External Secrets Configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# Install ESO via Helm
# helm repo add external-secrets https://charts.external-secrets.io
# helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
 
---
# ClusterSecretStore for AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        # Option 1: IAM Roles for Service Accounts (IRSA) - recommended
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
        # Option 2: Static credentials (not recommended)
        # secretRef:
        #   accessKeyIDSecretRef:
        #     name: aws-credentials
        #     key: access-key-id
        #   secretAccessKeySecretRef:
        #     name: aws-credentials
        #     key: secret-access-key
 
---
# ExternalSecret that syncs from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h  # How often to sync
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  
  target:
    name: db-credentials  # Name of K8s secret to create
    creationPolicy: Owner  # Delete K8s secret when ExternalSecret is deleted
    template:
      type: Opaque
      data:
        # Template the secret data
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.example.com/production"
  
  data:
    # Map external secret keys to K8s secret keys
    - secretKey: username
      remoteRef:
        key: prod/myapp/database  # AWS Secrets Manager secret name
        property: username         # JSON key within the secret
    
    - secretKey: password
      remoteRef:
        key: prod/myapp/database
        property: password
        
  # Or fetch entire secret as JSON
  dataFrom:
    - extract:
        key: prod/myapp/api-keys

Refresh Interval Strategy

Set refreshInterval based on your rotation cadence. Too short increases API calls and costs; too long means delayed credential updates. For secrets with automatic rotation, set refresh slightly shorter than rotation period to ensure timely updates.

Sealed Secrets for GitOps

Sealed Secrets by Bitnami solves the GitOps secrets problem: how do you store encrypted secrets in Git that can only be decrypted by the cluster? Sealed Secrets uses asymmetric encryption—secrets are encrypted with a public key and can only be decrypted by the controller with the private key.

How It Works:

Cluster runs Sealed Secrets controller with a generated key pair
You encrypt secrets using the kubeseal CLI with the public key
Encrypted SealedSecret can be safely committed to Git
Controller decrypts SealedSecret and creates native Kubernetes Secret

Sealed Secrets Workflow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
 
# Install kubeseal CLI
# macOS: brew install kubeseal
# Linux: wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
 
# Fetch the public key (for offline sealing)
kubeseal --fetch-cert \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    > sealed-secrets-pub.pem
 
# Create a regular secret, then seal it
kubectl create secret generic db-credentials \
    --from-literal=username=admin \
    --from-literal=password='MyS3cret!' \
    --dry-run=client -o yaml | \
kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --format yaml > db-credentials-sealed.yaml
 
# The sealed secret can be committed to Git
cat db-credentials-sealed.yaml

Backup Your Sealing Key!

If you lose the controller's private key, ALL sealed secrets become permanently undecryptable. Back up the sealing key immediately after installation: kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-key-backup.yaml. Store this backup securely (not in Git!).

Production Security Best Practices

Securing secrets in Kubernetes requires a defense-in-depth approach combining multiple techniques. Here are battle-tested practices for production environments:

Essential Security Measures

•Enable encryption at rest — Configure etcd encryption from day one; use KMS provider in managed clusters
•Use external secrets manager — Store secrets of record in Vault/AWS SM/Azure KV; sync to K8s with External Secrets Operator
•Minimize RBAC scope — Use resourceNames restrictions; separate secret admin from pod creator roles
•Prefer volume mounts — Avoid environment variables for secrets; use file mounts with auto-update
•Enable audit logging — Configure Kubernetes audit policy to log secret access at RequestResponse level
•Implement network policies — Restrict which pods can communicate with external secrets endpoints
•Use Pod Security Standards — Prevent privileged containers that could access host filesystems
•Rotate secrets regularly — Implement rotation workflows; avoid long-lived credentials

Security Configuration Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Kubernetes audit policy for secrets
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log secret access at RequestResponse level (includes response body)
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]
    
  # Log secret mutations
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["create", "update", "patch", "delete"]
    
  # Don't log reads of configmaps (reduce noise)
  - level: None
    resources:
      - group: ""
        resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

The Ideal Architecture

Best practice: Store secrets in an external manager (Vault, AWS Secrets Manager), sync to Kubernetes via External Secrets Operator, enable encryption at rest in etcd, use IRSA/Workload Identity for operator authentication, and mount secrets as volumes with auto-rotation. This provides defense in depth with minimal secrets exposure.

Summary: Kubernetes Secrets Mastery

Kubernetes secrets provide essential but limited secrets management for container orchestration. Let's consolidate the key concepts:

Key Takeaways

•Native secrets are base64, not encrypted — Enable encryption at rest as a fundamental security requirement
•Consume via volumes when possible — Volume mounts auto-update and avoid environment variable security issues
•RBAC has fundamental gaps — Pod creators with secret access can exfiltrate secrets; defense requires additional controls
•External Secrets Operator — Bridges K8s with enterprise secret managers; synchronizes secrets with configurable refresh
•Sealed Secrets enable GitOps — Asymmetric encryption allows committing secrets to version control safely
•Defense in depth is essential — Combine encryption at rest, external stores, audit logging, RBAC, and network policies
•Prefer external managers for secrets of record — Use K8s secrets as a synchronized cache, not the source of truth

What's Next:

Having explored individual secrets management tools, the final page helps you choose the right solution for your organization. You'll learn decision frameworks, comparison matrices, and architectural patterns for selecting and combining these tools based on your specific requirements.

Page Complete

You now have a comprehensive understanding of Kubernetes secrets, their security limitations, and the ecosystem of tools that enhance their security. This knowledge enables you to design secure secrets architectures for containerized workloads.

4 / 5

Loading learning content...

System Design (HLD)Secrets Management Tools

Secrets Management Tools

LevelAdvanced

Duration90 mins

TopicSecrets Management Tools

4 / 5

Kubernetes Secrets

Secrets in Container Orchestration

What You Will Learn

Native Kubernetes Secrets

Secret Types:

Kubernetes defines several built-in secret types, each with specific schema requirements:

Kubernetes Secret Types
Type	Purpose	Required Keys
Opaque	Arbitrary user-defined data	None (any keys)
kubernetes.io/service-account-token	Service account tokens	Auto-generated
kubernetes.io/dockerconfigjson	Docker registry credentials	.dockerconfigjson
kubernetes.io/basic-auth	Basic authentication	username, password
kubernetes.io/ssh-auth	SSH authentication	ssh-privatekey
kubernetes.io/tls	TLS certificates	tls.crt, tls.key
bootstrap.kubernetes.io/token	Bootstrap tokens	token-id, token-secret

Creating Kubernetes Secrets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Create a generic/opaque secret from literals
kubectl create secret generic db-credentials \
    --from-literal=username=admin \
    --from-literal=password='MyStr0ngP@ss!' \
    --namespace=production
 
# Create from files
kubectl create secret generic api-keys \
    --from-file=api-key=./api-key.txt \
    --from-file=api-secret=./api-secret.txt
 
# Create TLS secret from certificate files
kubectl create secret tls tls-cert \
    --cert=./tls.crt \
    --key=./tls.key \
    --namespace=production
 
# Create Docker registry secret
kubectl create secret docker-registry regcred \
    --docker-server=myregistry.azurecr.io \
    --docker-username=myuser \
    --docker-password=mypassword \
    --docker-email=user@example.com
 
# View secret (base64 encoded)
kubectl get secret db-credentials -o yaml
 
# Decode a specific value
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode
 
# List all secrets
kubectl get secrets -n production

Base64 is NOT Encryption

Consuming Secrets in Pods

Pods can consume secrets in two ways: as environment variables or as volume mounts. Each approach has trade-offs in terms of security, update behavior, and application compatibility.

Secret Consumption Patterns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
    - name: app
      image: myapp:latest
      
      # Method 1: Individual keys as env vars
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
              optional: false  # Pod fails if secret doesn't exist
      
      # Method 2: All keys as env vars (prefix optional)
      envFrom:
        - secretRef:
            name: api-keys
          prefix: API_  # Results in API_api-key, API_api-secret
 
---
# Deployment with secret references
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: myapp:latest
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: connection-string
          # Environment variables are NOT updated when secret changes
          # Pods must be restarted to pick up new values

Environment Variables

•❌ Not updated when secret changes
•❌ Visible in kubectl describe pod
•❌ May appear in logs/crash dumps
•❌ Inherited by child processes
•✓ Simple application integration
•✓ Works with any application

Volume Mounts

•✓ Auto-updated when secret changes
•✓ Not in kubectl describe output
•✓ Configurable file permissions
•✓ Supports tmpfs (memory-backed)
•~ Requires file watching in app
•~ Application must support file-based config

Prefer Volume Mounts for Rotation

Security Limitations of Native Secrets

Kubernetes native secrets have significant security limitations that are often misunderstood. Understanding these limitations is critical for designing secure systems.

Critical Limitation: etcd Storage

By default, Kubernetes stores secrets in etcd without encryption. This means:

Anyone with etcd access can read all secrets
etcd backups contain plaintext secrets
etcd disk storage contains plaintext secrets
Network traffic to etcd (if unencrypted) exposes secrets

Native Secrets Security Gaps

•No encryption at rest by default — Secrets stored as base64 in etcd; requires explicit configuration
•Broad API access — Any user with get secrets permission can read any secret in their namespace
•No audit trail for content access — Kubernetes audit logs show API calls, not secret content access
•No rotation support — No built-in mechanism for credential rotation or expiration
•GitOps challenges — Cannot commit encrypted secrets to git without additional tooling
•No secrets scanning — Kubernetes doesn't detect exposed secrets in logs or environment variables
•Namespace isolation only — Secrets can't be easily shared across namespaces securely

RBAC Risks:

Kubernetes RBAC for secrets operates at the resource level, not the secret content level:

Dangerous RBAC Patterns
RBAC Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# DANGEROUS: Grants access to ALL secrets in namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]  # Can read ANY secret
 
---
# BETTER: Restrict to specific secrets by name
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["myapp-secrets", "myapp-tls"]  # Only these secrets
    verbs: ["get"]
    # Note: 'list' cannot be restricted by resourceNames
 
---
# DANGEROUS: Pod creation = secret access
# Anyone who can create pods can mount any secret they can 'get'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-creator
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete"]
  # This user can create a pod that mounts secrets they have 'get' access to
  # The pod itself runs with the secret exposed
 
---
# Even with ValidatingWebhook, privilege escalation is complex
# Pod service accounts can often access more secrets than intended

The Pod Creation Loophole

Encryption at Rest

Kubernetes supports encrypting Secret resources at rest in etcd. This is a critical security configuration for production clusters that should be enabled from day one.

Encryption Configuration:

The API server reads encryption configuration from a file specified by --encryption-provider-config. This file defines which resources to encrypt and which providers to use.

Encryption Configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# /etc/kubernetes/encryption/config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps  # Optional: encrypt configmaps too
    providers:
      # Provider order matters: first provider is used for encryption
      # All providers are tried for decryption (for rotation)
      
      # Option 1: AES-GCM (recommended for most cases)
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
              # Generate: head -c 32 /dev/urandom | base64
      
      # Option 2: Secretbox (modern, high-performance)
      - secretbox:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      
      # Option 3: KMS provider (recommended for production)
      - kms:
          apiVersion: v2
          name: aws-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
          timeout: 3s
      
      # Identity provider allows reading old unencrypted secrets
      # Remove after all secrets are re-encrypted
      - identity: {}
 
# API server flag:
# --encryption-provider-config=/etc/kubernetes/encryption/config.yaml

After Enabling Encryption:

Existing secrets remain unencrypted until re-created. Force re-encryption of all secrets:

Re-encrypt Existing Secrets
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
# Re-encrypt all secrets in all namespaces
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -
 
# Verify encryption is working
# Check etcd directly (requires etcdctl and certificates)
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
# Encrypted output should start with 'k8s:enc:aescbc:v1:key1:'
# Unencrypted output shows plaintext JSON

Key Rotation Strategy

External Secrets Operator

Key Concepts:

SecretStore/ClusterSecretStore — Defines connection to external secret manager
ExternalSecret — Declares which secrets to sync and how to map them
Sync process — ESO periodically fetches from external store and updates K8s secrets

External Secrets Configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# Install ESO via Helm
# helm repo add external-secrets https://charts.external-secrets.io
# helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
 
---
# ClusterSecretStore for AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        # Option 1: IAM Roles for Service Accounts (IRSA) - recommended
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
        # Option 2: Static credentials (not recommended)
        # secretRef:
        #   accessKeyIDSecretRef:
        #     name: aws-credentials
        #     key: access-key-id
        #   secretAccessKeySecretRef:
        #     name: aws-credentials
        #     key: secret-access-key
 
---
# ExternalSecret that syncs from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h  # How often to sync
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  
  target:
    name: db-credentials  # Name of K8s secret to create
    creationPolicy: Owner  # Delete K8s secret when ExternalSecret is deleted
    template:
      type: Opaque
      data:
        # Template the secret data
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.example.com/production"
  
  data:
    # Map external secret keys to K8s secret keys
    - secretKey: username
      remoteRef:
        key: prod/myapp/database  # AWS Secrets Manager secret name
        property: username         # JSON key within the secret
    
    - secretKey: password
      remoteRef:
        key: prod/myapp/database
        property: password
        
  # Or fetch entire secret as JSON
  dataFrom:
    - extract:
        key: prod/myapp/api-keys

Refresh Interval Strategy

Sealed Secrets for GitOps

How It Works:

Cluster runs Sealed Secrets controller with a generated key pair
You encrypt secrets using the kubeseal CLI with the public key
Encrypted SealedSecret can be safely committed to Git
Controller decrypts SealedSecret and creates native Kubernetes Secret

Sealed Secrets Workflow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
 
# Install kubeseal CLI
# macOS: brew install kubeseal
# Linux: wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
 
# Fetch the public key (for offline sealing)
kubeseal --fetch-cert \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    > sealed-secrets-pub.pem
 
# Create a regular secret, then seal it
kubectl create secret generic db-credentials \
    --from-literal=username=admin \
    --from-literal=password='MyS3cret!' \
    --dry-run=client -o yaml | \
kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --format yaml > db-credentials-sealed.yaml
 
# The sealed secret can be committed to Git
cat db-credentials-sealed.yaml

Backup Your Sealing Key!

Production Security Best Practices

Securing secrets in Kubernetes requires a defense-in-depth approach combining multiple techniques. Here are battle-tested practices for production environments:

Essential Security Measures

•Enable encryption at rest — Configure etcd encryption from day one; use KMS provider in managed clusters
•Use external secrets manager — Store secrets of record in Vault/AWS SM/Azure KV; sync to K8s with External Secrets Operator
•Minimize RBAC scope — Use resourceNames restrictions; separate secret admin from pod creator roles
•Prefer volume mounts — Avoid environment variables for secrets; use file mounts with auto-update
•Enable audit logging — Configure Kubernetes audit policy to log secret access at RequestResponse level
•Implement network policies — Restrict which pods can communicate with external secrets endpoints
•Use Pod Security Standards — Prevent privileged containers that could access host filesystems
•Rotate secrets regularly — Implement rotation workflows; avoid long-lived credentials

Security Configuration Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Kubernetes audit policy for secrets
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log secret access at RequestResponse level (includes response body)
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]
    
  # Log secret mutations
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["create", "update", "patch", "delete"]
    
  # Don't log reads of configmaps (reduce noise)
  - level: None
    resources:
      - group: ""
        resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

The Ideal Architecture

Summary: Kubernetes Secrets Mastery

Kubernetes secrets provide essential but limited secrets management for container orchestration. Let's consolidate the key concepts:

Key Takeaways

•Native secrets are base64, not encrypted — Enable encryption at rest as a fundamental security requirement
•Consume via volumes when possible — Volume mounts auto-update and avoid environment variable security issues
•RBAC has fundamental gaps — Pod creators with secret access can exfiltrate secrets; defense requires additional controls
•External Secrets Operator — Bridges K8s with enterprise secret managers; synchronizes secrets with configurable refresh
•Sealed Secrets enable GitOps — Asymmetric encryption allows committing secrets to version control safely
•Defense in depth is essential — Combine encryption at rest, external stores, audit logging, RBAC, and network policies
•Prefer external managers for secrets of record — Use K8s secrets as a synchronized cache, not the source of truth

What's Next:

Page Complete

4 / 5