Secrets Management - Learning Module

Loading content...

0/273

Secrets Lifecycle

From Birth to Death: The Journey of a Secret

In January 2021, a major cloud provider discovered that an API key issued 18 months earlier was still actively circulating in their system—despite the original requester having left the company, the project it was created for being cancelled, and no service actually using it. This zombie credential had been granting access to production databases for over a year after it should have been revoked.

This scenario is alarmingly common. Organizations focus intensely on secrets storage and access control but neglect the complete secrets lifecycle: the journey from creation through distribution, usage, rotation, and ultimately revocation or expiration. A secret is not just data to be stored—it is a living entity with a birth, a productive life, and an eventual death that must be managed at each stage.

What You Will Learn

By the end of this page, you will understand all phases of the secrets lifecycle, be able to design policies and automation for each phase, know how to implement proper rotation and revocation processes, and understand how lifecycle management prevents the accumulation of zombie credentials.

The Complete Secrets Lifecycle

Every secret passes through distinct phases, each requiring specific processes, controls, and automation. Understanding this lifecycle is essential for comprehensive secrets management.

The Seven Phases:

┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                              │
│     ┌───────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐     │
│     │  CREATION │────►│ STORAGE   │────►│ DISTRIB-  │────►│   USAGE   │     │
│     │           │     │           │     │   UTION   │     │           │     │
│     └───────────┘     └───────────┘     └───────────┘     └───────────┘     │
│                                                                 │            │
│                                                                 ▼            │
│     ┌───────────┐     ┌───────────┐     ┌───────────┐                       │
│     │  REVOKE/  │◄────│ ROTATION  │◄────│ MONITORING│                       │
│     │  EXPIRE   │     │           │     │           │                       │
│     └───────────┘     └───────────┘     └───────────┘                       │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

Phase Overview:

Phase	Primary Concern	Key Questions
Creation	Secure generation	How is the secret generated? Who authorizes creation?
Storage	Protection at rest	Where is the secret stored? How is it encrypted?
Distribution	Secure delivery	How do consumers receive it? Is transit encrypted?
Usage	Runtime protection	How is the secret used? Is it exposed in memory/logs?
Monitoring	Access visibility	Who accessed it? Were there anomalies?
Rotation	Lifecycle renewal	When is it replaced? How is transition handled?
Revocation	End of life	How is it invalidated? Is the revocation complete?

Lifecycle Debt Accumulates

Organizations that implement only creation, storage, and distribution—ignoring monitoring, rotation, and revocation—accumulate 'secrets lifecycle debt.' This debt manifests as stale credentials, unknown access patterns, and inability to respond effectively to breaches. The cost of remediating lifecycle debt is always higher than implementing proper lifecycle management from the start.

Phase 1: Secret Creation

Secret creation is the first and often most overlooked phase. How a secret is generated significantly impacts its security throughout its lifetime. Weak creation practices undermine all subsequent protections.

Secure Creation Principles

•Cryptographically Secure Generation — Secrets must be generated using cryptographically secure random number generators (CSPRNGs). Never use Math.random(), timestamps, or predictable patterns. Use /dev/urandom, crypto.getRandomValues(), or HSM-based generation.
•Sufficient Entropy — Secrets must have enough randomness to resist brute-force attacks. API keys should be at least 32 bytes (256 bits). Passwords should have at least 128 bits of entropy. Cryptographic keys should meet algorithm requirements (256 bits for AES-256).
•Authorization Required — Secret creation should require explicit authorization. Who can create a new database password? What approval is needed for a new AWS access key? Creation without authorization leads to secret sprawl.
•Metadata Captured — At creation time, capture: creator identity, purpose, intended consumers, expiration date, rotation schedule. This metadata enables lifecycle management decisions later.
•Immediate Classification — Classify the secret at creation. Its sensitivity level determines all subsequent handling: storage location, access controls, rotation frequency, audit requirements.

Automated vs. Manual Creation:

Secrets can be created manually (an administrator generates an API key) or automatically (a vault generates database credentials on-demand). Each approach has implications:

Aspect	Manual Creation	Automated Creation
Generation quality	Risk of weak secrets	Consistently strong
Authorization	Typically enforced	Policy-driven
Metadata capture	Often incomplete	Automatically complete
Time to provision	Minutes to hours	Seconds
Audit trail	Depends on process	Automatic
Human error	Significant risk	Minimal
Scale	Doesn't scale	Scales infinitely

Dynamic Secrets: The Gold Standard

The most secure creation pattern is dynamic secrets: secrets that are generated on-demand for each requester and have short lifespans. Instead of creating a database password that lasts a year, the secrets vault generates a unique credential for each application instance that expires in hours.

┌─────────────────────────────────────────────────────────────────────────────┐
│                          DYNAMIC SECRET CREATION                             │
│                                                                             │
│    Application          Vault               Database                        │
│         │                 │                     │                           │
│         │  Request cred   │                     │                           │
│         │────────────────►│                     │                           │
│         │                 │   Create temp user  │                           │
│         │                 │────────────────────►│                           │
│         │                 │◄────────────────────│                           │
│         │  Return cred    │                     │                           │
│         │◄────────────────│                     │                           │
│         │                 │                     │                           │
│         │   (use cred for 1 hour)               │                           │
│         │                 │                     │                           │
│         │                 │   Revoke temp user  │                           │
│         │                 │────────────────────►│ (automatic after TTL)     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Dynamic secrets dramatically reduce attack surface: even if a credential is compromised, it's only valid for a short window.

Phase 2: Secret Storage

Once created, secrets must be stored securely. Storage is the phase most organizations focus on, but it's only effective when combined with the other lifecycle phases.

Secure Storage Requirements

•Encryption at Rest — All secrets must be encrypted using strong algorithms (AES-256-GCM minimum). The encryption keys must themselves be protected, ideally by Hardware Security Modules (HSMs).
•Access Control Separation — The system storing secrets should have different access than the systems using secrets. A database backup admin should not be able to read application secrets.
•Physical/Logical Isolation — Secrets storage should be isolated from application data storage. If an attacker compromises your application database, they shouldn't automatically access all secrets.
•Redundancy with Consistency — Secrets must be available when needed (high availability) but must be consistent across replicas. A stale secret replica can cause both security issues (using old credential) and operational issues (authentication failures).
•Audit Everything — Every read, write, delete, and administrative operation must be logged with full context: who, what, when, from where, and why (if available).

Storage Architecture Patterns:

1. Centralized Vault

A dedicated secrets management system (HashiCorp Vault, AWS Secrets Manager, etc.) serves as the single source of truth for all secrets.

Advantages: Consistent policies, comprehensive audit, specialized security features Challenges: Single point of failure, latency for secret access, capacity planning

2. Envelope Encryption

Secrets are encrypted with a Data Encryption Key (DEK), and the DEK is encrypted with a Key Encryption Key (KEK) stored in a key management system.

┌─────────────────────────────────────────────────────────────────────────────┐
│                          ENVELOPE ENCRYPTION                                 │
│                                                                             │
│    ┌─────────────────┐                                                      │
│    │     Secret      │                                                      │
│    │   (plaintext)   │                                                      │
│    └────────┬────────┘                                                      │
│             │ encrypted with                                                │
│             ▼                                                               │
│    ┌─────────────────┐                                                      │
│    │       DEK       │  (Data Encryption Key)                               │
│    │   (random key)  │                                                      │
│    └────────┬────────┘                                                      │
│             │ encrypted with                                                │
│             ▼                                                               │
│    ┌─────────────────┐                                                      │
│    │       KEK       │  (Key Encryption Key) — stored in HSM/KMS           │
│    │   (master key)  │                                                      │
│    └─────────────────┘                                                      │
│                                                                             │
│    Decryption reverses the process: KEK decrypts DEK, DEK decrypts secret  │
└─────────────────────────────────────────────────────────────────────────────┘

Advantages: Limits HSM/KMS load, enables secret rotation without re-encrypting everything, defense in depth Challenges: More complex implementation, multiple keys to manage

3. Distributed Storage with Central Policy

Secrets are stored close to their consumers (e.g., Kubernetes Secrets, cloud-native secret stores), but policies and audit are managed centrally.

Advantages: Low latency access, works with cloud-native patterns Challenges: Consistency across stores, ensuring encryption is configured everywhere, audit correlation

Phase 3: Secret Distribution

Distribution is how secrets get from secure storage to consuming applications. This phase presents unique challenges: secrets must traverse networks, be injected into applications, and be available at runtime without being exposed.

Insecure Distribution Patterns

•Secrets in configuration files checked into Git
•Secrets passed as command-line arguments
•Secrets shared via email, Slack, or wikis
•Secrets baked into container images
•Secrets in unencrypted CI/CD variables
•Secrets in plaintext environment variables visible to all processes

Secure Distribution Patterns

•Secrets fetched from vault at application startup
•Secrets injected via sidecar proxy (Vault Agent)
•Secrets mounted as in-memory volumes
•Secrets accessed via authenticated API calls
•Secrets delivered via encrypted channels only (mTLS)
•Secrets available only to the specific process that needs them

Distribution Mechanisms Deep Dive:

1. Pull Model (Application Fetches)

The application authenticates to the secrets vault and retrieves secrets it needs. This is the most common pattern.

Application                Vault
    │                        │
    │  Authenticate (JWT,    │
    │  Workload Identity)    │
    │───────────────────────►│
    │                        │
    │  Return auth token     │
    │◄───────────────────────│
    │                        │
    │  Request secrets       │
    │───────────────────────►│
    │                        │
    │  Return secrets        │
    │  (over mTLS)           │
    │◄───────────────────────│
    │                        │

Considerations: Application needs vault client library, must handle token renewal, vault must be highly available.

2. Push Model (Sidecar Injection)

A sidecar container or agent fetches secrets and writes them to a shared location (memory filesystem, Unix socket) that the application reads.

┌─────────────────────────────────────────────────────────────────────────────┐
│                              POD                                            │
│                                                                             │
│  ┌─────────────────┐        ┌─────────────────┐                            │
│  │   Application   │        │   Vault Agent   │                            │
│  │   Container     │        │   Sidecar       │                            │
│  └────────┬────────┘        └────────┬────────┘                            │
│           │                          │                                      │
│           │  reads from              │  writes to                           │
│           ▼                          ▼                                      │
│  ┌─────────────────────────────────────────────┐                           │
│  │          Shared Memory Volume               │                           │
│  │          /vault/secrets/                    │                           │
│  └─────────────────────────────────────────────┘                           │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Considerations: Application doesn't need vault awareness, secrets appear as files, but adds pod complexity.

3. Just-in-Time Injection (Kubernetes Mutating Webhook)

Secrets are injected into pods at creation time by a mutating admission webhook that modifies pod specifications.

Considerations: Transparent to application, works with existing deployments, but secrets are injected once at pod start.

Distribution Encryption:

All distribution must use encrypted channels:

TLS 1.2+ for REST API calls to vaults
mTLS (mutual TLS) for service-to-service communication
Encrypted volumes for at-rest storage during distribution
Memory-only storage (tmpfs) for runtime secrets

Phase 4: Secret Usage

Usage is where secrets do their job: authenticating to databases, signing tokens, encrypting data. This phase receives less attention than it deserves—secrets that are perfectly protected in storage can be exposed during use.

Common Usage-Phase Exposure Vectors

•Logging — Secrets included in connection strings, error messages, or debug output. Example: ERROR: Failed to connect to postgres://admin:password123@db:5432/app
•Stack Traces — Secrets passed as function arguments appearing in exception stack traces sent to error tracking services.
•Memory Exposure — Secrets remaining in memory long after use, vulnerable to memory-scraping attacks or appearing in core dumps.
•Process Environment — Secrets accessible to debugging tools, profilers, or other processes on the same host.
•Temporary Files — Secrets written to disk temporarily (e.g., for command-line tools) and not securely deleted.
•Clipboard/History — Secrets copied to clipboard during debugging or appearing in shell history files.
•Third-Party Libraries — Libraries that log request/response bodies, including authentication headers or API keys.

Secure Usage Practices:

1. Minimize Secret Lifetime in Memory

# BAD: Secret persists in memory indefinitely
class DatabaseConnection:
    def __init__(self, password):
        self.password = password  # Secret lives forever

# BETTER: Fetch secret only when needed, don't store
class DatabaseConnection:
    def __init__(self, vault_client, secret_path):
        self.vault_client = vault_client
        self.secret_path = secret_path
    
    def connect(self):
        password = self.vault_client.read(self.secret_path)
        conn = create_connection(password)
        del password  # Explicit cleanup (though Python GC may delay)
        return conn

2. Never Log Secrets

Implement log sanitization that redacts known secret patterns:

import re

SECRET_PATTERNS = [
    r'password[":\']\s*["\']?([^"\s]+)["\']?',
    r'api[_-]?key[":\']\s*["\']?([^"\s]+)["\']?',
    r'(Bearer\s+[A-Za-z0-9-._~+/]+=*)',
    r'(sk_live_[A-Za-z0-9]+)',  # Stripe secret key
]

def sanitize_log_message(message):
    for pattern in SECRET_PATTERNS:
        message = re.sub(pattern, '[REDACTED]', message, flags=re.IGNORECASE)
    return message

3. Use Secrets for Their Minimum Required Scope

Instead of a database superuser password used everywhere:

Create a read-only user for queries
Create a write user for updates
Create migration user for schema changes only
Use the least-privileged credential for each operation

4. Implement Secret Caching Carefully

Caching secrets reduces vault calls but increases exposure window:

Cache for the minimum viable time
Never cache to disk
Invalidate cache on rotation signals
Use encrypted in-memory storage

The Principle of Ephemeral Secrets

The safest secret is one that ceases to exist immediately after use. Design systems where secrets are held only for the microseconds needed to establish connections or perform cryptographic operations. The less time a secret exists in memory, the smaller your attack surface.

Phase 5: Secret Access Monitoring

Monitoring secret access provides visibility into who is using secrets, when, and how. This visibility is essential for detecting compromises, ensuring compliance, and planning rotation.

What to Monitor:

Event Type	What to Capture	Why It Matters
Secret read	Who, what secret, when, from where	Detects unauthorized access, usage patterns
Secret write	Who, what secret, old→new (hashed), when	Change management, debugging
Access denied	Who, what secret attempted, why denied	Attack detection, policy tuning
Secret creation	Who, what secret, purpose, expiration	Lifecycle tracking, sprawl prevention
Secret deletion	Who, what secret, when	Audit trail, compliance
Policy changes	Who, what policy, old→new	Privilege escalation detection
Auth failures	Who, how many, from where	Brute force detection

Alerting Patterns:

Not all access is equal. Configure alerts based on risk:

Critical (Immediate Response):

Critical secret accessed from unexpected source (new IP, new region)
Multiple authentication failures followed by success
Access to secrets by service that shouldn't need them
Admin operations outside approved windows

Warning (Investigate Within Hours):

Unusual access patterns (accessing more secrets than normal)
Access from new workload identities
Failed access attempts (may indicate misconfiguration)

Informational (Review During Audits):

Normal access patterns
Successful rotations
Standard lifecycle events

Building Access Baselines:

To detect anomalies, you need baselines:

┌─────────────────────────────────────────────────────────────────────────────┐
│                       ACCESS BASELINE EXAMPLE                                │
│                                                                             │
│  Payment Service typical access:                                            │
│  - Reads 'db/payment-service/creds' 2-4 times per hour (pod restarts)      │
│  - Reads 'stripe/api-key' once at startup                                  │
│  - Never reads 'db/user-service/creds'                                      │
│  - Access from IPs in us-east-1 and us-west-2 only                         │
│  - Access times: 24/7 (always running)                                      │
│                                                                             │
│  Anomaly indicators:                                                        │
│  - 50+ reads to any secret in an hour  → Alert                             │
│  - Access to user-service credentials  → Critical alert                    │
│  - Access from eu-central-1            → Critical alert                    │
│  - Access from non-registered IP       → Critical alert                    │
└─────────────────────────────────────────────────────────────────────────────┘

Monitoring Enables Response

The value of monitoring is directly proportional to your ability to act on it. If an alert fires at 2 AM and nobody responds until 9 AM, your detection capability provided seven hours of warning that went unused. Ensure monitoring is connected to on-call, automated response, or both.

Phase 6: Secret Rotation

Rotation is the periodic replacement of secrets with new values. It's one of the most impactful lifecycle practices—and one of the hardest to implement well. Rotation limits the blast radius of compromise: a leaked secret is only useful until the next rotation.

Rotation Strategy Considerations

•Rotation Frequency — Balance security (shorter is better) vs. operational cost (each rotation has risk). Critical secrets: rotate frequently. Less critical: less often. Regulatory requirements may mandate minimums.
•Rotation Method — Manual rotation doesn't scale and is error-prone. Automated rotation is essential for any secrets beyond a handful.
•Consumer Coordination — When a secret changes, consumers must receive the new value. This is the hard part of rotation—coordinating the transition.
•Rollback Capability — Rotation can fail. Previous secrets must remain valid until rotation is confirmed complete, or systems fail during the transition.
•Audit Trail — All rotations must be logged: when, by whom (or what automation), and confirmation of successful propagation.

Rotation Patterns:

1. Dual-Active (Overlapping Validity)

During rotation, both old and new secrets are valid for a transition period. Consumers gradually switch to the new secret.

            Time ──────────────────────────────────────────────►
            
Old Secret: ████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░
            (valid)                      (expired)
            
New Secret: ░░░░░░░░░░░░░░░░░░░░░░░████████████████████████████
                                    (valid)
            
            ◄──── Overlap Period ────►
            (both valid, consumers transition)

Best for: API keys, service tokens, situations where you can't synchronize all consumers simultaneously.

2. Immediate Replacement

Old secret is invalidated immediately when new secret is created. All consumers must have the new secret before rotation.

Best for: Dynamic secrets with short lifespans, situations where you control all consumers.

3. Blue-Green Rotation

Two secret "slots" exist; consumers know to check both. Rotation alternates between slots.

Slot A: [password-123]  ← current
Slot B: [password-456]  (will be current after rotation)

Consumer logic:
  try authenticate(SlotA)
  if failed: try authenticate(SlotB)

Best for: Situations where you can modify consumer logic but can't perfectly coordinate timing.

Automated Rotation Architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     AUTOMATED ROTATION FLOW                                  │
│                                                                             │
│  ┌─────────┐     ┌─────────┐     ┌─────────┐     ┌─────────┐               │
│  │Schedule │     │ Generate│     │  Store  │     │Propagate│               │
│  │ Trigger │────►│New Secret────►│in Vault │────►│    to   │               │
│  │         │     │         │     │         │     │Consumers│               │
│  └─────────┘     └─────────┘     └─────────┘     └────┬────┘               │
│                                                       │                     │
│                  ┌───────────────────────────────────┘                     │
│                  ▼                                                          │
│            ┌──────────┐     ┌──────────┐     ┌──────────┐                  │
│            │  Verify  │     │ Revoke   │     │  Report  │                  │
│            │Consumers │────►│   Old    │────►│ Success  │                  │
│            │  Healthy │     │  Secret  │     │          │                  │
│            └──────────┘     └──────────┘     └──────────┘                  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Rotation Failures are Production Incidents

Failed rotations can cause outages. If the new secret is invalid, or consumers don't receive it, authentication fails. Treat rotation as a production change: test in non-production first, have rollback plans, monitor during and after rotation, and schedule rotations during maintenance windows if possible.

Phase 7: Revocation and Expiration

Every secret must eventually cease to exist. Whether through scheduled expiration, explicit revocation after a breach, or cleanup when a project ends, proper termination prevents zombies credentials that persist indefinitely.

Expiration vs. Revocation:

Aspect	Expiration	Revocation
Trigger	Time-based (automatic)	Event-based (manual or automated)
Planned	Yes	Often emergency/unplanned
Grace period	Usually none	May have brief overlap
Consumer preparation	Consumers know when	Consumers surprised
Use case	Normal lifecycle	Breach response, employee offboarding

Revocation Readiness Checklist

•Inventory Complete — Can you identify all secrets in your system? You can't revoke what you don't know exists.
•Revocation Procedure Documented — For each secret type, is there a documented procedure to revoke it? Including the system it authenticates to?
•Consumer Mapping Known — Do you know which services use which secrets? A revocation that breaks production wasn't properly planned.
•Revocation Tested — Have you practiced revoking secrets? Revocation under breach pressure is the wrong time to discover your procedure doesn't work.
•Timing Acceptable — How quickly can you revoke a secret from decision to completion? For critical secrets, this should be minutes, not hours.
•Verification Possible — After revocation, can you confirm the secret no longer works? Test it against the target system.

Emergency Revocation Playbook:

When a breach is suspected, every minute counts. Having a prepared playbook enables rapid response:

┌─────────────────────────────────────────────────────────────────────────────┐
│                   EMERGENCY SECRET REVOCATION PLAYBOOK                       │
│                                                                             │
│  MINUTE 0-5: TRIAGE                                                         │
│  □ Identify which secret(s) are compromised                                 │
│  □ Determine blast radius (what can the secret access?)                     │
│  □ Notify incident commander                                                │
│                                                                             │
│  MINUTE 5-10: REVOKE                                                        │
│  □ Revoke secret at the target system (database, API, etc.)                │
│  □ Revoke secret in the vault                                              │
│  □ Block any known malicious sources at network level                       │
│                                                                             │
│  MINUTE 10-20: REPLACE                                                      │
│  □ Generate new secret                                                       │
│  □ Deploy to legitimate consumers                                           │
│  □ Verify legitimate consumers authenticate successfully                    │
│                                                                             │
│  MINUTE 20+: INVESTIGATE                                                    │
│  □ Review access logs for unauthorized usage                                │
│  □ Determine scope of potential data access                                 │
│  □ Initiate full incident response if data accessed                         │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Preventing Zombie Credentials:

Zombie credentials are secrets that should be revoked but weren't:

Employee leaves → their API keys should be revoked
Project ends → its service accounts should be deleted
Service decommissioned → its credentials should be revoked

Prevent zombies with:

Maximum lifespans: All secrets expire after N days, forcing renewal or removal
Owner assignment: Every secret has an owner responsible for lifecycle
Periodic audits: Regular reviews of all active secrets, questioning whether each is still needed
Offboarding checklists: HR and IT processes that trigger credential revocation

Summary: The Complete Secrets Lifecycle

The secrets lifecycle encompasses seven distinct phases, each requiring specific practices and automation. Complete lifecycle management is what separates mature secrets management from mere storage.

Key Takeaways

•Seven phases require attention — Creation, storage, distribution, usage, monitoring, rotation, and revocation. Neglecting any phase creates security gaps.
•Dynamic secrets reduce attack surface — Secrets generated on-demand with short lifespans limit the window of opportunity for attackers.
•Distribution is where secrets are most vulnerable — Transit between storage and application is a common exposure point. Use encrypted channels exclusively.
•Usage-phase protection is often overlooked — Logging, memory exposure, and temporary files can leak secrets that were perfectly protected in storage.
•Monitoring enables detection and response — Without visibility into access patterns, you can't detect compromise or plan effective rotation.
•Rotation limits blast radius — Regular rotation means compromised secrets have limited utility. Automation makes rotation sustainable.
•Revocation must be fast and tested — In a breach, the ability to revoke within minutes is often the difference between incident and catastrophe.

What's Next:

With a complete understanding of the secrets lifecycle, the next page examines Common Mistakes in secrets management. Despite all the best practices we've discussed, organizations repeatedly fall into predictable traps. Understanding these mistakes—and why they're so common—helps you avoid them in your own systems.

Page Complete

You now understand the complete secrets lifecycle from creation through revocation, can design policies and automation for each phase, know best practices for secure distribution, usage, and rotation, and can plan for emergency revocation when breaches occur.