In October 2023, security researchers reported finding over 4,500 valid AWS credentials in a single day's worth of public GitHub commits. Each credential represented an organization that made at least one of the common mistakes we'll examine in this page. Some were Fortune 500 companies. Some were government agencies. Some were security companies themselves.

These mistakes aren't made by inexperienced or careless engineers. They're made by talented professionals working under real-world constraints: tight deadlines, legacy systems, unclear requirements, and organizational gaps. Understanding why these mistakes are so common—and designing systems that make them less likely—is the mark of mature security engineering.

This is the most common and most thoroughly studied secrets management failure. Despite years of education and tooling, organizations continue to commit secrets to Git repositories at alarming rates.

The Permanence Problem:

When a secret is committed to Git, it exists in repository history forever. Simply deleting the file in a subsequent commit doesn't help—the secret remains in git log, accessible to anyone who clones the repository or has access to backups.

True Remediation Requires:

┌─────────────────────────────────────────────────────────────────────────────┐
│                       SECRET REMEDIATION CHECKLIST                           │
│                                                                             │
│  □ 1. ROTATE THE SECRET IMMEDIATELY                                        │
│       - Generate new credential                                              │
│       - Deploy to all legitimate consumers                                   │
│       - Revoke the compromised credential                                    │
│                                                                             │
│  □ 2. CONSIDER REWRITING GIT HISTORY                                        │
│       - Use git filter-branch or BFG Repo Cleaner                           │
│       - Force-push to all remotes                                           │
│       - All developers must re-clone                                         │
│       - Cannot help if repo was ever public or forked                        │
│                                                                             │
│  □ 3. AUDIT ACCESS LOGS                                                     │
│       - Check if secret was used maliciously                                 │
│       - Review who had repository access                                     │
│       - Check clone/pull history if available                                │
│                                                                             │
│  □ 4. ADD PREVENTION                                                        │
│       - Install pre-commit hooks                                             │
│       - Enable CI/CD scanning                                                │
│       - Conduct team education                                               │
└─────────────────────────────────────────────────────────────────────────────┘

Common Patterns That Get Committed:

When creating credentials, the path of least resistance is to grant broad permissions. "Just give it admin access and we'll lock it down later" becomes permanent policy. This dramatically increases the blast radius when (not if) credentials are compromised.

The Principle of Least Privilege:

Every credential should have the minimum permissions required for its specific purpose. This isn't just security advice—it's operational hygiene that makes systems more understandable:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    PERMISSION SCOPING EXAMPLES                               │
│                                                                             │
│  BEFORE: Single AWS key with AdministratorAccess                            │
│                                                                             │
│  AFTER: Multiple scoped credentials                                         │
│  ├── deploy-frontend: S3 write (ui-bucket), CloudFront invalidate          │
│  ├── deploy-backend: ECS deploy (backend-cluster), ECR push                │
│  ├── monitoring: CloudWatch read, X-Ray read                                │
│  ├── backup: S3 read (backup-bucket), RDS snapshot                         │
│  └── alerting: SNS publish (alert-topic)                                    │
│                                                                             │
│  BENEFIT: Compromised backup credentials can't:                             │
│  - Deploy malicious code                                                    │
│  - Access application data                                                  │
│  - Modify infrastructure                                                    │
│  - Exfiltrate to external S3 buckets                                        │
└─────────────────────────────────────────────────────────────────────────────┘

Implementation Approach:

1. Start denied, add specific permissions — Default to zero access, add only what's needed
2. Create credentials per use case — Don't share credentials across different purposes
3. Review and prune regularly — Permissions accumulate over time; periodic reviews remove unused access
4. Use policy conditions — Restrict by IP, time of day, MFA requirement, not just actions and resources

Many organizations implement excellent secret storage but never rotate credentials. Database passwords from 2018 remain in use. API keys issued for a proof-of-concept become permanent fixtures. The longer a secret exists, the more likely it has been exposed somewhere.

Recommended Rotation Frequencies:

Dynamic Secrets Eliminate the Problem:

The ultimate rotation strategy is to use secrets that rotate themselves by design:

┌─────────────────────────────────────────────────────────────────────────────┐
│                      DYNAMIC VS STATIC SECRETS                               │
│                                                                             │
│  STATIC SECRET:                                                             │
│  - Created once, used for months/years                                       │
│  - Same password for all application instances                               │
│  - Rotation is a scheduled operation                                         │
│  - Blast radius: potentially years of access if compromised                  │
│                                                                             │
│  DYNAMIC SECRET:                                                             │
│  - Generated on-demand per application instance                              │
│  - Each instance gets unique credential                                       │
│  - Expires automatically (typically 1-24 hours)                              │
│  - Blast radius: hours of access if compromised                              │
│                                                                             │
│  Example: Vault's database secrets engine                                    │
│  - App requests database creds from Vault                                    │
│  - Vault creates temporary database user                                     │
│  - User valid for configured TTL (e.g., 1 hour)                             │
│  - Vault automatically revokes after TTL                                     │
│  - No rotation needed—secrets are naturally ephemeral                        │
└─────────────────────────────────────────────────────────────────────────────┘

Application logs are essential for debugging and monitoring, but they're also a common secret exposure vector. Secrets logged to stdout, error tracking services, or centralized logging systems persist in systems that weren't designed for secret storage.

Prevention Strategies:

1. Implement Log Redaction:

import re
from typing import Dict, List

class SecretRedactor:
    PATTERNS: List[Dict] = [
        {'name': 'password', 'regex': r'password["\']?\s*[:=]\s*["\']?([^"\'\ 
,}]+)'},
        {'name': 'bearer', 'regex': r'Bearer\s+([A-Za-z0-9-._~+/]+=*)'},
        {'name': 'aws_secret', 'regex': r'aws_secret_access_key["\']?\s*[:=]\s*["\']?([A-Za-z0-9/+=]{40})'},
        {'name': 'api_key', 'regex': r'api[_-]?key["\']?\s*[:=]\s*["\']?([A-Za-z0-9-_]{20,})'},
        {'name': 'stripe', 'regex': r'(sk_live_[A-Za-z0-9]{24})'},
        {'name': 'connection_string', 'regex': r'://[^:]+:([^@]+)@'},
    ]
    
    @classmethod
    def redact(cls, message: str) -> str:
        for pattern in cls.PATTERNS:
            message = re.sub(
                pattern['regex'], 
                f"{pattern['name'].upper()}=[REDACTED]", 
                message, 
                flags=re.IGNORECASE
            )
        return message

# Use in logging handler
class RedactingHandler(logging.StreamHandler):
    def emit(self, record):
        record.msg = SecretRedactor.redact(str(record.msg))
        super().emit(record)

2. Never Log Secret-Containing Objects:

# BAD: Logging entire config object that may contain secrets
logger.info(f"Configuration loaded: {config}")

# GOOD: Log only safe fields
logger.info(f"Configuration loaded: host={config.host}, port={config.port}")

# BETTER: Create a safe representation
class Config:
    def __str__(self):
        return f"Config(host={self.host}, port={self.port}, password=***)"

3. Configure Libraries to Not Log Secrets:

Many HTTP libraries log requests by default. Configure them:

# Disable request logging in popular libraries
import requests
import logging

logging.getLogger("urllib3").setLevel(logging.WARNING)
logging.getLogger("requests").setLevel(logging.WARNING)

# For libraries that do log, set up redaction at the logging infrastructure level

Sharing the same credential across multiple services seems efficient but dramatically increases blast radius and makes access control impossible. When multiple services share a database password, you can't revoke one service's access without affecting all others.

The Shared Credential Problem:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ANTI-PATTERN: SHARED CREDENTIALS                          │
│                                                                             │
│                    ┌─────────────────────────┐                              │
│                    │   Database Password:    │                              │
│                    │   "super-secret-123"    │                              │
│                    └───────────┬─────────────┘                              │
│                                │                                             │
│          ┌─────────────────────┼─────────────────────┐                      │
│          │                     │                     │                       │
│          ▼                     ▼                     ▼                       │
│    ┌──────────┐          ┌──────────┐          ┌──────────┐                 │
│    │ Service  │          │ Service  │          │ Service  │                 │
│    │    A     │          │    B     │          │    C     │                 │
│    └──────────┘          └──────────┘          └──────────┘                 │
│                                                                             │
│  PROBLEMS:                                                                   │
│  • Can't revoke just Service B's access                                     │
│  • Can't audit which service made a query                                   │
│  • Rotating requires coordinating all services                               │
│  • Compromise of any service compromises all data                           │
│  • Can't apply per-service database permissions                             │
└─────────────────────────────────────────────────────────────────────────────┘

The Correct Pattern:

┌─────────────────────────────────────────────────────────────────────────────┐
│                   PATTERN: UNIQUE CREDENTIALS PER SERVICE                    │
│                                                                             │
│    ┌──────────┐          ┌──────────┐          ┌──────────┐                 │
│    │ Service  │          │ Service  │          │ Service  │                 │
│    │    A     │          │    B     │          │    C     │                 │
│    └────┬─────┘          └────┬─────┘          └────┬─────┘                 │
│         │                     │                     │                        │
│         ▼                     ▼                     ▼                        │
│  ┌─────────────┐       ┌─────────────┐       ┌─────────────┐                │
│  │ db_user_a   │       │ db_user_b   │       │ db_user_c   │                │
│  │ Tables: x,y │       │ Tables: y,z │       │ Tables: z   │                │
│  │ Read-only   │       │ Read-write  │       │ Delete OK   │                │
│  └─────────────┘       └─────────────┘       └─────────────┘                │
│                                                                             │
│  BENEFITS:                                                                  │
│  • Revoke Service B independently                                           │
│  • Audit knows which service made each query                                 │
│  • Rotate services independently                                             │
│  • Compromise of Service A can't modify data                                │
│  • Each service has minimum necessary permissions                           │
└─────────────────────────────────────────────────────────────────────────────┘

Secrets must travel between systems—from vaults to applications, from developers to servers. Insecure transmission exposes secrets to interception and creates persistent exposure in communication systems.

Why Common Channels Are Dangerous:

Secure Secret Sharing Between Humans:

When humans need to share secrets (rare but sometimes necessary):

1. Use purpose-built tools: 1Password's secure sharing, HashiCorp Vault's response wrapping
2. One-time links: Generate a URL that can only be viewed once, then self-destructs
3. Separate channels: Send username via Slack, password via different channel, never together
4. Immediate rotation: After human-to-human sharing, rotate the secret to a new value

One-Time Secret Sharing Pattern:

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ONE-TIME SECRET SHARING                                   │
│                                                                             │
│    Sender                    Sharing Service               Recipient         │
│       │                           │                           │             │
│       │  Store secret             │                           │             │
│       │──────────────────────────►│                           │             │
│       │                           │                           │             │
│       │  Return one-time URL      │                           │             │
│       │◄──────────────────────────│                           │             │
│       │                           │                           │             │
│       │  Share URL via chat       │                           │             │
│       │───────────────────────────────────────────────────────►             │
│       │                           │                           │             │
│       │                           │  Retrieve secret (once)   │             │
│       │                           │◄──────────────────────────│             │
│       │                           │                           │             │
│       │                           │  Return secret + delete   │             │
│       │                           │──────────────────────────►│             │
│       │                           │                           │             │
│       │  (URL now returns 404)    │                           │             │
│                                                                             │
│    Tools: Vault response wrapping, onetimesecret.com, Keybase exploding     │
│    messages, 1Password's secure sharing                                     │
└─────────────────────────────────────────────────────────────────────────────┘

Many organizations implement secret storage without audit logging. When a breach occurs, they can't answer basic questions: Who accessed this secret? When? How many times? From what IP address? Without audit logs, incident response is blind.

Example Audit Log Entry:

{
  "timestamp": "2024-01-15T14:32:07.123Z",
  "request_id": "req-abc123",
  "type": "secret_read",
  "identity": {
    "type": "service_account",
    "name": "payment-service",
    "namespace": "production"
  },
  "secret": {
    "path": "database/creds/payment-db-readonly",
    "version": 3
  },
  "source": {
    "ip": "10.0.15.42",
    "port": 52341,
    "pod": "payment-service-7d9f8c6b4-abc12",
    "node": "eks-node-a"
  },
  "auth_method": "kubernetes",
  "result": "success",
  "response_ttl": 3600
}

Using Audit Logs for Detection:

A surprisingly common pattern: encrypt secrets, then store the encryption key in the same location. This provides the illusion of security while actually protecting nothing. If an attacker accesses the encrypted secrets, they also access the key to decrypt them.

Common Key Management Mistakes:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     KEY MANAGEMENT ANTI-PATTERNS                             │
│                                                                             │
│  1. KEY IN SAME REPOSITORY AS ENCRYPTED SECRETS                            │
│     /config/                                                                 │
│       ├── secrets.enc        (encrypted secrets)                            │
│       └── encryption.key     (key to decrypt them!)                         │
│                                                                             │
│  2. KEY IN ENVIRONMENT VARIABLE ON SAME SYSTEM                              │
│     > cat /path/to/secrets.enc.json                                         │
│     > echo $ENCRYPTION_KEY                                                  │
│     (both accessible to anyone on the system)                               │
│                                                                             │
│  3. HARDCODED KEY IN APPLICATION CODE                                       │
│     private static final String KEY = "SuperSecretKey123";                  │
│     (committed to version control, visible in binaries)                     │
│                                                                             │
│  4. SYMMETRIC KEY FOR BACKUP ENCRYPTION STORED WITH BACKUPS                 │
│     /backups/                                                                │
│       ├── database_2024_01_15.sql.enc                                       │
│       └── backup_key.txt                                                    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

The Correct Pattern: Key Hierarchy and Separation

┌─────────────────────────────────────────────────────────────────────────────┐
│                        KEY HIERARCHY (ENVELOPE ENCRYPTION)                   │
│                                                                             │
│                      ┌─────────────────────────────┐                        │
│                      │      ROOT KEY (KEK)         │                        │
│                      │   Stored in HSM/AWS KMS     │                        │
│                      │   Never leaves secure HW    │                        │
│                      └─────────────┬───────────────┘                        │
│                                    │ encrypts                               │
│                                    ▼                                        │
│                      ┌─────────────────────────────┐                        │
│                      │      DATA KEY (DEK)         │                        │
│                      │ Encrypted copy stored with  │                        │
│                      │ encrypted data              │                        │
│                      └─────────────┬───────────────┘                        │
│                                    │ encrypts                               │
│                                    ▼                                        │
│                      ┌─────────────────────────────┐                        │
│                      │      ENCRYPTED DATA         │                        │
│                      │    secrets.enc              │                        │
│                      └─────────────────────────────┘                        │
│                                                                             │
│  DECRYPTION REQUIRES:                                                        │
│  1. Access to encrypted data                                                │
│  2. Access to encrypted DEK                                                 │
│  3. Permission to call HSM/KMS to decrypt DEK                               │
│                                                                             │
│  Compromising (1) and (2) alone is useless without (3)                      │
└─────────────────────────────────────────────────────────────────────────────┘

Key Management Services:

Use dedicated key management services rather than managing keys yourself:

• AWS KMS: Managed service, HSM-backed, IAM integration
• GCP Cloud KMS: Similar capabilities, CMEK for many GCP services
• Azure Key Vault: Key management + secret storage
• HashiCorp Vault Transit: Encryption as a service, not just storage

Container images are immutable, versioned, and often stored in registries accessible to many people. Secrets baked into images persist in all copies of that image, including old versions, registry caches, and any system that pulled the image.

Why This Is Especially Dangerous:

1. Image layers are permanent: Even if you delete the secret in a later layer, it exists in the earlier layer
2. Images are distributed: Pushed to registries, pulled to many machines, cached locally
3. Old images persist: Previous versions with secrets remain in registry
4. Access is broad: Anyone who can pull the image can inspect layers

Detection:

# Check for secrets in image layers
docker history <image> --no-trunc

# Inspect image filesystem
docker save <image> | tar -tvf -

# Use dedicated scanning tools
trivy image <image>              # Secrets scanning
dockle <image>                   # Best practices checker

Prevention:

# BAD: Secret becomes part of image
COPY .env /app/.env
RUN npm install

# GOOD: Multi-stage build, secrets only in builder stage
FROM node:18 AS builder
RUN --mount=type=secret,id=npm_token 
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm install

FROM node:18-slim
COPY --from=builder /app/node_modules /app/node_modules
# Final image has no trace of npm_token

Runtime Secret Injection:

# Kubernetes: Mount secrets at runtime, not build time
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: myapp:latest  # No secrets in image
    volumeMounts:
    - name: secrets
      mountPath: /vault/secrets
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: app-secrets

Despite all precautions, secrets will eventually be exposed. Organizations without a rehearsed incident response plan waste critical time during breaches, making decisions under stress that they could have made calmly in advance.

Pre-Incident Preparation:

Incident Response Playbook Template:

┌─────────────────────────────────────────────────────────────────────────────┐
│                SECRET COMPROMISE INCIDENT RESPONSE                           │
│                                                                             │
│  PHASE 1: DETECTION (Minutes 0-5)                                           │
│  □ Confirm the exposure (is this a real secret? Is it still valid?)        │
│  □ Identify the secret type and sensitivity level                           │
│  □ Page incident commander if HIGH/CRITICAL                                 │
│                                                                             │
│  PHASE 2: CONTAINMENT (Minutes 5-15)                                        │
│  □ Revoke the exposed secret at the target system                           │
│  □ Block known malicious IPs/sources if applicable                          │
│  □ Generate new secret                                                       │
│                                                                             │
│  PHASE 3: REMEDIATION (Minutes 15-60)                                       │
│  □ Deploy new secret to all legitimate consumers                            │
│  □ Verify legitimate consumers can authenticate                             │
│  □ Confirm revoked secret no longer works                                   │
│                                                                             │
│  PHASE 4: INVESTIGATION (Hours)                                             │
│  □ Review access logs for unauthorized usage                                │
│  □ Determine scope of potential data access                                 │
│  □ Identify root cause of exposure                                          │
│  □ Document timeline and decisions                                          │
│                                                                             │
│  PHASE 5: REMEDIATION & PREVENTION (Days)                                   │
│  □ Implement controls to prevent recurrence                                 │
│  □ Update runbooks with lessons learned                                     │
│  □ Complete incident report                                                 │
│  □ Notify affected parties if required                                      │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

The ten mistakes we've examined represent the most common and damaging secrets management failures. Most are preventable with awareness, tooling, and process discipline.

Module Conclusion:

This concludes the Secrets Management module. You now understand:

• What secrets are and why they require special handling
• The critical distinction between secrets and configuration
• The complete secrets lifecycle from creation through revocation
• The most common mistakes and how to prevent them

In the next module, we'll explore Secrets Management Tools—the platforms and systems that implement these principles: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and how to choose between them.