Secrets Management - Learning Module

Loading content...

0/273

Secrets vs Configuration

The Blurred Line That Causes Breaches

In 2019, a major financial services company suffered a data breach affecting 100 million customers. The root cause? Their CI/CD pipeline treated database credentials with the same access controls as environment names and feature flags. When a contractor was granted access to "configuration" for debugging, they inadvertently received access to production database passwords, which were later discovered on a dark web marketplace.

This incident exemplifies a confusion that plagues software engineering: the failure to distinguish secrets from configuration. Both are external to code, both are injected at runtime, and both vary across environments. But treating them the same way is a recipe for security failures.

What You Will Learn

By the end of this page, you will understand the fundamental differences between secrets and configuration, be able to classify any piece of external data correctly, know the appropriate handling mechanisms for each category, and understand why the distinction matters for security, operations, and compliance.

The Definition Spectrum: Configuration to Secrets

External application data exists on a spectrum from purely public to absolutely secret. Understanding where different data types fall on this spectrum is essential for correct handling.

The Spectrum Visualized:

┌─────────────────────────────────────────────────────────────────────────────┐
│                                                                             │
│  PUBLIC        INTERNAL         SENSITIVE         SECRET         CRITICAL  │
│  CONFIG        CONFIG           DATA              CREDENTIALS    SECRETS   │
│                                                                             │
│  ◄────────────────────────────────────────────────────────────────────────► │
│                                                                             │
│  • App name     • Internal       • User PII       • API keys     • Master  │
│  • Timeouts       endpoints      • Email           • Database       keys   │
│  • Feature       • Service         addresses        passwords    • Signing │
│    flags          ports          • Analytics      • OAuth          keys    │
│  • UI themes    • Cache           data              secrets      • Root    │
│                   TTLs                            • Encryption     creds   │
│                                                     keys                   │
│                                                                             │
│  ◄── Can be public ──►◄── Internal only ──►◄── Encrypted, audited ───────► │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Key Distinction: Disclosure Impact

The fundamental difference between configuration and secrets is the impact of disclosure:

Configuration: Disclosure may be embarrassing or reveal architectural details, but doesn't directly enable attacks
Secrets: Disclosure directly enables unauthorized access to systems or data

This distinction drives every decision about storage, access control, transmission, and auditing.

Configuration vs Secrets: Head-to-Head Comparison
Characteristic	Configuration	Secrets
Disclosure impact	Indirect: May reveal architecture	Direct: Enables unauthorized access
Requires encryption at rest	No (usually)	Yes (always)
Requires encryption in transit	Recommended	Required
Access control granularity	Team or role-based	Individual or service-based
Audit requirements	Recommended	Required with alerting
Rotation frequency	As needed for changes	Mandatory regular schedule
Version control storage	Acceptable (usually)	Never acceptable
Log visibility	Can be logged	Must never be logged
Sharing mechanism	Wiki, Confluence, Slack	Encrypted vault with access control
Backup considerations	Normal backup procedures	Encrypted backups only

Classification Decision Framework

When encountering external application data, use this rigorous decision framework to classify it correctly. The framework asks three essential questions, and the answers determine the classification.

The Three Essential Questions

•Does this value grant access? — Would possessing this value allow someone to authenticate, read data, perform actions, or impersonate a service? If yes, it's likely a secret.
•Would disclosure require immediate action? — If this value appeared publicly right now, would you need to rotate, revoke, or replace it immediately? If yes, it's definitely a secret.
•Is this value unique or derivable? — Is this something that could be guessed, computed from public information, or is publicly specified? Secrets are values that cannot be legitimately obtained through any means other than authorized access.

The Classification Flow:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     DOES THIS VALUE GRANT ACCESS?                           │
│                              │                                              │
│                    ┌─────────┴─────────┐                                    │
│                   YES                   NO                                  │
│                    │                    │                                   │
│                    ▼                    ▼                                   │
│        ┌──────────────────┐   ┌─────────────────────────┐                   │
│        │     SECRET       │   │ Would disclosure require │                  │
│        │                  │   │    immediate action?     │                  │
│        └──────────────────┘   └───────────┬─────────────┘                   │
│                                          │                                  │
│                              ┌───────────┴───────────┐                      │
│                             YES                       NO                    │
│                              │                        │                     │
│                              ▼                        ▼                     │
│                   ┌──────────────────┐    ┌─────────────────────────┐       │
│                   │     SECRET       │    │ Is this value unique or │       │
│                   │                  │    │   could it be guessed?  │       │
│                   └──────────────────┘    └───────────┬─────────────┘       │
│                                                       │                     │
│                                           ┌───────────┴───────────┐         │
│                                        UNIQUE              DERIVABLE        │
│                                           │                    │            │
│                                           ▼                    ▼            │
│                               ┌──────────────────┐ ┌──────────────────────┐ │
│                               │ SENSITIVE CONFIG │ │   CONFIGURATION      │ │
│                               │ (treat carefully)│ │                      │ │
│                               └──────────────────┘ └──────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

When in Doubt, Treat as Secret

If you're uncertain whether something is configuration or a secret, treat it as a secret. The operational overhead of over-protecting configuration is minimal. The cost of under-protecting a secret can be catastrophic. This conservative approach is standard in security engineering.

Common Examples: Applying the Framework

Let's apply the classification framework to common data types found in software systems. This exercise builds intuition for making correct classifications in your own systems.

Classification of Common External Application Data
Data Item	Grants Access?	Immediate Action?	Classification	Rationale
Database password	Yes	Yes	SECRET	Direct data access, requires immediate rotation
Database hostname	No	No	CONFIG	Internal detail, not access-enabling
Database port	No	No	CONFIG	Public information (5432, 3306, etc.)
AWS access key	Yes	Yes	SECRET	Full account access possible
AWS region	No	No	CONFIG	Public information (us-east-1, etc.)
API endpoint URL	No	No	CONFIG	Architecture detail, not credential
Stripe API key (secret)	Yes	Yes	SECRET	Enables financial transactions
Stripe API key (publishable)	Partially	No	CONFIG	Designed for public exposure, limited scope
JWT signing key	Yes	Yes	SECRET	Enables token forgery
JWT expiration time	No	No	CONFIG	Policy setting, not credential
SMTP password	Yes	Yes	SECRET	Enables email sending, potential phishing
SMTP server address	No	No	CONFIG	Public service endpoint
Feature flag value	No	No	CONFIG	Business logic, not access control
OAuth client secret	Yes	Yes	SECRET	Enables impersonation of application
OAuth client ID	No	No	CONFIG	Designed for public exposure
Encryption key	Yes (to data)	Yes	SECRET	Enables decryption of protected data
Encryption algorithm	No	No	CONFIG	Public standard (AES-256-GCM, etc.)
Service account private key	Yes	Yes	SECRET	Full service impersonation
Log level (DEBUG/INFO)	No	No	CONFIG	Runtime behavior, not access
Rate limit values	No	No	CONFIG	Policy setting

The Publishable Key Exception

Some API keys are designed for public exposure (Stripe publishable keys, Google Maps client-side keys). These are intentionally scoped to safe operations and rate-limited. However, their 'secret' counterparts require full secret handling. Always verify with the provider's documentation which keys are safe for public use.

The Danger of Conflating Secrets and Configuration

When organizations fail to distinguish between secrets and configuration, predictable security failures occur. Understanding these failure modes helps build the case for proper separation.

Consequences of Treating Secrets Like Config

•Secrets committed to version control
•Secrets visible in deployment logs
•Secrets accessible to all developers
•No audit trail for secret access
•Secrets persisting in backups indefinitely
•Secrets exposed via configuration APIs
•Secrets shared via insecure channels

Consequences of Treating Config Like Secrets

•Operational friction for routine changes
•Audit log noise obscuring real threats
•Unnecessary complexity in deployment
•Slower incident response debugging
•Reduced developer productivity
•Over-burdened secrets vault systems
•Compliance confusion during audits

Real-World Case Studies:

Case 1: The Config File Breach

A SaaS company stored all external configuration in a single config.yaml file per environment. This included feature flags, timeout values, and database passwords. When a support engineer needed to check the production timeout values to debug a customer issue, they were given read access to the entire configuration—including production database credentials. The credentials were later found in a paste site.

Case 2: The Environment Variable Explosion

Another organization put all secrets in environment variables, following the "12-factor app" methodology literally. However, environment variables appeared in process listings, crash dumps, and were accessible to any debugging tool. A junior developer running ps eww to debug a memory issue accidentally captured and shared database passwords in a bug report.

Case 3: The Terraform State File

A DevOps team used Terraform to manage infrastructure. They understood that Terraform state files contain sensitive data and stored them in an encrypted S3 bucket. However, they also stored non-sensitive configuration in the same state. When they needed to share configuration values with a partner team, they exported the state—including database passwords that Terraform had provisioned.

The Common Thread:

In each case, the failure wasn't in security controls themselves, but in the decision about what those controls should protect. Proper classification would have prevented every incident.

The False Economy

Teams often resist separating secrets from configuration because 'it's simpler to have everything in one place.' This simplicity is a false economy. The cost of implementing separate handling is paid once. The cost of a breach from improper handling can be paid repeatedly—in incident response, regulatory fines, reputational damage, and engineering time spent on remediation.

Proper Handling Mechanisms for Each Category

Once data is correctly classified, the handling mechanism follows naturally. Different categories require fundamentally different tooling and processes.

Secrets Handling Requirements

•Storage: Dedicated secrets vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager). Never in files, environment variables, or repositories.
•Encryption: Encryption at rest using AES-256 or equivalent. Encryption in transit via TLS 1.2+. Consider envelope encryption for defense in depth.
•Access Control: Identity-based access with explicit policies. Principle of least privilege. No shared credentials for secret access.
•Audit Logging: Complete audit trail of all access (who, what, when, from where). Real-time alerting on anomalous access patterns.
•Rotation: Automated rotation on regular schedules. Maximum secret lifetime policies. No manual rotation processes.
•Injection: Secrets injected at runtime, never baked into images or configurations. Prefer in-memory secret handling.
•Revocation: Documented, tested revocation procedures. Ability to invalidate within minutes of discovering compromise.
•Backup: Encrypted backups only. Consider whether backup of secrets is necessary—vault replication may be preferable.

The Separation Architecture:

Proper separation requires different storage and access patterns:

┌─────────────────────────────────────────────────────────────────────────────┐
│                           APPLICATION                                        │
│                               │                                              │
│              ┌────────────────┼────────────────┐                            │
│              │                │                │                             │
│              ▼                ▼                ▼                             │
│    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐              │
│    │ Config Service  │ │ Secrets Vault   │ │ Feature Flags   │              │
│    │ (Consul, etcd)  │ │ (Vault, ASM)    │ │ (LaunchDarkly)  │              │
│    └────────┬────────┘ └────────┬────────┘ └────────┬────────┘              │
│             │                   │                   │                        │
│             ▼                   ▼                   ▼                        │
│    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐              │
│    │  Git Repository │ │  HSM / KMS      │ │   Flag Store    │              │
│    │  (backing)      │ │  (backing)      │ │   (backing)     │              │
│    └─────────────────┘ └─────────────────┘ └─────────────────┘              │
│                                                                              │
│    Access: All devs      Access: Prod team   Access: Product team           │
│    Audit: Git history    Audit: Real-time    Audit: Change log              │
│    Encryption: None      Encryption: Yes     Encryption: None               │
└─────────────────────────────────────────────────────────────────────────────┘

This architecture provides:

Right-sized security for each data type
Appropriate access patterns for each team
Auditing proportional to sensitivity
Operational simplicity where security tolerates it

The Environment Variables Debate

The 12-factor app methodology popularized environment variables for configuration, leading many to use them for secrets as well. This conflation deserves careful examination.

What 12-Factor Actually Says:

The 12-factor app methodology states:

"Store config in the environment."

But the same document defines configuration as:

"Everything that is likely to vary between deploys (staging, production, developer environments, etc.)"

This definition encompasses both configuration (database hostnames) and secrets (database passwords). The methodology was written in 2011, before secrets management tools were mature. Taking it literally in 2024 ignores a decade of security engineering advancement.

Problems with Environment Variables for Secrets:

Security Risks of Secrets in Environment Variables

•Process Visibility: Environment variables appear in /proc/<pid>/environ and process listings. Any process on the system can potentially read them.
•Child Process Inheritance: Environment variables are automatically inherited by child processes, expanding the attack surface beyond the intended application.
•Crash Dump Inclusion: Core dumps and crash reports often include the full environment, leaking secrets to debugging systems.
•No Access Logging: There's no way to audit when environment variables are read. You cannot detect suspicious access.
•No Rotation Mechanism: Changing environment variables requires process restart or complex signaling, making rotation operationally difficult.
•Shell History Exposure: Setting environment variables often leaves traces in shell history files.
•Container Orchestration Risks: In Kubernetes, environment variables from ConfigMaps and Secrets are stored in etcd and visible in pod specifications.

The Modern Approach:

Environment variables remain useful for pointing to secrets, not storing them:

# BAD: Secret in environment variable
DATABASE_PASSWORD="super-secret-password-123"

# GOOD: Pointer to secret location
DATABASE_PASSWORD_PATH="/vault/secrets/database/password"
VAULT_ADDR="https://vault.internal:8200"
VAULT_ROLE="application-service"

The application reads these environment variables to determine where to fetch secrets from, then uses authenticated API calls to retrieve the actual secret values. This approach:

Maintains 12-factor container portability
Keeps secrets out of the process environment
Enables proper access control and audit logging
Supports dynamic secret rotation without restarts

The Pragmatic Middle Ground

For development environments and non-production systems, environment variables for secrets may be acceptable given security/productivity tradeoffs. For production systems handling real user data, proper secrets management is non-negotiable. Your CI/CD pipeline should enforce this distinction—fail builds that attempt to deploy environment-variable secrets to production.

Organizational Implications of Proper Separation

Separating secrets from configuration isn't just a technical decision—it has organizational implications for team structure, access policies, and operational practices.

Access Control Implications:

Proper separation enables sensible access policies:

Role	Config Access	Secret Access
All Developers	Full read, environment-specific write	Development only
Senior Developers	Full read/write	Staging + Development
DevOps/SRE	Full read/write	All environments
Security Team	Audit only	Full, including audit
Contractors	Project-specific read	None (default)
Automated Systems	Service-specific	Service-specific

Operational Implications:

Separation enables different operational processes:

Configuration changes: Can be made through standard deployment pipelines with code review
Secret changes: Require elevated access, out-of-band verification, and immediate audit notification

This separation means configuration changes are routine (multiple per day), while secret changes are exceptional events that warrant attention.

Compliance Implications:

Regulatory frameworks (SOC 2, HIPAA, PCI-DSS, GDPR) require evidence that sensitive credentials are:

Encrypted at rest and in transit
Subject to access controls
Covered by audit logging
Regularly rotated

When secrets are mixed with configuration, demonstrating these controls becomes difficult. Auditors must examine your entire configuration management system rather than a focused secrets vault. Proper separation simplifies compliance evidence collection significantly.

The Cultural Shift

Implementing proper separation often requires cultural change. Developers accustomed to having all configuration in one place must learn new patterns. This is an investment that pays dividends in security posture, operational clarity, and compliance simplification. Frame it as enabling developer productivity in safe ways, not as adding security burden.

Summary: Secrets vs Configuration

The distinction between secrets and configuration is fundamental to secure system design. Let's consolidate the key insights:

Key Takeaways

•Configuration survives disclosure; secrets do not — This fundamental difference drives all handling decisions. Publishing your timeout is harmless; publishing your database password requires immediate action.
•Use the three-question framework — Does it grant access? Would disclosure require immediate action? Is it unique or derivable? These questions reliably classify any external data.
•When in doubt, treat as secret — The cost of over-protecting configuration is operational friction. The cost of under-protecting secrets is security breach.
•Environment variables are pointers, not storage — Use environment variables to tell applications where to find secrets, not to store the secrets themselves.
•Separation enables appropriate access — Different data types need different access patterns. Mixing them forces either over-permissive or operationally painful access grants.
•Separation simplifies compliance — Auditors can focus on your secrets vault rather than examining all configuration handling.

What's Next:

With a clear understanding of what secrets are and how they differ from configuration, the next page explores the Secrets Lifecycle. Understanding how secrets are created, distributed, used, rotated, and revoked is essential for designing complete secrets management systems that handle the full lifecycle rather than just storage.

Page Complete

You now understand the critical distinction between secrets and configuration, can apply the classification framework to any external data, know the appropriate handling mechanisms for each category, and understand the organizational implications of proper separation.