What Are Secrets - Secrets Management | OneNoughtOne

0/273

System Design HLDSecrets Management

Secrets Management

LevelIntermediate

Duration60 mins

TopicSecrets Management

1 / 4

What Are Secrets

The Crown Jewels of Software Systems

In December 2023, a Fortune 500 company discovered that their AWS root credentials had been exposed in a public GitHub repository for 47 days. The credentials, accidentally committed by a junior developer, were scraped by automated bots within 11 minutes of being pushed. By the time the breach was detected, attackers had provisioned over $2.3 million in cryptocurrency mining infrastructure and exfiltrated customer data affecting 4.2 million users.

This incident—one of thousands that occur annually—illustrates why secrets are often called the crown jewels of software systems. A single exposed secret can unravel an entire organization's security posture, bypass every firewall, and render all other security investments meaningless.

What You Will Learn

By the end of this page, you will understand exactly what constitutes a secret in software systems, the taxonomy of secret types and their sensitivity levels, why secrets require fundamentally different handling than regular configuration, and the core principles that govern effective secrets management at any scale.

Defining Secrets: The Fundamental Concept

A secret is any piece of information that, if disclosed to unauthorized parties, could compromise the security, integrity, or availability of a system. While this definition seems straightforward, its implications are profound: secrets are not merely "sensitive data"—they are the authentication and authorization primitives that systems use to establish trust.

The Formal Definition:

From a security engineering perspective, a secret is data that satisfies three criteria:

Confidentiality-Critical: Disclosure to unauthorized parties causes harm
Access-Enabling: The secret grants capabilities or access to resources
Non-Derivable: The secret cannot be computed from publicly available information

This tripartite definition helps us distinguish true secrets from data that is merely "private" or "sensitive." Your social security number is sensitive but doesn't grant system access. Your database password satisfies all three criteria—it must remain confidential, it enables access, and it cannot be guessed from public information.

The Litmus Test for Secrets

Ask yourself: If this value were posted publicly on Twitter right now, would I need to immediately revoke or rotate it? If yes, it's a secret. If no—even if the disclosure would be embarrassing—it's probably configuration or sensitive data, not a secret in the security-engineering sense.

Why the Distinction Matters:

Many security breaches occur because organizations treat secrets like ordinary configuration. They store database passwords in the same config files as application settings, manage API keys in the same repositories as feature flags, and grant the same access controls to encryption keys as to timeout values.

The fundamental difference is that configuration survives disclosure; secrets do not. You can publish your application's default timeout (30 seconds) without consequence. Publishing your database password requires immediate rotation, potential breach investigation, and possibly regulatory notification.

The Trust Chain:

Secrets form the basis of trust in distributed systems. When Service A calls Service B with an API key, that key represents a compressed form of trust: "The holder of this key has been authorized to perform certain actions." This trust chain propagates through entire architectures:

Your application trusts the database connection string
Your CI/CD pipeline trusts the deployment credentials
Your users trust that SSL certificates are genuine
Your services trust each other through mutual TLS certificates

Compromise any link in this chain, and the entire trust model collapses.

Taxonomy of Secrets: Types and Categories

Secrets in software systems come in various forms, each with distinct characteristics, lifecycles, and management requirements. Understanding this taxonomy is essential for designing appropriate handling mechanisms.

Comprehensive Taxonomy of Secrets in Software Systems
Secret Type	Examples	Sensitivity	Rotation Frequency	Typical Lifespan
Cryptographic Keys	AES keys, RSA private keys, HMAC secrets	Critical	Annually or per-incident	1-3 years
Database Credentials	PostgreSQL passwords, MongoDB connection strings	High	Quarterly	90 days
API Keys	Stripe API keys, AWS access keys, third-party service tokens	High	Quarterly to annually	90-365 days
Service Account Tokens	Kubernetes service accounts, GCP service account keys	High	Quarterly	90 days
OAuth Client Secrets	Client secrets for OAuth2 flows	High	Annually	1 year
TLS/SSL Certificates	Server certificates, client certificates	High	Annually	1 year (Let's Encrypt: 90 days)
SSH Keys	Developer SSH keys, deploy keys	Medium-High	Annually	1 year
JWT Signing Keys	Keys used to sign JWT tokens	Critical	Annually	1-2 years
Encryption Passphrases	Master passwords, key encryption keys	Critical	Per-incident only	Indefinite
Session Secrets	Cookie signing keys, session encryption keys	High	Monthly to quarterly	30-90 days

Deep Dive: Critical Secret Categories

1. Cryptographic Keys

These are the most sensitive secrets in any system. A compromised cryptographic key can:

Decrypt all data ever encrypted with that key
Forge signatures that appear legitimate
Impersonate the key holder indefinitely
Undermine the entire security model of applications

Cryptographic keys are unique because historical exposure matters. Unlike a password that can be changed, an encryption key that was once exposed means all data encrypted with that key—past, present, and future—must be considered compromised.

2. Database Credentials

Database credentials represent direct access to your data layer. Compromise typically enables:

Complete data exfiltration
Data modification or deletion
Privilege escalation if the database user has admin rights
Lateral movement to other systems via stored procedures or linked servers

3. Cloud Provider Credentials

AWS access keys, GCP service account keys, and Azure service principals are particularly dangerous because they often carry broad permissions. A single AWS root access key can:

Provision unlimited infrastructure (cryptomining attacks)
Access all S3 buckets and their contents
Modify IAM policies to create persistent backdoors
Delete CloudTrail logs to cover tracks

4. Signing Keys

JWT signing keys, code signing certificates, and API request signing keys enable forgery attacks. An attacker with your JWT signing key can:

Generate valid tokens for any user
Escalate privileges at will
Bypass all application-level access controls
Create tokens that never expire

The Hierarchy of Secrets

Not all secrets are equal. Encryption keys that protect other secrets (Key Encryption Keys, or KEKs) are more critical than the secrets they protect. Your secrets management strategy must account for this hierarchy—losing a database password is bad, but losing the master key that encrypts all your secrets is catastrophic.

Sensitivity Levels and Classification

Effective secrets management requires classifying secrets by sensitivity level. This classification drives decisions about storage mechanisms, access controls, audit requirements, and rotation policies.

Secret Sensitivity Classification Framework

•CRITICAL (Level 4) — Master keys, root credentials, signing keys. Compromise enables complete system takeover. Examples: AWS root account credentials, production database master password, JWT signing keys. Requirements: HSM storage, dual-control access, real-time alerting, maximum 2-person access.
•HIGH (Level 3) — Service credentials, API keys, database passwords. Compromise enables significant data access or service impersonation. Examples: Production database credentials, payment provider API keys, service-to-service authentication tokens. Requirements: Encrypted vault storage, role-based access, automated rotation, audit logging.
•MEDIUM (Level 2) — Developer credentials, staging environment secrets, non-critical service tokens. Compromise enables limited access or development environment exposure. Examples: Staging database passwords, development API keys, internal tool credentials. Requirements: Encrypted storage, team-based access controls, periodic rotation.
•LOW (Level 1) — Public API keys with rate limiting, read-only service tokens, monitoring credentials. Compromise causes inconvenience but limited harm. Examples: Public search API keys, read-only dashboard tokens, log aggregation credentials. Requirements: Basic encryption, broad team access acceptable, annual rotation.

Classification Decision Framework:

When classifying a secret, consider the following impact dimensions:

Dimension	Questions to Ask
Data Access	What data becomes accessible? Customer PII? Financial records? All data?
System Control	What actions become possible? Read-only? Read-write? Administrative?
Blast Radius	How many systems/users are affected? Single service? Entire platform?
Recoverability	Can the impact be reversed? Data restored? Or is damage permanent?
Detection Time	How quickly would compromise be detected? Minutes? Days? Never?
Regulatory Impact	Does exposure trigger compliance notifications? GDPR? HIPAA? PCI-DSS?

The Principle of Least Authority:

Secrets should be scoped to the minimum authority necessary. Instead of one database credential that accesses all tables, create separate credentials for each service with access limited to required tables. Instead of one AWS access key with AdministratorAccess, create keys scoped to specific services and actions.

This principle serves two purposes:

Blast radius reduction: A compromised secret has limited impact
Audit clarity: Access patterns become meaningful signal, not noise

Secrets in Distributed Systems

Modern distributed systems present unique challenges for secrets management. Unlike monolithic applications where secrets might exist in a single configuration file, microservices architectures distribute secrets across dozens or hundreds of services, each requiring secure access to different secrets.

Distributed System Challenges

•Secrets must be distributed to many services
•Services run in ephemeral containers
•Multiple environments (dev, staging, prod) multiply secret count
•Auto-scaling creates dynamic secret consumers
•Network boundaries require secret transmission
•Audit trails become complex across services

Solutions and Patterns

•Centralized secrets vaults with distributed access
•Dynamic secrets with short lifespans
•Environment-specific secret namespaces
•Identity-based access (workload identity)
•Encrypted transmission with mTLS
•Centralized audit logging with correlation IDs

The Secrets Distribution Problem:

Consider a typical microservices architecture with 50 services. Each service needs:

Database credentials (potentially different databases)
API keys for third-party services it integrates with
Service-to-service authentication tokens
Encryption keys for data at rest
TLS certificates for encrypted communication

With 50 services, you might manage 200+ secrets, each requiring:

Secure storage with encryption at rest
Access controls limiting which services can read which secrets
Rotation capability without service disruption
Audit logging of all access
Different values across dev, staging, and production

Multiplying 200 secrets by 3 environments gives 600 secret entries to manage. This is the secrets sprawl problem that necessitates dedicated secrets management solutions.

Workload Identity: The Modern Approach:

Traditional approaches gave every service instance a copy of its secrets. Modern systems use workload identity—each service instance proves its identity ("I am an instance of the payment service") and receives secrets dynamically based on that identity.

This approach provides:

No static secrets on disk: Secrets exist only in memory, for short durations
Fine-grained access control: Policies define which identities access which secrets
Automatic rotation: New instances automatically receive current secrets
Revocation capability: Revoking a workload's identity immediately cuts access

The Zero-Trust Principle for Secrets

In a zero-trust architecture, no service is inherently trusted to hold secrets. Every secret access requires authentication (prove who you are), authorization (prove you should have access), and generates an audit record. This principle applies even to internal services—especially to internal services, since most breaches involve compromised internal components.

Where Secrets Live: Storage Locations and Risks

Understanding where secrets reside in a system is critical for security assessment. Secrets can accumulate in surprising places, often persisting long after their intended use.

Common Secret Storage Locations and Associated Risks
Location	Risk Level	Common Issues	Mitigation
Environment Variables	Medium	Visible in process listings, crash dumps, logs	Use only for pointing to vault, not storing actual secrets
Configuration Files	High	Committed to version control, backed up unencrypted	Never store secrets in config files
Version Control (Git)	Critical	History preserves secrets forever, even after deletion	Use git-secrets, pre-commit hooks, history scanning
CI/CD Systems	High	Logs may expose secrets, broad access	Use CI/CD secret features, mask outputs
Container Images	High	Layers preserve secrets, images distributed widely	Multi-stage builds, never bake in secrets
Cloud Metadata	Medium	SSRF attacks can access, overly broad by default	Limit metadata access, use IMDSv2
Kubernetes Secrets	Medium	Base64 encoded (not encrypted), etcd storage	Enable encryption at rest, use external secrets operators
Developer Machines	High	Stolen laptops, malware, insecure backups	Avoid local secrets, use short-lived tokens
Log Files	High	Secrets in error messages, debug output	Log redaction, never log secrets
Memory/Core Dumps	Medium	Crash dumps contain in-memory secrets	Disable core dumps in production, secure dump storage

The Permanence Problem:

Secrets are difficult to fully remove once they've been in the wrong place. Consider:

A database password committed to Git exists in repository history forever
A secret logged to CloudWatch persists until the log group expires (possibly years)
A secret in a container image layer exists in every registry and every system that pulled that image
A secret emailed to a team member exists in multiple email servers' backups indefinitely

Defense in Depth for Secret Storage:

Effective secrets protection requires multiple layers:

┌─────────────────────────────────────────────────────────────┐
│                     Access Control Layer                      │
│  (Authentication, Authorization, Policy Enforcement)          │
├─────────────────────────────────────────────────────────────┤
│                     Encryption Layer                          │
│  (Encryption at Rest, Encryption in Transit)                  │
├─────────────────────────────────────────────────────────────┤
│                     Audit Layer                               │
│  (Access Logging, Anomaly Detection, Alerting)                │
├─────────────────────────────────────────────────────────────┤
│                     Secrets Vault                             │
│  (HashiCorp Vault, AWS Secrets Manager, etc.)                 │
└─────────────────────────────────────────────────────────────┘

Each layer provides protection independent of the others:

If access control fails, encryption prevents disclosure
If encryption is somehow bypassed, access logs enable detection
If audit alerting fails, limited access scope contains damage

The Threat Landscape: How Secrets Get Compromised

Understanding how secrets get compromised helps design effective protection. Most secret exposures fall into predictable categories, many of which are preventable with proper tooling and practices.

Primary Attack Vectors for Secrets

•Accidental Commits to Version Control — The #1 cause of secret exposure. Developers commit secrets to public or even private repositories. Automated scanners continuously scrape GitHub, GitLab, and Bitbucket for exposed credentials. Studies show exposed AWS keys are exploited within 15 minutes on average.
•Insecure Configuration Transmission — Secrets shared via Slack, email, Confluence, or other collaboration tools. These channels lack encryption, access control, and audit logging. Screenshots, forwards, and backups preserve secrets indefinitely.
•Server-Side Request Forgery (SSRF) — Attackers trick servers into making requests to internal endpoints like cloud metadata APIs (169.254.169.254), exposing IAM credentials and other instance metadata.
•Log Injection and Exposure — Secrets included in log messages, error reports, or stack traces. Centralized logging systems then distribute these secrets to log aggregators, SIEMs, and backup systems.
•Supply Chain Attacks — Compromised dependencies exfiltrate environment variables or file contents. NPM, PyPI, and other package repositories regularly see malicious packages.
•Insider Threats — Employees or contractors with legitimate access misuse their privileges. Over-permissive access grants create opportunities for both malicious actors and accidental exposure.
•Credential Stuffing and Brute Force — Weak or default secrets become vulnerable to automated attack tools. This is especially common with database credentials and admin panels.
•Memory Scraping — Malware or exploits dump process memory to extract secrets. Especially dangerous for secrets that remain in memory long-term.

The Attacker's Perspective

Attackers specifically target secrets because they provide immediate, high-value access. A single valid AWS access key is worth more than any vulnerability discovery—it provides direct access without exploitation complexity. This is why secrets are the first thing attackers look for after gaining any foothold.

Core Principles of Secrets Management

Before diving into specific tools and techniques in subsequent pages, it's essential to understand the foundational principles that guide all effective secrets management.

The Seven Principles of Secrets Management

•Never Store Secrets in Code — Source code is versioned, shared, and often made public. Secrets must be external to code and injected at runtime. This principle has no exceptions.
•Encrypt Secrets at Rest and in Transit — Secrets must be encrypted when stored (at rest) and when transmitted (in transit). Both are necessary—encryption at rest protects against physical access; encryption in transit protects against network interception.
•Implement Least-Privilege Access — Every service, user, and system should have access only to secrets they specifically need. Broad access grants are convenient but catastrophically increase blast radius.
•Audit All Secret Access — Every read, write, and administrative action on secrets must generate an audit record. These logs enable breach detection, forensics, and compliance demonstration.
•Rotate Secrets Regularly — Regular rotation limits the window of opportunity for compromised secrets. Automated rotation ensures consistency and removes manual intervention risks.
•Plan for Revocation — Every secret must have a clear, tested revocation procedure. When a breach occurs, the ability to quickly revoke and replace secrets is often the difference between incident and catastrophe.
•Treat Secrets as Ephemeral — Prefer short-lived secrets over long-lived ones. Secrets that expire in hours are less valuable to attackers than secrets that last months.

Why These Principles Matter:

Each principle addresses specific failure modes:

Principle	Failure Mode Prevented
Never store in code	Accidental commits, version control exposure
Encrypt at rest/transit	Theft from storage, network interception
Least-privilege access	Blast radius amplification, insider threats
Audit all access	Undetected compromise, compliance failures
Rotate regularly	Extended compromise windows, stale credentials
Plan for revocation	Slow incident response, incomplete remediation
Prefer ephemeral	Long-term exposure value, credential theft ROI

These principles are not optional best practices—they are the minimum viable security posture for any system handling sensitive data.

Summary: Understanding Secrets

We've established a comprehensive foundation for understanding secrets in software systems. Let's consolidate the key concepts:

Key Takeaways

•Secrets are access-enabling credentials — Any data that, if disclosed, grants unauthorized access to systems or data. They are the foundation of trust in software systems.
•Not all secrets are equal — Classification by sensitivity drives appropriate protection measures. Critical secrets like encryption keys require HSM storage and dual-control access.
•Distributed systems multiply complexity — Modern architectures require managing hundreds of secrets across ephemeral infrastructure, necessitating automated, policy-driven management.
•Secrets persist in unexpected places — Version control history, logs, container layers, and backups can harbor secrets long after rotation, requiring defense in depth.
•Most exposures are preventable — The primary attack vectors (accidental commits, insecure sharing, log exposure) have well-known mitigations.
•Seven core principles guide all decisions — Never in code, encrypt at rest/transit, least privilege, audit all access, rotate regularly, plan revocation, prefer ephemeral.

What's Next:

Now that we understand what secrets are and why they matter, the next page explores a crucial distinction: Secrets vs. Configuration. Many security failures occur because teams conflate these two concepts, managing database passwords with the same rigor (or lack thereof) as feature flag values. Understanding this distinction is fundamental to designing appropriate management strategies for each.

Page Complete

You now have a comprehensive understanding of what secrets are, their taxonomy, sensitivity classification, storage locations, threat landscape, and the core principles of secrets management. This foundation prepares you to distinguish secrets from configuration, understand the secrets lifecycle, and avoid common mistakes in subsequent pages.