Secrets Tools - Learning Module

Loading content...

0/273

AWS Secrets Manager

Managed Secrets for the AWS Ecosystem

AWS Secrets Manager represents Amazon's answer to the secrets management challenge—a fully managed service that eliminates the operational burden of running and scaling secrets infrastructure. For organizations deeply invested in the AWS ecosystem, Secrets Manager provides native integrations that make secrets access seamless across AWS services.

Unlike self-hosted solutions, Secrets Manager abstracts away the complexity of high availability, encryption key management, and infrastructure maintenance. You get a secrets API that integrates naturally with IAM, CloudFormation, and the broader AWS service catalog—paying only for what you use.

What You Will Learn

By the end of this page, you will understand AWS Secrets Manager's architecture, automatic rotation capabilities, integration patterns with AWS services, cross-account access strategies, and cost optimization techniques. You'll be equipped to design secrets management solutions for AWS-centric workloads.

AWS Secrets Manager Fundamentals

AWS Secrets Manager is a secrets management service that enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. It was purpose-built to solve the challenge of secrets sprawl in cloud environments.

Core Capabilities:

Key Features

•Centralized Secret Storage — Store secrets as encrypted JSON blobs with versioning support
•Automatic Rotation — Built-in rotation for RDS, Redshift, DocumentDB with Lambda-based custom rotation
•Fine-Grained Access Control — IAM policies and resource-based policies for precise permission management
•Encryption with KMS — Secrets encrypted using AWS KMS; supports customer-managed keys (CMKs)
•Cross-Region Replication — Replicate secrets to multiple regions for disaster recovery
•Audit Trail — CloudTrail integration provides complete audit logging of all secret access
•AWS Service Integration — Native support in RDS, ECS, Lambda, EKS, and more

Secret Structure:

Secrets Manager stores secrets as encrypted blobs that can contain any data up to 64KB. The most common pattern is storing structured data as JSON:

Secret Examples
JSON Secret
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// Database credential secret
{
  "username": "admin",
  "password": "MyStr0ngP@ssword!",
  "engine": "postgres",
  "host": "mydb.cluster-abc123.us-east-1.rds.amazonaws.com",
  "port": 5432,
  "dbname": "production",
  "dbClusterIdentifier": "mydb-cluster"
}
 
// API key secret
{
  "api_key": "sk_live_abc123xyz789",
  "api_secret": "whsec_def456uvw321",
  "environment": "production"
}
 
// Third-party integration
{
  "client_id": "oauth_client_abc123",
  "client_secret": "cs_secret_xyz789",
  "token_endpoint": "https://auth.example.com/oauth/token",
  "scopes": ["read", "write"]
}

Secret Naming Conventions

Use hierarchical naming with prefixes: 'prod/myapp/database/primary', 'dev/myapp/api/stripe'. This enables IAM policies that grant access to prefixes (e.g., 'prod/*') rather than individual secrets, simplifying permission management as your secret count grows.

Encryption and Security Model

AWS Secrets Manager implements multiple layers of security to protect secrets at rest and in transit. Understanding this security model is essential for designing compliant solutions.

Encryption Architecture:

Every secret is encrypted using envelope encryption with AWS KMS:

A unique data key is generated for each secret version
The data key encrypts the secret value
The data key itself is encrypted by your chosen KMS key
Both the encrypted data key and encrypted secret are stored together

This means that to decrypt a secret, the caller must have:

IAM permission to call Secrets Manager
KMS permission to decrypt with the associated KMS key

Encryption Flow

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    SECRET CREATION FLOW                         │
└─────────────────────────────────────────────────────────────────┘
 
  Application                  Secrets Manager              AWS KMS
      │                              │                          │
      │  CreateSecret(value)         │                          │
      │──────────────────────────────>│                          │
      │                              │  GenerateDataKey()       │
      │                              │─────────────────────────>│
      │                              │                          │
      │                              │  Return:                 │
      │                              │  - Plaintext data key    │
      │                              │  - Encrypted data key    │
      │                              │<─────────────────────────│
      │                              │                          │
      │                              │  Encrypt secret with     │
      │                              │  plaintext data key      │
      │                              │                          │
      │                              │  Discard plaintext key   │
      │                              │                          │
      │                              │  Store:                  │
      │                              │  [encrypted_data_key]    │
      │                              │  [encrypted_secret]      │
      │                              │                          │
      │  Success (ARN, VersionId)    │                          │
      │<─────────────────────────────│                          │
 
┌─────────────────────────────────────────────────────────────────┐
│                    SECRET RETRIEVAL FLOW                        │
└─────────────────────────────────────────────────────────────────┘
 
  Application                  Secrets Manager              AWS KMS
      │                              │                          │
      │  GetSecretValue(id)          │                          │
      │──────────────────────────────>│                          │
      │                              │  Decrypt(encrypted_key)  │
      │                              │─────────────────────────>│
      │                              │                          │
      │                              │  Return: plaintext key   │
      │                              │<─────────────────────────│
      │                              │                          │
      │                              │  Decrypt secret with     │
      │                              │  plaintext data key      │
      │                              │                          │
      │  Return: secret value        │                          │
      │<─────────────────────────────│                          │

Choosing KMS Keys:

You have three options for the KMS key used to encrypt secrets:

AWS Managed Key (aws/secretsmanager) — Default, free, automatic rotation, but limited cross-account capabilities
Customer Managed Key (CMK) — Full control over key policy, rotation schedule, and cross-account access
Customer Managed Key from Another Account — For centralized key management in multi-account setups

KMS Key Options Comparison
Aspect	AWS Managed Key	Customer Managed Key
Cost	Free	$1/month per key + API calls
Cross-Account Access	Not supported	Fully supported via key policy
Key Rotation Control	Automatic (annual)	Configurable (automatic or manual)
Key Policy Customization	Not allowed	Full control
CloudTrail Key Usage	Limited visibility	Full visibility
Compliance Requirements	May not meet strict requirements	Meets most compliance needs

Cross-Account Secrets Require CMK

If you need to share secrets across AWS accounts, you must use a customer managed KMS key. The AWS managed key cannot grant cross-account decrypt permissions. Plan your KMS strategy early if you have multi-account architectures.

Automatic Rotation - The Killer Feature

Automatic rotation is AWS Secrets Manager's most powerful feature, transforming credential rotation from a risky, manual process into an automated operation. Secrets Manager provides built-in rotation for AWS database services and supports custom rotation for any secret type.

Built-in Rotation Support:

Natively Supported Rotation
Service	Rotation Type	Mechanism
Amazon RDS (all engines)	Single user or alternating users	Lambda function manages DB users
Amazon Aurora	Single user or alternating users	Same as RDS
Amazon Redshift	Single user or alternating users	Lambda manages cluster users
Amazon DocumentDB	Single user or alternating users	Lambda manages DB users
Amazon ElastiCache Redis	Password rotation	Lambda updates replication group

Rotation Strategies:

Secrets Manager supports two rotation strategies for database credentials:

1. Single User Rotation:

The same user's password is changed in place
Simpler setup, but brief connection disruption possible
Suitable for applications with connection retry logic

2. Alternating Users Rotation:

Two users are created (e.g., user_a, user_b)
Rotation alternates between them
Zero-downtime rotation: one user always has valid credentials
Recommended for production workloads

Rotation Configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Create a secret for RDS database
aws secretsmanager create-secret \
    --name prod/myapp/db/primary \
    --secret-string '{"username":"admin","password":"InitialP@ssword123!"}' \
    --kms-key-id alias/myapp-secrets
 
# Configure automatic rotation (alternating users strategy)
aws secretsmanager rotate-secret \
    --secret-id prod/myapp/db/primary \
    --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSRotation \
    --rotation-rules "{
        "AutomaticallyAfterDays": 30,
        "ScheduleExpression": "cron(0 4 ? * SUN *)",
        "Duration": "2h"
    }"
 
# Trigger immediate rotation
aws secretsmanager rotate-secret \
    --secret-id prod/myapp/db/primary
 
# Check rotation status
aws secretsmanager describe-secret \
    --secret-id prod/myapp/db/primary \
    --query 'RotationEnabled, LastRotatedDate, NextRotationDate'

Custom Rotation with Lambda:

For secrets not natively supported, you implement rotation using a Lambda function. The function receives events for four rotation steps:

Custom Rotation Lambda
Python Handler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
import boto3
import json
import string
import secrets as secrets_lib
 
def lambda_handler(event, context):
    """
    Rotation Lambda that follows the four-step rotation protocol.
    This example rotates an API key for a third-party service.
    """
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']
    
    secrets_client = boto3.client('secretsmanager')
    
    if step == "createSecret":
        create_secret(secrets_client, arn, token)
    elif step == "setSecret":
        set_secret(secrets_client, arn, token)
    elif step == "testSecret":
        test_secret(secrets_client, arn, token)
    elif step == "finishSecret":
        finish_secret(secrets_client, arn, token)
    else:
        raise ValueError(f"Invalid step: {step}")
 
def create_secret(client, arn, token):
    """Step 1: Create new secret version in AWSPENDING stage."""
    # Get current secret
    current = client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")
    current_dict = json.loads(current['SecretString'])
    
    # Generate new API key
    new_api_key = ''.join(secrets_lib.choice(
        string.ascii_letters + string.digits
    ) for _ in range(64))
    
    # Create pending version with new key
    current_dict['api_key'] = new_api_key
    
    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps(current_dict),
        VersionStages=['AWSPENDING']
    )
 
def set_secret(client, arn, token):
    """Step 2: Set the new credentials in the target service."""
    pending = client.get_secret_value(
        SecretId=arn, 
        VersionId=token,
        VersionStage="AWSPENDING"
    )
    pending_dict = json.loads(pending['SecretString'])
    
    # Call external API to register new key
    # This is service-specific
    response = register_api_key_with_service(
        pending_dict['api_key'],
        pending_dict['client_id']
    )
    
    if not response.get('success'):
        raise Exception("Failed to register new API key")
 
def test_secret(client, arn, token):
    """Step 3: Test that the new credentials work."""
    pending = client.get_secret_value(
        SecretId=arn,
        VersionId=token,
        VersionStage="AWSPENDING"
    )
    pending_dict = json.loads(pending['SecretString'])
    
    # Test the new API key
    if not test_api_key(pending_dict['api_key']):
        raise Exception("New API key validation failed")
 
def finish_secret(client, arn, token):
    """Step 4: Move AWSPENDING to AWSCURRENT."""
    # Get current version
    metadata = client.describe_secret(SecretId=arn)
    current_version = None
    
    for version_id, stages in metadata['VersionIdsToStages'].items():
        if 'AWSCURRENT' in stages:
            current_version = version_id
            break
    
    # Move staging labels
    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage='AWSCURRENT',
        MoveToVersionId=token,
        RemoveFromVersionId=current_version
    )
    
    # Optionally revoke old API key
    # revoke_old_api_key(old_key)

Rotation Lambda Best Practices

Rotation Lambdas should be idempotent—they may be retried if they fail. Deploy them in the same VPC as your databases for network access. Set appropriate timeouts (default 30 seconds may be too short). Monitor Lambda errors and set up alerting for rotation failures.

Access Control Patterns

AWS Secrets Manager access control combines IAM policies with resource-based policies to provide fine-grained permissions. Understanding these patterns is essential for implementing least-privilege access.

Access Control Layers:

IAM Identity Policies — Attached to users, roles, or groups
Resource-Based Policies — Attached to individual secrets
KMS Key Policies — Control encryption/decryption access
VPC Endpoint Policies — Limit access through private endpoints

IAM Policy Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Read-only access to production secrets
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadProductionSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:VersionStage": "AWSCURRENT"
        }
      }
    },
    {
      "Sid": "ListSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:ListSecrets",
      "Resource": "*"
    },
    {
      "Sid": "DecryptWithKMS",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}
 
// Full management for secret administrators
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FullSecretsAccess",
      "Effect": "Allow",
      "Action": "secretsmanager:*",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:*"
    },
    {
      "Sid": "KMSAccess",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/*"
    }
  ]
}

Common Access Patterns:

Access Pattern Recommendations
Scenario	Pattern	Implementation
Lambda function needs secret	IAM role attached to Lambda	Add GetSecretValue + KMS Decrypt to Lambda execution role
ECS task needs secret	Task role + native integration	Reference secret ARN in task definition; ECS injects at runtime
EC2 application needs secret	Instance profile role	Grant role access; use SDK with instance metadata credentials
Cross-account access	Resource policy + KMS policy	Both secret and KMS key must allow the external account
Developer access for debugging	IAM group + MFA requirement	Time-limited access with assume role and MFA condition
CI/CD pipeline access	OIDC federation or IAM user	Short-lived credentials via OIDC preferred; rotate IAM keys

Cross-Account Access Requires Both Policies

For cross-account secret access, you must configure BOTH: (1) a resource-based policy on the secret allowing the external principal, AND (2) a key policy on the KMS key granting kms:Decrypt to the external principal. Missing either will result in access denied errors.

AWS Service Integrations

One of Secrets Manager's greatest strengths is deep integration with AWS services. These integrations enable secrets injection without application code changes.

Native Service Integrations:

Service Integration Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "family": "my-app",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
      "secrets": [
        {
          "name": "DB_USERNAME",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db:username::"
        },
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db:password::"
        },
        {
          "name": "API_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/api-key"
        }
      ],
      "environment": [
        {
          "name": "DB_HOST",
          "value": "mydb.cluster-abc123.us-east-1.rds.amazonaws.com"
        }
      ]
    }
  ],
  "taskRoleArn": "arn:aws:iam::123456789012:role/MyAppTaskRole",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
}
 
// The execution role needs:
// - secretsmanager:GetSecretValue
// - kms:Decrypt (if using CMK)

Use the Lambda Extension for Caching

For Lambda functions that frequently access secrets, use the AWS Parameters and Secrets Lambda Extension. It provides local caching with configurable TTL, reducing API calls, latency, and costs. The extension runs as a sidecar process and serves secrets over localhost HTTP.

Cost Optimization and Pricing

AWS Secrets Manager pricing is straightforward but can accumulate in high-scale environments. Understanding the cost model helps you optimize usage.

Current Pricing (as of 2024):

Secrets Manager Pricing Components
Component	Price	Notes
Secret storage	$0.40/secret/month	Prorated; minimum 7 days per secret
API calls	$0.05 per 10,000 calls	GetSecretValue, DescribeSecret, etc.
Rotation Lambda	Standard Lambda pricing	Duration + invocation charges
KMS (if using CMK)	$1/key/month + $0.03/10K requests	Only if using customer managed keys

Cost Optimization Strategies:

Reducing Secrets Manager Costs

•Consolidate Related Secrets — Store related credentials (e.g., DB username, password, host) in a single secret as JSON rather than separate secrets. This reduces the $0.40/secret/month charge.
•Implement Client-Side Caching — Use the Lambda extension or SDK caching to minimize GetSecretValue API calls. Cache secrets in application memory with appropriate TTL.
•Use AWS Managed KMS Key — Unless you need cross-account access or specific compliance, use the free aws/secretsmanager key instead of a CMK.
•Clean Up Unused Secrets — Secrets continue to incur charges until deleted. Implement tagging and lifecycle policies to identify and remove unused secrets.
•Consider SSM Parameter Store for Non-Sensitive Config — Standard SSM parameters are free (up to 10,000). Use Secrets Manager only for actual secrets; use Parameter Store for configuration.

Secrets Manager vs Parameter Store

For simple secrets without rotation needs, SSM Parameter Store SecureString is an alternative at no per-secret charge. However, it lacks automatic rotation, cross-region replication, and fine-grained resource policies. Use Secrets Manager for database credentials and secrets requiring rotation.

Multi-Region Cost Consideration

Replica secrets (for cross-region replication) are charged at the same $0.40/month rate. A secret replicated to 3 regions costs $1.20/month total. Budget accordingly for disaster recovery configurations.

Monitoring and Troubleshooting

Effective monitoring ensures your secrets management implementation remains secure and operational. AWS provides comprehensive observability through CloudTrail, CloudWatch, and EventBridge.

Monitoring Setup
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# CloudFormation for secrets monitoring
Resources:
  # Alarm for rotation failures
  RotationFailureAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: SecretsManager-RotationFailure
      AlarmDescription: Alert when secret rotation fails
      MetricName: RotationFailed
      Namespace: AWS/SecretsManager
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref AlertSNSTopic
      Dimensions:
        - Name: SecretName
          Value: prod/myapp/db
 
  # EventBridge rule for secret access
  SecretAccessRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Capture all secret access events
      EventPattern:
        source:
          - aws.secretsmanager
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          eventName:
            - GetSecretValue
            - DeleteSecret
            - PutSecretValue
      State: ENABLED
      Targets:
        - Arn: !GetAtt SecretAccessLogGroup.Arn
          Id: LogSecretAccess
 
  # Alert on suspicious access patterns
  UnauthorizedAccessRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Alert on unauthorized secret access attempts
      EventPattern:
        source:
          - aws.secretsmanager
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          errorCode:
            - AccessDenied
            - UnauthorizedAccess
      Targets:
        - Arn: !Ref SecurityAlertSNS
          Id: SecurityAlert

Common Troubleshooting Scenarios:

Troubleshooting Guide
Issue	Likely Cause	Resolution
AccessDenied on GetSecretValue	Missing IAM or KMS permission	Check both secretsmanager and kms:Decrypt permissions
Rotation marked as failed	Lambda timeout or network issue	Check Lambda logs; verify VPC connectivity to RDS
Application sees old credentials	Caching stale secrets	Reduce cache TTL; force cache refresh after rotation
Cross-account access fails	Missing resource policy or KMS policy	Add both secret resource policy AND KMS key policy
Slow secret retrieval	No VPC endpoint configured	Create secretsmanager VPC endpoint for private access

Rotation Failure Recovery

If rotation fails mid-process, the secret may have an AWSPENDING version that's stuck. To recover: (1) Check Lambda logs for the failure cause, (2) Fix the underlying issue, (3) Trigger rotation again, or (4) Manually finalize by moving version stages using the CLI. Never delete the AWSPENDING version directly.

Summary: AWS Secrets Manager Mastery

AWS Secrets Manager provides a fully managed secrets solution deeply integrated with the AWS ecosystem. Let's consolidate the key concepts:

Key Takeaways

•Fully managed service — No infrastructure to run; automatic scaling, HA, and backups included
•Native automatic rotation — Built-in rotation for AWS databases; Lambda-based custom rotation for any secret
•Envelope encryption with KMS — Secrets encrypted with data keys protected by KMS; use CMK for cross-account access
•Fine-grained access control — IAM identity policies, resource-based policies, and KMS policies work together
•Deep AWS integration — Native support in RDS, ECS, Lambda, EKS with automatic secret injection
•Cost model — $0.40/secret/month + API calls; optimize with caching and secret consolidation
•Comprehensive audit trail — CloudTrail logs every access; EventBridge enables real-time monitoring and alerting

What's Next:

Having explored AWS Secrets Manager, the next page examines Azure Key Vault—Microsoft's comprehensive secrets, keys, and certificate management service. You'll learn how Azure's approach differs, its unique features for enterprise environments, and when to choose it for hybrid or Azure-centric workloads.

Page Complete

You now have a comprehensive understanding of AWS Secrets Manager's capabilities, integration patterns, and operational considerations. This knowledge enables you to design secure, scalable secrets management for AWS-native workloads.