System Design (HLD)Fallacies of Distributed Computing

Fallacies of Distributed Computing

LevelIntermediate

Duration90 mins

TopicFallacies of Distributed Computing

4 / 5

The Network Is Secure (It's Not)

Your Traffic Is Being Watched

The first three fallacies concern reliability, latency, and capacity. The fourth fallacy is fundamentally different—it concerns trust: The network is secure.

This assumption is perhaps the most dangerous of all. When you send a request from Service A to Service B, you might assume:

Only Service B receives the request
The request arrives unmodified
Service B is actually who it claims to be
No one else can observe the traffic

In reality, every packet traverses infrastructure operated by others—routers, switches, ISPs, cloud providers, potential attackers. Without explicit cryptographic protection, none of those assumptions hold.

The Real Threat Landscape

Network traffic is intercepted constantly. Nation-states tap undersea cables. Cloud providers have access to your unencrypted traffic. Coffee shop WiFi can be trivially eavesdropped. BGP hijacking redirects traffic through attacker-controlled infrastructure. Assuming the network is secure in 2024 is professional negligence.

This page examines the network security threat model, cryptographic fundamentals for distributed systems, and the architectural patterns needed to build systems that assume the network is hostile.

The Network Threat Model

To design secure systems, you must understand what threats exist. Network security threats fall into several categories based on the attacker's capabilities and goals.

Network Attack Classification
Attack Type	Description	Example	Countermeasure
Eavesdropping	Passive observation of traffic	Capturing passwords over HTTP	Encryption (TLS)
Man-in-the-Middle	Active interception and modification	Injecting malicious JavaScript	Mutual TLS, Certificate Pinning
Replay Attack	Capturing and resending valid requests	Re-executing a funds transfer	Timestamps, Nonces, Idempotency
Spoofing	Impersonating a legitimate party	Fake DNS responses	DNSSEC, Certificate validation
Denial of Service	Overwhelming network resources	SYN flood, DDoS	Rate limiting, CDN, DDoS protection
BGP Hijacking	Redirecting traffic via routing manipulation	Traffic re-routed through attacker AS	BGP validation, path monitoring

Attacker capabilities vary widely:

Passive Attacker (Reconnaissance):

Can observe traffic on networks they control or access (WiFi, compromised router)
Cannot modify traffic in transit
Goal: Gather credentials, sensitive data, or information for later attacks

Active Attacker (Manipulation):

Can intercept and modify packets
Can forge packets from arbitrary sources
Goal: Steal data, inject malicious content, or disrupt services

Sophisticated Attacker (Nation-state, APT):

May have access to backbone infrastructure
Can perform large-scale man-in-the-middle attacks
May have advanced cryptanalysis capabilities
Can execute multi-stage, long-term campaigns

The Insider Threat

Don't forget threats from within your own organization. Employees, contractors, and cloud provider staff may have access to internal networks. Zero Trust Architecture assumes that even internal network traffic is hostile—a philosophy increasingly necessary in modern environments.

Cryptographic Foundations for Network Security

Security in distributed systems is built on cryptographic primitives. Understanding these foundations is essential for making informed security decisions.

Core Cryptographic Concepts

•Symmetric Encryption — Same key encrypts and decrypts. Fast (AES-256 can process 1GB/s on modern CPUs). Problem: How do you share the key securely?
•Asymmetric Encryption — Public key encrypts, private key decrypts. Slower (~1000x) but solves key distribution. Used to establish symmetric keys.
•Digital Signatures — Private key signs, public key verifies. Provides non-repudiation—prove a message came from a specific sender.
•Hash Functions — One-way transformation producing fixed-size digest. Verify data integrity without revealing content (SHA-256, SHA-3).
•Message Authentication Codes (MAC) — Hash with a secret key. Verifies both integrity and authenticity (HMAC).
•Key Exchange — Protocols to establish shared secrets over insecure channels. Diffie-Hellman, ECDHE for perfect forward secrecy.

TLS (Transport Layer Security) combines these primitives:

Handshake Phase:
- Client sends supported cipher suites
- Server presents certificate (public key + identity verification)
- Client verifies certificate chain to trusted CA
- Key exchange (usually ECDHE) establishes session keys
- Optional: mutual TLS (mTLS) for client certificate verification
Data Transfer Phase:
- All data encrypted with negotiated symmetric cipher (e.g., AES-256-GCM)
- Messages authenticated with HMAC or AEAD
- Sequence numbers prevent replay attacks

Why TLS everywhere?

TLS provides confidentiality (can't read), integrity (can't modify), and authentication (verify identity). These three properties are the minimum requirements for secure network communication.

Perfect Forward Secrecy

Modern TLS configurations use ephemeral key exchange (ECDHE). Even if an attacker captures encrypted traffic today and later obtains the server's private key, they cannot decrypt the captured traffic because session keys were never transmitted. This is Perfect Forward Secrecy—always enable it.

Authentication in Distributed Systems

Encryption keeps traffic private, but how do you know you're talking to the right service? Authentication—verifying identity—is equally critical in distributed systems.

Service Authentication Patterns

•API Keys — Simple shared secrets. Easy to implement but hard to rotate, no mutual auth, often logged in URLs. Minimal security.
•OAuth 2.0 / JWT — Token-based auth with scopes. Standard for user authentication flows. Beware JWT pitfalls (algorithm confusion, no revocation).
•Mutual TLS (mTLS) — Both client and server present certificates. Strong authentication, automatic rotation via cert lifecycle, but complex to manage.
•Service Mesh Identity — SPIFFE/SPIRE provides workload identity. Each service gets a cryptographic identity regardless of network location.
•Cloud IAM — AWS IAM roles, GCP service accounts, Azure managed identities. Identity tied to cloud resources, automatic credential rotation.

API Key Vulnerabilities

•Keys often hardcoded or in config files
•Leaked in logs, URLs, browser history
•No expiration or rotation
•One key compromised = persistent access
•No way to know who used a shared key
•Single factor authentication only

mTLS Advantages

•Cryptographic identity verification
•Short-lived certificates auto-rotate
•Can't be accidentally logged or leaked
•Compromised cert expires quickly
•Full audit trail of certificate usage
•Works at network layer, app-agnostic

Zero Trust Architecture

Zero Trust means 'never trust, always verify.' Every request is authenticated and authorized, regardless of network location. In Zero Trust, there's no 'internal' vs 'external' network—all traffic is treated as potentially hostile. Google's BeyondCorp and similar architectures embody this principle.

Common Security Vulnerabilities in Distributed Systems

Beyond the network layer, distributed systems introduce unique security challenges. Understanding these vulnerabilities helps you avoid building them into your designs.

Critical Distributed System Vulnerabilities

•Insecure Service-to-Service Communication — Services communicating over HTTP within 'private' networks. Once an attacker gets internal access, everything is exposed.
•Secrets in Configuration — Database passwords, API keys in environment variables, config files, or worse—source code. A single container escape exposes all secrets.
•JWT Token Vulnerabilities — Algorithm confusion (accepting 'none'), no expiration, missing signature validation, sensitive data in payload (it's only Base64, not encrypted).
•SSRF (Server-Side Request Forgery) — Attacker tricks your server into making requests to internal services. Cloud metadata endpoints (169.254.169.254) are prime targets.
•Deserialization Attacks — Trusting serialized data from untrusted sources. Deserializing attacker-controlled objects can lead to remote code execution.
•Message Queue Poisoning — Injecting malicious messages into queues. Workers that trust queue content can be exploited.
•Dependency Confusion — Attacker publishes malicious package with internal package name to public registry. Build systems download the malicious version.
•Insufficient Authorization — Service authenticates but doesn't authorize. Knowing the URL means access—no role or permission checking.

ssrf-vulnerability.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
/**
 * SSRF Vulnerability Example
 * DO NOT USE - This code is vulnerable!
 */
 
// ❌ VULNERABLE: User controls the URL
app.get('/proxy', async (req, res) => {
    const url = req.query.url as string;
    
    // Attacker provides: ?url=http://169.254.169.254/latest/meta-data/
    // Server fetches AWS instance credentials!
    
    const response = await fetch(url); // DANGER!
    res.send(await response.text());
});
 
// ❌ VULNERABLE: User controls hostname
app.get('/image', async (req, res) => {
    const imageId = req.query.id as string;
    
    // Attacker provides: ?id=../../../internal-service/admin
    // Path traversal + SSRF!
    
    const url = `http://image-service/images/${imageId}`;
    const response = await fetch(url);
    res.send(await response.arrayBuffer());
});
 
// ✅ SECURE: Validate and restrict
app.get('/proxy-safe', async (req, res) => {
    const url = req.query.url as string;
    
    // Parse and validate URL
    const parsed = new URL(url);
    
    // Allowlist check
    const allowedHosts = ['api.trusted.com', 'cdn.trusted.com'];
    if (!allowedHosts.includes(parsed.hostname)) {
        return res.status(400).json({ error: 'Host not allowed' });
    }
    
    // Block internal IPs
    const blocked = ['localhost', '127.0.0.1', '169.254.169.254', '10.', '192.168.'];
    if (blocked.some(b => parsed.hostname.includes(b))) {
        return res.status(400).json({ error: 'Internal hosts not allowed' });
    }
    
    // Additional checks: protocol, port, etc.
    if (parsed.protocol !== 'https:') {
        return res.status(400).json({ error: 'HTTPS required' });
    }
    
    const response = await fetch(url);
    res.send(await response.text());
});

Cloud Metadata Endpoints

AWS/GCP/Azure all expose metadata at 169.254.169.254. If an attacker can make your service fetch this URL (via SSRF), they get IAM credentials, instance identity, and network configuration. AWS IMDSv2 requires session tokens to mitigate this, but many services still run IMDSv1.

Secrets Management in Distributed Systems

Distributed systems need credentials—database passwords, API keys, encryption keys, certificates. How you manage these secrets is critical to system security.

Secrets Anti-Patterns

•Hardcoded in source code — Permanent exposure via Git history, even if removed later
•Plain text config files — Exposed via backup, logging, or container inspection
•Environment variables — Visible in process listings, crash dumps, and logging
•Shared service accounts — Single compromise affects all services
•Never rotated — Old credentials remain valid after employee departure
•Manual distribution — Secrets shared via Slack, email, or shared drives

Secrets Management Best Practices

•Use a secrets manager — HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault. Centralized, audited, rotatable.
•Dynamic secrets — Generate short-lived credentials on demand. Vault can create temporary database users that auto-expire.
•Encryption at rest — Secrets stored encrypted with HSM-backed keys. Even storage compromise doesn't expose secrets.
•Rotation automation — Secrets should rotate regularly without manual intervention. Design systems to handle rotation seamlessly.
•Principle of least privilege — Each service gets only the secrets it needs. Granular access control prevents lateral movement.
•Audit logging — Every secret access logged with who, what, when, from where. Detect and investigate anomalous access.

vault-integration.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
/**
 * Example: HashiCorp Vault Integration
 * Demonstrates secure secrets retrieval with automatic rotation
 */
 
import Vault from 'node-vault';
 
class SecureSecretsManager {
    private vault: ReturnType<typeof Vault>;
    private cache = new Map<string, { value: string; expiresAt: number }>();
    
    constructor() {
        // Authenticate with Vault using workload identity
        // (IAM role, Kubernetes service account, etc.)
        this.vault = Vault({
            endpoint: process.env.VAULT_ADDR,
            // Kubernetes auth - no hardcoded token!
            token: undefined
        });
    }
 
    async initialize(): Promise<void> {
        // Authenticate using Kubernetes service account
        const k8sJwt = await this.getKubernetesServiceAccountToken();
        
        const authResult = await this.vault.kubernetesLogin({
            role: 'my-service-role',
            jwt: k8sJwt
        });
        
        this.vault.token = authResult.auth.client_token;
    }
 
    async getDatabaseCredentials(): Promise<{
        username: string;
        password: string;
    }> {
        // Dynamic secrets: Vault generates unique, short-lived credentials
        const leaseData = await this.vault.read('database/creds/my-role');
        
        const credentials = {
            username: leaseData.data.username,
            password: leaseData.data.password
        };
        
        // Schedule lease renewal before expiry
        const leaseDuration = leaseData.lease_duration;
        setTimeout(
            () => this.renewLease(leaseData.lease_id),
            (leaseDuration - 60) * 1000 // Renew 1 minute before expiry
        );
        
        return credentials;
    }
 
    async getSecret(path: string): Promise<string> {
        // Check cache first
        const cached = this.cache.get(path);
        if (cached && cached.expiresAt > Date.now()) {
            return cached.value;
        }
        
        // Fetch from Vault
        const result = await this.vault.read(`secret/data/${path}`);
        const value = result.data.data.value;
        
        // Cache with TTL
        this.cache.set(path, {
            value,
            expiresAt: Date.now() + 60000 // 1 minute cache
        });
        
        return value;
    }
 
    private async getKubernetesServiceAccountToken(): Promise<string> {
        const fs = await import('fs/promises');
        return fs.readFile(
            '/var/run/secrets/kubernetes.io/serviceaccount/token',
            'utf-8'
        );
    }
    
    private async renewLease(leaseId: string): Promise<void> {
        try {
            await this.vault.leaseRenew(leaseId);
        } catch (error) {
            console.error('Lease renewal failed, fetching new credentials');
            // Application should handle credential refresh
        }
    }
}

The Twelve-Factor App on Config

The Twelve-Factor App recommends storing config in environment variables. This is a starting point, not the final answer. For secrets, environment variables are insufficient—they're exposed in too many ways. Use proper secrets managers for sensitive credentials.

Defense in Depth

No single security measure is sufficient. Security requires defense in depth—multiple layers of protection so that if one layer fails, others still protect the system.

Defense in Depth Layers
Layer	Controls	Examples
Network	Firewalls, segmentation, DDoS protection	VPC, Security Groups, Cloudflare
Transport	Encryption in transit, mTLS	TLS 1.3, Certificate verification
Application	Input validation, parameterized queries	WAF rules, OWASP recommendations
Data	Encryption at rest, access controls	AES-256, Column-level encryption
Identity	Strong auth, MFA, least privilege	SAML, OIDC, RBAC
Monitoring	Anomaly detection, audit logging	SIEM, IDS/IPS, CloudTrail
Recovery	Backups, disaster recovery	Point-in-time recovery, DR sites

The Castle vs. Airport Security Model:

Castle Model (Traditional Perimeter Security):

Hard shell, soft center
Strong firewall at the edge
Internal network trusted
Problem: Once inside, attackers move freely

Airport Model (Zero Trust):

Security everywhere
Every person verified at every stage
No trusted zones
Lateral movement restricted
Principle: Assume breach has occurred

Modern distributed systems should adopt the Airport Model. With cloud, containers, and remote work, there is no "perimeter" to defend.

Security as Code

Security policies should be version-controlled, reviewable, and automatically enforced. Tools like Open Policy Agent (OPA), Sentinel, and cloud security posture management (CSPM) enable policy-as-code. Security shouldn't depend on manual processes that can be skipped.

Security Monitoring and Incident Response

Prevention alone isn't enough. You must assume breaches will occur and prepare to detect and respond to them quickly.

Essential Security Monitoring

•Centralized Logging — All systems ship logs to a central, tamper-evident store. Attackers can't cover tracks by deleting local logs.
•Authentication Events — Log all logins, failures, privilege escalations. Detect password spraying, brute force, credential stuffing.
•Authorization Failures — Track access denied events. Indicates reconnaissance or misconfiguration.
•Anomaly Detection — Baseline normal behavior, alert on deviations. Unusual access patterns, geographic anomalies, time-based patterns.
•Network Flows — Log connections between services. Detect unexpected lateral movement.
•Secrets Access — Audit every secret retrieval. Alert on bulk access or unusual patterns.
•Data Egress — Monitor outbound traffic volumes. Large data exfiltration attempts should trigger alerts.

Incident Response Framework:

Preparation — Runbooks, communication channels, escalation paths, practice drills
Identification — Detect and confirm the incident. Is this a false positive or real threat?
Containment — Limit damage. Isolate affected systems without destroying evidence.
Eradication — Remove the threat. Patch vulnerabilities, remove backdoors.
Recovery — Restore systems to normal operation with confidence they're clean.
Lessons Learned — Post-incident review. What failed? What worked? Update defenses.

Key Metrics:

MTTD (Mean Time to Detect) — How long until you notice a breach?
MTTR (Mean Time to Respond) — How long from detection to containment?

Industry average for breach detection is 200+ days. Best-in-class is under an hour. The difference is preparation and monitoring.

Assume You're Already Compromised

This isn't paranoia—it's realistic. Many breaches are discovered when attackers announce them, not when defenders detect them. Design your monitoring assuming attackers are already inside, and your job is to find them.

Summary: The Network Is Never Secure

We've explored the fourth fallacy of distributed computing: the assumption that the network is secure. Let's consolidate the key insights:

Key Takeaways

•Assume the network is hostile — Every packet traverses infrastructure you don't control. Encryption and authentication are mandatory, not optional.
•TLS everywhere — All service communication should use TLS. Internal 'private' networks are not secure. Perfect forward secrecy protects past traffic.
•Strong authentication matters — Move beyond API keys to mTLS, service mesh identity, or cloud IAM. Cryptographic identity verification beats shared secrets.
•Zero Trust Architecture — Never trust, always verify. No privileged internal networks. Every request authenticated and authorized.
•Secrets management is critical — Use proper secrets managers with dynamic secrets, automatic rotation, and audit logging.
•Defense in depth — Multiple security layers ensure single failures don't compromise the system. Assume breach has occurred.
•Detection and response matter — Prevention will fail. Prepare to detect and respond to incidents quickly.

What's next:

We've covered the four most commonly discussed fallacies: network reliability, latency, bandwidth, and security. The next page examines all eight fallacies together, including topology, administrator, transport cost, and network homogeneity—completing your understanding of distributed computing's fundamental challenges.

Page Complete

You now understand why assuming network security leads to devastating breaches. The patterns you've learned—encryption everywhere, strong authentication, secrets management, Zero Trust architecture, and defense in depth—are essential for building secure distributed systems.

4 / 5

Loading learning content...

System Design (HLD)Fallacies of Distributed Computing

Fallacies of Distributed Computing

LevelIntermediate

Duration90 mins

TopicFallacies of Distributed Computing

4 / 5

The Network Is Secure (It's Not)

Your Traffic Is Being Watched

The first three fallacies concern reliability, latency, and capacity. The fourth fallacy is fundamentally different—it concerns trust: The network is secure.

This assumption is perhaps the most dangerous of all. When you send a request from Service A to Service B, you might assume:

Only Service B receives the request
The request arrives unmodified
Service B is actually who it claims to be
No one else can observe the traffic

The Real Threat Landscape

This page examines the network security threat model, cryptographic fundamentals for distributed systems, and the architectural patterns needed to build systems that assume the network is hostile.

The Network Threat Model

To design secure systems, you must understand what threats exist. Network security threats fall into several categories based on the attacker's capabilities and goals.

Network Attack Classification
Attack Type	Description	Example	Countermeasure
Eavesdropping	Passive observation of traffic	Capturing passwords over HTTP	Encryption (TLS)
Man-in-the-Middle	Active interception and modification	Injecting malicious JavaScript	Mutual TLS, Certificate Pinning
Replay Attack	Capturing and resending valid requests	Re-executing a funds transfer	Timestamps, Nonces, Idempotency
Spoofing	Impersonating a legitimate party	Fake DNS responses	DNSSEC, Certificate validation
Denial of Service	Overwhelming network resources	SYN flood, DDoS	Rate limiting, CDN, DDoS protection
BGP Hijacking	Redirecting traffic via routing manipulation	Traffic re-routed through attacker AS	BGP validation, path monitoring

Attacker capabilities vary widely:

Passive Attacker (Reconnaissance):

Can observe traffic on networks they control or access (WiFi, compromised router)
Cannot modify traffic in transit
Goal: Gather credentials, sensitive data, or information for later attacks

Active Attacker (Manipulation):

Can intercept and modify packets
Can forge packets from arbitrary sources
Goal: Steal data, inject malicious content, or disrupt services

Sophisticated Attacker (Nation-state, APT):

May have access to backbone infrastructure
Can perform large-scale man-in-the-middle attacks
May have advanced cryptanalysis capabilities
Can execute multi-stage, long-term campaigns

The Insider Threat

Cryptographic Foundations for Network Security

Security in distributed systems is built on cryptographic primitives. Understanding these foundations is essential for making informed security decisions.

Core Cryptographic Concepts

•Symmetric Encryption — Same key encrypts and decrypts. Fast (AES-256 can process 1GB/s on modern CPUs). Problem: How do you share the key securely?
•Asymmetric Encryption — Public key encrypts, private key decrypts. Slower (~1000x) but solves key distribution. Used to establish symmetric keys.
•Digital Signatures — Private key signs, public key verifies. Provides non-repudiation—prove a message came from a specific sender.
•Hash Functions — One-way transformation producing fixed-size digest. Verify data integrity without revealing content (SHA-256, SHA-3).
•Message Authentication Codes (MAC) — Hash with a secret key. Verifies both integrity and authenticity (HMAC).
•Key Exchange — Protocols to establish shared secrets over insecure channels. Diffie-Hellman, ECDHE for perfect forward secrecy.

TLS (Transport Layer Security) combines these primitives:

Handshake Phase:
- Client sends supported cipher suites
- Server presents certificate (public key + identity verification)
- Client verifies certificate chain to trusted CA
- Key exchange (usually ECDHE) establishes session keys
- Optional: mutual TLS (mTLS) for client certificate verification
Data Transfer Phase:
- All data encrypted with negotiated symmetric cipher (e.g., AES-256-GCM)
- Messages authenticated with HMAC or AEAD
- Sequence numbers prevent replay attacks

Why TLS everywhere?

TLS provides confidentiality (can't read), integrity (can't modify), and authentication (verify identity). These three properties are the minimum requirements for secure network communication.

Perfect Forward Secrecy

Authentication in Distributed Systems

Encryption keeps traffic private, but how do you know you're talking to the right service? Authentication—verifying identity—is equally critical in distributed systems.

Service Authentication Patterns

•API Keys — Simple shared secrets. Easy to implement but hard to rotate, no mutual auth, often logged in URLs. Minimal security.
•OAuth 2.0 / JWT — Token-based auth with scopes. Standard for user authentication flows. Beware JWT pitfalls (algorithm confusion, no revocation).
•Mutual TLS (mTLS) — Both client and server present certificates. Strong authentication, automatic rotation via cert lifecycle, but complex to manage.
•Service Mesh Identity — SPIFFE/SPIRE provides workload identity. Each service gets a cryptographic identity regardless of network location.
•Cloud IAM — AWS IAM roles, GCP service accounts, Azure managed identities. Identity tied to cloud resources, automatic credential rotation.

API Key Vulnerabilities

•Keys often hardcoded or in config files
•Leaked in logs, URLs, browser history
•No expiration or rotation
•One key compromised = persistent access
•No way to know who used a shared key
•Single factor authentication only

mTLS Advantages

•Cryptographic identity verification
•Short-lived certificates auto-rotate
•Can't be accidentally logged or leaked
•Compromised cert expires quickly
•Full audit trail of certificate usage
•Works at network layer, app-agnostic

Zero Trust Architecture

Common Security Vulnerabilities in Distributed Systems

Beyond the network layer, distributed systems introduce unique security challenges. Understanding these vulnerabilities helps you avoid building them into your designs.

Critical Distributed System Vulnerabilities

•Insecure Service-to-Service Communication — Services communicating over HTTP within 'private' networks. Once an attacker gets internal access, everything is exposed.
•Secrets in Configuration — Database passwords, API keys in environment variables, config files, or worse—source code. A single container escape exposes all secrets.
•JWT Token Vulnerabilities — Algorithm confusion (accepting 'none'), no expiration, missing signature validation, sensitive data in payload (it's only Base64, not encrypted).
•SSRF (Server-Side Request Forgery) — Attacker tricks your server into making requests to internal services. Cloud metadata endpoints (169.254.169.254) are prime targets.
•Deserialization Attacks — Trusting serialized data from untrusted sources. Deserializing attacker-controlled objects can lead to remote code execution.
•Message Queue Poisoning — Injecting malicious messages into queues. Workers that trust queue content can be exploited.
•Dependency Confusion — Attacker publishes malicious package with internal package name to public registry. Build systems download the malicious version.
•Insufficient Authorization — Service authenticates but doesn't authorize. Knowing the URL means access—no role or permission checking.

ssrf-vulnerability.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
/**
 * SSRF Vulnerability Example
 * DO NOT USE - This code is vulnerable!
 */
 
// ❌ VULNERABLE: User controls the URL
app.get('/proxy', async (req, res) => {
    const url = req.query.url as string;
    
    // Attacker provides: ?url=http://169.254.169.254/latest/meta-data/
    // Server fetches AWS instance credentials!
    
    const response = await fetch(url); // DANGER!
    res.send(await response.text());
});
 
// ❌ VULNERABLE: User controls hostname
app.get('/image', async (req, res) => {
    const imageId = req.query.id as string;
    
    // Attacker provides: ?id=../../../internal-service/admin
    // Path traversal + SSRF!
    
    const url = `http://image-service/images/${imageId}`;
    const response = await fetch(url);
    res.send(await response.arrayBuffer());
});
 
// ✅ SECURE: Validate and restrict
app.get('/proxy-safe', async (req, res) => {
    const url = req.query.url as string;
    
    // Parse and validate URL
    const parsed = new URL(url);
    
    // Allowlist check
    const allowedHosts = ['api.trusted.com', 'cdn.trusted.com'];
    if (!allowedHosts.includes(parsed.hostname)) {
        return res.status(400).json({ error: 'Host not allowed' });
    }
    
    // Block internal IPs
    const blocked = ['localhost', '127.0.0.1', '169.254.169.254', '10.', '192.168.'];
    if (blocked.some(b => parsed.hostname.includes(b))) {
        return res.status(400).json({ error: 'Internal hosts not allowed' });
    }
    
    // Additional checks: protocol, port, etc.
    if (parsed.protocol !== 'https:') {
        return res.status(400).json({ error: 'HTTPS required' });
    }
    
    const response = await fetch(url);
    res.send(await response.text());
});

Cloud Metadata Endpoints

Secrets Management in Distributed Systems

Distributed systems need credentials—database passwords, API keys, encryption keys, certificates. How you manage these secrets is critical to system security.

Secrets Anti-Patterns

•Hardcoded in source code — Permanent exposure via Git history, even if removed later
•Plain text config files — Exposed via backup, logging, or container inspection
•Environment variables — Visible in process listings, crash dumps, and logging
•Shared service accounts — Single compromise affects all services
•Never rotated — Old credentials remain valid after employee departure
•Manual distribution — Secrets shared via Slack, email, or shared drives

Secrets Management Best Practices

•Use a secrets manager — HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault. Centralized, audited, rotatable.
•Dynamic secrets — Generate short-lived credentials on demand. Vault can create temporary database users that auto-expire.
•Encryption at rest — Secrets stored encrypted with HSM-backed keys. Even storage compromise doesn't expose secrets.
•Rotation automation — Secrets should rotate regularly without manual intervention. Design systems to handle rotation seamlessly.
•Principle of least privilege — Each service gets only the secrets it needs. Granular access control prevents lateral movement.
•Audit logging — Every secret access logged with who, what, when, from where. Detect and investigate anomalous access.

vault-integration.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
/**
 * Example: HashiCorp Vault Integration
 * Demonstrates secure secrets retrieval with automatic rotation
 */
 
import Vault from 'node-vault';
 
class SecureSecretsManager {
    private vault: ReturnType<typeof Vault>;
    private cache = new Map<string, { value: string; expiresAt: number }>();
    
    constructor() {
        // Authenticate with Vault using workload identity
        // (IAM role, Kubernetes service account, etc.)
        this.vault = Vault({
            endpoint: process.env.VAULT_ADDR,
            // Kubernetes auth - no hardcoded token!
            token: undefined
        });
    }
 
    async initialize(): Promise<void> {
        // Authenticate using Kubernetes service account
        const k8sJwt = await this.getKubernetesServiceAccountToken();
        
        const authResult = await this.vault.kubernetesLogin({
            role: 'my-service-role',
            jwt: k8sJwt
        });
        
        this.vault.token = authResult.auth.client_token;
    }
 
    async getDatabaseCredentials(): Promise<{
        username: string;
        password: string;
    }> {
        // Dynamic secrets: Vault generates unique, short-lived credentials
        const leaseData = await this.vault.read('database/creds/my-role');
        
        const credentials = {
            username: leaseData.data.username,
            password: leaseData.data.password
        };
        
        // Schedule lease renewal before expiry
        const leaseDuration = leaseData.lease_duration;
        setTimeout(
            () => this.renewLease(leaseData.lease_id),
            (leaseDuration - 60) * 1000 // Renew 1 minute before expiry
        );
        
        return credentials;
    }
 
    async getSecret(path: string): Promise<string> {
        // Check cache first
        const cached = this.cache.get(path);
        if (cached && cached.expiresAt > Date.now()) {
            return cached.value;
        }
        
        // Fetch from Vault
        const result = await this.vault.read(`secret/data/${path}`);
        const value = result.data.data.value;
        
        // Cache with TTL
        this.cache.set(path, {
            value,
            expiresAt: Date.now() + 60000 // 1 minute cache
        });
        
        return value;
    }
 
    private async getKubernetesServiceAccountToken(): Promise<string> {
        const fs = await import('fs/promises');
        return fs.readFile(
            '/var/run/secrets/kubernetes.io/serviceaccount/token',
            'utf-8'
        );
    }
    
    private async renewLease(leaseId: string): Promise<void> {
        try {
            await this.vault.leaseRenew(leaseId);
        } catch (error) {
            console.error('Lease renewal failed, fetching new credentials');
            // Application should handle credential refresh
        }
    }
}

The Twelve-Factor App on Config

Defense in Depth

No single security measure is sufficient. Security requires defense in depth—multiple layers of protection so that if one layer fails, others still protect the system.

Defense in Depth Layers
Layer	Controls	Examples
Network	Firewalls, segmentation, DDoS protection	VPC, Security Groups, Cloudflare
Transport	Encryption in transit, mTLS	TLS 1.3, Certificate verification
Application	Input validation, parameterized queries	WAF rules, OWASP recommendations
Data	Encryption at rest, access controls	AES-256, Column-level encryption
Identity	Strong auth, MFA, least privilege	SAML, OIDC, RBAC
Monitoring	Anomaly detection, audit logging	SIEM, IDS/IPS, CloudTrail
Recovery	Backups, disaster recovery	Point-in-time recovery, DR sites

The Castle vs. Airport Security Model:

Castle Model (Traditional Perimeter Security):

Hard shell, soft center
Strong firewall at the edge
Internal network trusted
Problem: Once inside, attackers move freely

Airport Model (Zero Trust):

Security everywhere
Every person verified at every stage
No trusted zones
Lateral movement restricted
Principle: Assume breach has occurred

Modern distributed systems should adopt the Airport Model. With cloud, containers, and remote work, there is no "perimeter" to defend.

Security as Code

Security Monitoring and Incident Response

Prevention alone isn't enough. You must assume breaches will occur and prepare to detect and respond to them quickly.

Essential Security Monitoring

•Centralized Logging — All systems ship logs to a central, tamper-evident store. Attackers can't cover tracks by deleting local logs.
•Authentication Events — Log all logins, failures, privilege escalations. Detect password spraying, brute force, credential stuffing.
•Authorization Failures — Track access denied events. Indicates reconnaissance or misconfiguration.
•Anomaly Detection — Baseline normal behavior, alert on deviations. Unusual access patterns, geographic anomalies, time-based patterns.
•Network Flows — Log connections between services. Detect unexpected lateral movement.
•Secrets Access — Audit every secret retrieval. Alert on bulk access or unusual patterns.
•Data Egress — Monitor outbound traffic volumes. Large data exfiltration attempts should trigger alerts.

Incident Response Framework:

Preparation — Runbooks, communication channels, escalation paths, practice drills
Identification — Detect and confirm the incident. Is this a false positive or real threat?
Containment — Limit damage. Isolate affected systems without destroying evidence.
Eradication — Remove the threat. Patch vulnerabilities, remove backdoors.
Recovery — Restore systems to normal operation with confidence they're clean.
Lessons Learned — Post-incident review. What failed? What worked? Update defenses.

Key Metrics:

MTTD (Mean Time to Detect) — How long until you notice a breach?
MTTR (Mean Time to Respond) — How long from detection to containment?

Industry average for breach detection is 200+ days. Best-in-class is under an hour. The difference is preparation and monitoring.

Assume You're Already Compromised

Summary: The Network Is Never Secure

We've explored the fourth fallacy of distributed computing: the assumption that the network is secure. Let's consolidate the key insights:

Key Takeaways

•Assume the network is hostile — Every packet traverses infrastructure you don't control. Encryption and authentication are mandatory, not optional.
•TLS everywhere — All service communication should use TLS. Internal 'private' networks are not secure. Perfect forward secrecy protects past traffic.
•Strong authentication matters — Move beyond API keys to mTLS, service mesh identity, or cloud IAM. Cryptographic identity verification beats shared secrets.
•Zero Trust Architecture — Never trust, always verify. No privileged internal networks. Every request authenticated and authorized.
•Secrets management is critical — Use proper secrets managers with dynamic secrets, automatic rotation, and audit logging.
•Defense in depth — Multiple security layers ensure single failures don't compromise the system. Assume breach has occurred.
•Detection and response matter — Prevention will fail. Prepare to detect and respond to incidents quickly.

What's next:

Page Complete

4 / 5