Scalability

A system that works for 100 users may crumble under 10,000. One that handles 10,000 might collapse at a million. Scalability is the ability of a system to handle increased load by adding resources—and it's perhaps the most critical architectural concern for any successful service.

The client-server model inherently creates scaling challenges: one server, many clients. As clients multiply, the server becomes a bottleneck. Solving this requires understanding the dimensions of scale, the techniques available, and the tradeoffs each involves. Great engineers don't just build systems that work—they build systems that continue working as demand grows.

Scalability is the capability of a system to handle a growing amount of work, or its potential to accommodate growth. A scalable system can increase its capacity by adding resources (hardware, servers, instances) in a way that maintains or improves its performance characteristics.

Dimensions of Scalability:

Scalability vs. Performance:

These related concepts are often confused:

• Performance: How fast a system responds to individual requests (latency) or how many it handles per second (throughput)
• Scalability: How performance changes as load increases, and ability to improve by adding resources

A system can be performant but not scalable (fast for low load but degrades severely under high load), or scalable but not performant (handles load gracefully but each request is slow).

Vertical scaling means adding more power to an existing machine: more CPU, more RAM, faster storage. It's the conceptually simplest approach—make the server bigger.

When Vertical Scaling Works Well:

• Early-stage systems — Simpler to start with a bigger server than design for distribution
• Database servers — Many databases scale vertically better than horizontally (ACID is hard to distribute)
• Low-latency requirements — Single server avoids network latency between components
• Limited concurrent users — If peak load fits on one machine, why add complexity?

Practical Limits:

As of 2024, the largest cloud instances offer:

• CPUs: 400+ cores
• Memory: 24+ TB RAM
• Storage: Millions of IOPS

This seems enormous, but traffic for popular services can easily exceed what any single machine can handle. Twitter processes ~500 million tweets/day. Netflix serves ~15% of global internet bandwidth. No single server can handle these loads.

Vertical Scaling in Practice:

Load increases 5x:
  → Upgrade from 8 CPU to 32 CPU: $500/mo → $2,000/mo
  → Still one server; works immediately
  → If load increases 10x more, hit ceiling

Compare to horizontal scaling:
  → Add 5 servers at original spec: 5 × $500/mo = $2,500/mo
  → More complex setup, but no ceiling
  → Built-in redundancy

Horizontal scaling means adding more machines to distribute the load. Instead of one big server, you have many smaller servers working together.

Requirements for Horizontal Scaling:

To scale horizontally, your application must meet certain requirements:

1. Stateless or Externalized State — Servers can't rely on local state if any server might handle any request. Either design stateless, or store state in shared systems (database, cache).

2. Shared-Nothing Architecture — Each server operates independently; doesn't assume what's in other servers' memory or storage.

3. Idempotent Operations — Requests may be retried on different servers; operations must handle this safely.

4. External Session Storage — If sessions are needed, store in Redis/database accessible to all servers.

5. Centralized or Distributed Data Store — Servers need somewhere to read/write persistent data.

Load balancing is the distribution of incoming traffic across multiple servers. It's the essential mechanism that enables horizontal scaling—without it, clients would still go to a single server.

Load Balancing Algorithms:

Health Checks:

Load balancers must detect when servers are unhealthy and stop sending traffic to them:

// Health check configuration example
healthCheck:
  path: /health              # Endpoint to check
  interval: 10s              # How often to check
  timeout: 5s                # How long to wait for response
  healthyThreshold: 2        # Consecutive successes to mark healthy
  unhealthyThreshold: 3      # Consecutive failures to mark unhealthy

Types of Health Checks:

Load Balancer Types:

1. Layer 4 (Transport) — Balances based on TCP/UDP connection; doesn't inspect content

   • Very fast; handles any protocol
   • Can't make decisions based on URL, headers, etc.

2. Layer 7 (Application) — Inspects HTTP (or other protocol) content

   • Can route based on URL path, headers, cookies
   • Can terminate SSL, inject headers, modify requests
   • Higher overhead; protocol-specific

Caching stores copies of frequently accessed data closer to where it's needed, reducing load on origin servers and improving response times. It's one of the most powerful scalability techniques.

Caching Layers:

Caching Patterns:

Cache-Aside (Lazy Loading):

async function getUser(userId) {
    // 1. Check cache first
    let user = await cache.get(`user:${userId}`);
    if (user) return user;  // Cache hit
    
    // 2. Cache miss: fetch from database
    user = await db.query('SELECT * FROM users WHERE id = ?', [userId]);
    
    // 3. Populate cache for next time
    await cache.set(`user:${userId}`, user, {ttl: 3600});
    
    return user;
}

• Application manages cache explicitly
• Only requested data is cached (no wasted cache space)
• First request always hits database (cold cache problem)

Write-Through:

async function updateUser(userId, data) {
    // 1. Write to database
    await db.query('UPDATE users SET ? WHERE id = ?', [data, userId]);
    
    // 2. Immediately update cache
    await cache.set(`user:${userId}`, {...existingData, ...data});
}

• Cache always consistent with database
• Write latency includes cache update
• Every write updates cache (even if rarely read)

Write-Behind (Write-Back):

async function updateUser(userId, data) {
    // 1. Write to cache only (fast return)
    await cache.set(`user:${userId}`, data);
    
    // 2. Asynchronously persist to database
    writeQueue.enqueue({table: 'users', id: userId, data});
}

• Very fast writes (cache only)
• Risk of data loss if cache fails before persist
• Complexity of managing write queue

Databases are often the hardest component to scale because they maintain authoritative state and must preserve consistency. While application servers can be stateless and easily replicated, databases require more sophisticated approaches.

Read Replicas:

Most applications have far more reads than writes. Read replicas scale read capacity while maintaining a single write source.

                    ┌──────────────┐
      Writes ──────▶│   Primary    │───── Replication ─────┐
                    │   Database   │                       │
                    └──────────────┘                       ▼
                                                    ┌──────────────┐
      Reads ───────────────────────────────────────▶│   Replica 1  │
                                                    └──────────────┘
                                                           │
                                                           ▼
                                                    ┌──────────────┐
      Reads ───────────────────────────────────────▶│   Replica 2  │
                                                    └──────────────┘

• Pros: Simple setup; widely supported; no application changes for reads
• Cons: Replication lag means replicas may be slightly stale; doesn't scale writes

Sharding (Horizontal Partitioning):

Divide data across multiple databases, each holding a subset.

                    ┌──────────────┐
    user_id 1-1M ──▶│   Shard 1    │
                    └──────────────┘
                    
                    ┌──────────────┐
   user_id 1M-2M ──▶│   Shard 2    │
                    └──────────────┘
                    
                    ┌──────────────┐
   user_id 2M-3M ──▶│   Shard 3    │
                    └──────────────┘

Sharding Strategies:

Modern Database Options:

Beyond specific techniques, certain patterns and principles guide the design of scalable systems.

Asynchronous Processing Pattern:

Synchronous (doesn't scale well):
[User Request] → [Process Immediately] → [Wait] → [Return Result]

Asynchronous (scales much better):
[User Request] → [Enqueue Job] → [Return Accepted]
                      ↓
         [Worker Pool Processes When Available]
                      ↓
              [Notify User When Done]

Use asynchronous processing when:

• Operation takes more than a few seconds
• Result isn't needed immediately
• Load is spiky (queue absorbs spikes)
• Operation can be retried on failure

The CQRS Pattern (Command Query Responsibility Segregation):

Separate read and write paths, allowing each to scale independently:

┌───────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Write Path   │────▶│  Write Database │────▶│  Event Stream   │
│  (Commands)   │     │  (Normalized)   │     │  (Updates)      │
└───────────────┘     └─────────────────┘     └────────┬────────┘
                                                       │
┌───────────────┐     ┌─────────────────┐              │
│   Read Path   │◀────│   Read Store    │◀─────────────┘
│   (Queries)   │     │  (Denormalized) │
└───────────────┘     └─────────────────┘

• Reads scale differently than writes
• Read store can be optimized for query patterns
• Eventual consistency between write and read stores