Throughput: Requests per Second

If latency tells us how fast a single request completes, throughput tells us how much work the system handles simultaneously. It's the difference between asking "How quickly can you run to the store?" versus "How many trips to the store can you make in an hour?"

Throughput—measured in requests per second (RPS), queries per second (QPS), or transactions per second (TPS)—defines your system's capacity. It determines how many users you can serve, how much data you can process, and ultimately, how big your business can grow before infrastructure becomes the bottleneck.

Understanding throughput is essential because it governs capacity planning, cost optimization, and scalability decisions. A system with beautiful 10ms latency but maxing out at 100 RPS won't survive a product launch. Conversely, a system capable of 1 million RPS but with 5-second latency delivers an unusable experience.

Throughput measures the rate of completed work over time. While the concept seems simple, precision matters significantly in system design conversations.

Common Throughput Metrics

What Counts as a "Request"?

Throughput measurements require clarity on what constitutes a unit of work:

• A single HTTP request? But a page load involves 50+ requests.
• A single API call? But one call might trigger 5 database queries.
• A user action? But "login" is simpler than "checkout with payment."

Different operations have vastly different costs. A read-only cache hit costs microseconds; a complex database transaction costs hundreds of milliseconds. 10,000 cache RPS represents less load than 1,000 complex-write RPS.

The Solution: Weighted Throughput

Sophisticated capacity planning weights operations by their cost:

Effective Load = Σ (operation_count × operation_cost)

Rather than reporting "1000 RPS," you might report "1000 RPS (70% reads, 25% simple writes, 5% complex transactions)" to distinguish from "1000 RPS (100% complex transactions)."

Every system has theoretical maximum throughput determined by its bottleneck resource. Understanding these limits helps you predict capacity and identify optimization targets.

Little's Law: The Fundamental Relationship

Little's Law is one of the most important equations in system performance:

L = λ × W

Where:

• L = Average number of items in the system (concurrent requests)
• λ (lambda) = Average arrival rate (throughput)
• W = Average time spent in system (latency)

Rearranged for throughput:

Throughput = Concurrency / Latency

Or: λ = L / W

This fundamental law has profound implications:

CPU-Bound Throughput Limits

For compute-intensive workloads, CPU becomes the bottleneck:

Max Throughput = (CPU_cores × CPU_utilization_target) / CPU_time_per_request

Example:

• 8 CPU cores
• Target 75% utilization (leaving headroom)
• 10ms CPU time per request

Max Throughput = (8 × 0.75) / 0.01 = 600 RPS

I/O-Bound Throughput Limits

For I/O-intensive workloads, storage or network becomes the bottleneck:

Max Throughput = Available_IOPs / IOPs_per_request

Example:

• SSD with 50,000 IOPS
• Each request requires 5 disk reads

Max Throughput = 50,000 / 5 = 10,000 RPS

The actual limit is the minimum of CPU limit and I/O limit.

Latency and throughput are deeply interconnected, but their relationship is often misunderstood. They are not simply inversely proportional—the relationship is more nuanced and has critical implications for system design.

The Queueing Theory Perspective

As throughput approaches system capacity, latency doesn't just increase—it explodes. This is the queueing theory effect:

• At 50% capacity: Latency is near baseline
• At 70% capacity: Latency starts to rise noticeably
• At 90% capacity: Latency may be 2-3x baseline
• At 95% capacity: Latency may be 5-10x baseline
• At 99% capacity: Latency approaches infinity

The mathematical reason: at high utilization, requests spend most of their time waiting in queues rather than being processed.

Understanding the Curve

The latency-throughput relationship follows characteristic patterns:

The Throughput-Latency Trade-off

You often cannot maximize both throughput and minimize latency simultaneously:

• Optimizing for throughput: Batch operations, full utilization, queue depth management. Latency suffers because requests wait in batches.

• Optimizing for latency: Dedicated resources, immediate processing, over-provisioning. Throughput suffers because resources sit idle.

The right balance depends on your use case:

• Trading systems: Minimize latency (cost is secondary)
• Batch processing: Maximize throughput (latency is irrelevant)
• Web applications: Balance both (SLO-driven trade-offs)

Accurate throughput measurement requires attention to methodology. Common mistakes lead to overly optimistic or misleading results.

Measurement Intervals Matter

Throughput varies over time. The interval you choose for measurement affects the story:

• 1-second intervals: Capture bursts and spikes. Good for alerting.
• 1-minute averages: Smooth out micro-bursts. Good for dashboards.
• 1-hour averages: Show daily patterns. Good for capacity planning.
• Daily/weekly averages: Show long-term trends. Good for growth planning.

Reporting "1000 RPS" could mean:

• Sustained 1000 RPS for hours (impressive)
• Peak 1000 RPS in a single second, averaging 100 RPS (less impressive)
• 1000 RPS in a benchmark, real traffic at 300 RPS (misleading)

Successful vs. Total Throughput

A critical distinction that catches teams:

• Total throughput: All requests received, including failures
• Successful throughput: Only requests that completed successfully
• Goodput: Successful requests that met latency SLOs

A system handling 10,000 RPS with 30% error rate has only 7,000 successful RPS. If 20% exceed latency SLOs, goodput is only 5,600 RPS.

Always measure goodput—it's what actually matters for users.

Load Testing for Throughput

To find maximum throughput, use proper load testing methodology:

1. Open-loop testing: Send requests at a fixed rate regardless of responses. Reveals how the system behaves under load.
2. Closed-loop testing: Wait for each response before sending the next request. Underestimates real capacity due to coordinated omission.
3. Stepped load: Gradually increase load while measuring latency. Find the point where latency degrades unacceptably.
4. Soak testing: Sustained load over hours/days to find capacity under real conditions (memory leaks, GC pressure, etc.).

Every system has a bottleneck—the component limiting overall throughput. Following Amdahl's Law, improving non-bottleneck components provides diminishing returns. You must identify and address the actual bottleneck.

Common Throughput Bottlenecks

When current throughput is insufficient, you have two fundamental scaling options: vertical (bigger machines) and horizontal (more machines). Each has distinct throughput implications.

Throughput Scaling Techniques

A system's throughput under normal conditions is interesting; its throughput under failure conditions is what matters for reliability. How does your system behave when things go wrong?

Graceful Degradation of Throughput

Well-designed systems degrade gracefully rather than collapsing:

• Healthy state: 10,000 RPS at full capacity
• One server down: Load redistributes, 9,000 RPS sustainable
• Database slow: Non-critical queries timeout, 7,000 RPS of core functionality
• Cache failure: Database handles load, 3,000 RPS with degraded performance
• Major outage: Minimal functionality, 500 RPS of critical paths only

This graceful degradation shouldn't happen by accident—it requires explicit design.

Load Shedding for Throughput Protection

When incoming throughput exceeds capacity, you must shed load deliberately:

• Queue length limits — Reject requests when queue exceeds threshold. Fast rejection is better than slow failure.
• Rate limiting — Cap requests per client/IP/API. Prevents one client from consuming all capacity.
• Priority queues — Serve high-priority requests first. Let low-priority requests fail under overload.
• Circuit breakers — Stop calling failing dependencies. Fast-fail rather than waiting for timeouts.

Backpressure Propagation

Rather than silently dropping requests, signal overload upstream:

• Return 429 (Too Many Requests) with Retry-After headers
• Return 503 (Service Unavailable) with expected recovery time
• Use HTTP/2 flow control to slow client sending rate
• Signal queue depth in responses so clients can adjust

Backpressure allows the entire system to slow down gracefully rather than suffering uncontrolled failure.