Latency: Response Time

In the world of distributed systems, latency is the pulse that reveals everything. It's the time a user waits between clicking a button and seeing a response. It's the delay between a service making a request and receiving an answer. It's the invisible tax on every interaction in your system.

Latency matters because users experience it directly. A page that loads in 100ms feels instantaneous. The same page at 3 seconds feels broken. Research from Google, Amazon, and Microsoft consistently shows that increased latency directly correlates with decreased user engagement, lower conversion rates, and reduced revenue. Amazon famously discovered that every 100ms of added latency cost them 1% in sales.

Latency is often casually defined as "how long something takes," but this imprecision causes significant confusion in system design conversations. Let's establish rigorous definitions that eliminate ambiguity.

Response Time vs. Latency

These terms are often used interchangeably, but they have subtle differences:

• Latency refers specifically to the delay introduced by the system—the time the request spends waiting or being processed, excluding time required by fundamental physics (like the speed of light across network wires).

• Response time is the total time from when a client sends a request to when it receives the complete response. It includes network transit time, queuing time, processing time, and all other delays.

In practice, when engineers say "latency," they usually mean response time. For clarity in this module, we'll use latency to mean the end-to-end response time as experienced by the caller.

The Anatomy of a Request's Lifetime

To truly understand latency, we must decompose the journey of a single request through a distributed system. Consider a user loading their profile page:

The Critical Insight:

A single "simple" request can touch dozens of components, each contributing latency. A profile page request might involve:

• 1 load balancer hop
• 3 microservice calls (user service, preferences service, activity service)
• 5 database queries
• 2 cache lookups
• 1 external API call (for avatar from CDN)

Each hop adds latency. Understanding this decomposition is the first step to optimization—you can't fix what you can't measure and attribute.

Latency doesn't come from nowhere—it has identifiable sources. Expert engineers develop an intuition for these sources so they can quickly diagnose where delays originate. Let's examine each major category:

Measurement is fundamental to performance engineering. You cannot optimize what you don't measure, and you cannot trust measurements you don't understand. Latency measurement comes with significant pitfalls that trip up even experienced engineers.

Where to Measure

The location of measurement determines what you learn:

The Coordinated Omission Problem

One of the most insidious measurement errors is coordinated omission. Most benchmarking tools measure latency like this:

1. Send request 1, measure response time
2. After response 1 returns, send request 2, measure response time
3. Repeat...

The problem: if request 1 takes 5 seconds (due to a GC pause, for example), request 2 never gets measured during that slow period. The benchmark appears to show all requests completing quickly, hiding the fact that the system was unresponsive for 5 seconds.

The fix: Measure at the intended send time, not the actual send time. If you intended to send 100 requests/second, measure the latency from when each request should have been sent, even if the previous request blocked you.

Timing Resolution and Accuracy

The clock you use matters:

• Wall-clock time (System.currentTimeMillis(), Date.now()): Millisecond resolution, can jump forward or backward due to NTP adjustments. Fine for coarse measurements.

• Monotonic clocks (System.nanoTime(), performance.now()): Won't go backward, microsecond or better resolution. Required for accurate short-duration measurements.

• CPU cycle counters (RDTSC): Nanosecond precision but complex to use correctly. Reserved for extreme precision needs.

Rule of thumb: For latencies under 100ms, use monotonic clocks. The error from wall-clock adjustments can exceed your measurement.

A critical insight that separates novice from expert thinking: latency is not a single number—it's a probability distribution.

When someone asks "What's the latency of your API?", there's no single correct answer. Every request has a different latency. Some complete in 50ms, others in 200ms, and occasionally one takes 5 seconds. The question should be "What does your latency distribution look like?"

Why Averages Lie

The arithmetic mean (average) is the most commonly reported statistic and the most misleading for latency:

• Consider a service where 99 requests complete in 100ms and 1 request takes 10 seconds.
• Average latency: (99 × 100ms + 1 × 10,000ms) / 100 = 199ms
• The average claims excellent performance while hiding that 1% of users wait 10 seconds!

Averages are dominated by the bulk of requests and hide the long tail. Real user experience includes the tail.

Percentiles Tell the Real Story

Percentiles (quantiles) describe the distribution without hiding the tail:

• Median (p50): 50% of requests are faster than this, 50% are slower. Represents the "typical" experience.
• p90: 90% of requests are faster. The top 10% of users experience worse.
• p99: 99% of requests are faster. Only 1% experience worse—but at scale, 1% is a lot of users.
• p99.9, p99.99: The extreme tail. Critical for high-availability systems.

A complete latency profile might be: p50: 50ms, p90: 75ms, p95: 100ms, p99: 250ms, p99.9: 2s

This tells us most requests are fast (50ms), but the tail is long (2s at p99.9). An average of 60ms would hide this entirely.

Latency metrics only matter because they affect real users. Understanding the human perception of latency helps you set appropriate targets and prioritize optimization work.

Human Perception Thresholds

Decades of human-computer interaction research have established clear perception boundaries:

The 100ms Rule

For interactive elements (buttons, links, controls), the 100ms threshold is critical. Below 100ms, users perceive instant response. Above 100ms, they perceive a delay. This isn't opinion—it's neurological. The human brain's processing latency is approximately 100ms, so responses faster than this appear simultaneous with the action.

Implications for system design:

• Critical path latency should target sub-100ms for interactive operations
• If you can't achieve sub-100ms, provide immediate visual feedback (loading states, skeletons)
• Background tasks can take longer if the user sees responsive UI

The Business Impact

Study after study confirms latency's business consequences:

• Google: 500ms additional delay reduced search traffic by 20%
• Amazon: Every 100ms latency cost 1% of sales
• Walmart: Every 100ms improvement increased revenue by up to 1%
• Bing: 2-second delay reduced revenue per user by 4.3%
• Aberdeen Group: 1-second delay correlates with 7% reduction in conversions

These aren't edge cases—they're consistent patterns across industries. Latency is not just an engineering concern; it's a business-critical metric.

Professional systems engineering uses latency budgets to allocate the acceptable delay across system components. This transforms vague goals ("make it fast") into precise, measurable contracts.

What is a Latency Budget?

A latency budget defines how much time each component is "allowed" to contribute to total latency. For a 300ms total budget on a user request:

Service Level Objectives (SLOs)

Latency SLOs define the target latency at specific percentiles:

SLO: 99% of requests complete in < 300ms
SLO: 99.9% of requests complete in < 1s
SLO: 99.99% of requests complete in < 5s

Note the structure: percentage of requests + latency threshold. The percentage is crucial—a 300ms p50 target is very different from a 300ms p99 target.

Why Different Percentile Targets?

Higher percentiles are exponentially harder to achieve. The difference between p99 and p99.9 often requires different architectural approaches:

• p99: Optimize the normal case, handle common failure modes
• p99.9: Eliminate GC pauses, add redundancy, implement request hedging
• p99.99: Geographic distribution, aggressive caching, multi-path routing

Armed with understanding of where latency comes from, we can systematically reduce it. Different strategies address different latency sources: