Latency SLIs

If availability is the prerequisite for user experience—the service must exist to be used—then latency is the quality of that experience. A service can be perfectly available yet feel broken if every interaction takes 10 seconds. Conversely, a fast service creates the sensation of power and responsiveness that defines great software.

Latency is deceptively complex to measure correctly. Unlike availability's binary success/failure, latency is a continuous distribution. Every request has a latency, and these latencies vary—sometimes dramatically. A service might respond in 50ms usually, but occasionally spike to 5 seconds. What is "the latency" of this service? 50ms? 5 seconds? Something in between? The answer profoundly affects your SLI.

Worse, user perception of latency isn't linear. The difference between 50ms and 100ms is barely noticeable. The difference between 1 second and 2 seconds feels significant. And the difference between 10 seconds and 20 seconds is essentially irrelevant—both feel like "forever." A good latency SLI must account for these perceptual realities.

Latency, in its simplest form, is the time between a user initiating an action and receiving a response. But this simple definition hides substantial complexity.

The Request Timeline

Consider a typical web request. The user clicks a button, and a cascade of operations begins:

1. Client processing (1-50ms): JavaScript event handling, request preparation
2. Connection establishment (0-300ms): DNS lookup, TCP handshake, TLS negotiation (may be cached/reused)
3. Request transmission (10-500ms): Data travels from client through network to server, varies by payload size and network quality
4. Server queue time (0-?ms): Request waits for available worker thread
5. Server processing (10-1000ms): Application logic, database queries, external service calls
6. Response transmission (10-500ms): Data travels back from server to client
7. Client rendering (10-500ms): Browser parses response, executes JavaScript, updates DOM
8. User perception (variable): User cognitively processes the visual change

The "latency" could be measured at many points along this chain, and different measurements capture different realities.

User-Centric Latency Measurement

For user-centric SLIs, we want to measure what users perceive. This typically means:

• For UI interactions: Time from user action (click, keypress) to visual feedback indicating completion
• For API calls: Time from request initiation to response processing completion
• For page loads: Time to first meaningful content or interactivity
• For background operations: Time to completion notification

The ideal is to measure from the user's device—Real User Monitoring (RUM). When that's not possible, measure as close to the network edge as feasible, and acknowledge the gap between your measurement and true user experience.

Client vs. Server Latency

Server-side latency is what your application directly controls. Client-side latency includes everything:

• Server-side latency: Typically stable, predictable
• Network latency: Varies by geography, network quality, congestion
• Client processing: Varies dramatically by device capability

A user on a 3G mobile connection in a rural area experiences different latency than a user on gigabit fiber in a data center city—even though your server response time is identical.

Implication: If you only measure server-side latency, you may miss that your mobile users are suffering. Client-side measurement, while noisier, captures the true user experience distribution.

Average latency is the most commonly reported metric and the most misleading. Understanding why requires examining how latency actually distributes.

Latency Distributions Are Not Normal

If human heights followed a normal (Gaussian) distribution, you'd expect approximately the same number of people significantly below average as above average. This is not how latency works.

Latency distributions are typically right-skewed (or "long-tailed"):

• Many requests cluster around a typical value
• A "tail" of requests takes much longer
• Extreme outliers can be orders of magnitude slower than typical

This skew occurs because latency has a floor (you can't respond faster than physics allows) but no ceiling (failures can cause nearly infinite delays).

The Problem with Averages

Consider two services with identical average latency of 200ms:

Service A: 100 requests, all at 200ms

• Average: 200ms
• Every user experiences 200ms

Service B: 99 requests at 100ms, 1 request at 10,100ms

• Average: (99 × 100 + 1 × 10100) / 100 = 200ms
• 99% of users experience 100ms (great!)
• 1% of users wait over 10 seconds (terrible!)

Both services have the same average, but Service B has a severe tail latency problem that the average completely obscures. If you have 1 million daily users, that "1%" is 10,000 frustrated users per day.

Tail latencies matter because:

• High-frequency users are more likely to encounter tail latencies (more attempts = more chances)
• In microservices, tail latencies compound (one slow service slows the whole request)
• Tail latencies often indicate system problems that will worsen
• The users experiencing tail latencies don't care about the average

Which Percentile to Use?

The choice of percentile depends on your context:

p50 (Median):

• Shows what a "typical" user experiences
• Good baseline, but doesn't surface tail problems
• Use for understanding typical case, not for SLIs

p90 or p95:

• Common choice for SLIs
• Captures most users' experience while allowing for some outliers
• Balances sensitivity to problems with tolerance for noise

p99:

• More aggressive; doesn't tolerate tail latency
• Important for high-frequency users and chained services
• Can be noisy for low-traffic services (1 in 100 is still a sample size issue)

p99.9 or higher:

• Catches the worst outliers
• Relevant for very high-traffic services
• Challenging to reason about and often dominated by pathological cases

Practical guidance: Most services should track p50, p90, p95, and p99. Set SLO targets at p90 or p95 for the primary SLI, with a complementary (looser) p99 target to ensure tails are bounded.

Beyond tracking specific percentiles, understanding your full latency distribution provides insights that point metrics miss.

Distribution Shapes and Their Meaning

Unimodal right-skewed: The classic latency distribution. A single peak (mode) near the minimum, with a tail extending toward higher latencies. This is "normal" for most services.

Bimodal: Two distinct peaks. Often indicates two different code paths (cached vs. uncached) or two distinct user populations (local vs. international). Not inherently problematic, but should be understood.

Uniform/flat: No clear typical value. Often indicates a queueing problem—requests aren't processed on arrival but wait in line.

Heavy-tailed: Extreme outliers far beyond the typical range. May indicate occasional catastrophic slowdowns (GC pauses, lock contention, cold caches).

Histograms for Human Understanding

While percentiles are precise, latency histograms provide intuitive understanding of distribution shape. When analyzing latency, always visualize:

1. Linear-scale histogram: Shows the bulk of the distribution clearly
2. Log-scale histogram: Reveals tail behavior that's invisible on linear scale
3. Time-series of percentiles: Shows how the distribution changes over time

Key patterns to watch for:

• Shifting mode: If the peak "typical" latency moves, something fundamental changed (deployment, traffic pattern, dependency)
• Growing tail: If p99/p50 ratio increases over time, tail problems are worsening
• New modes appearing: Sudden bimodality suggests a new code path or a subset of users hitting problems
• Periodic spikes: Regular tail latency spikes often correlate with batch jobs, garbage collection, or maintenance operations

Choosing the right latency target requires balancing user expectations, technical constraints, and business requirements.

Framework for Target Selection

Step 1: Understand user expectations

User tolerance for latency depends on the interaction type. Research and industry practice provide benchmarks:

• Perceived instant (< 100ms): Simple UI updates, toggle switches, button feedback
• Perceptibly responsive (< 300ms): Navigation, simple searches, form submissions
• Noticeable but acceptable (< 1s): Complex searches, page loads, moderate data processing
• Requires feedback (1-5s): Heavy computations, complex operations, needs progress indication
• Frustration zone (> 5s): Users question whether it's working, may retry or abandon

Step 2: Analyze current performance

Before setting targets, understand where you are:

• What are your current percentiles (p50, p90, p95, p99)?
• How stable are these over time?
• What's causing tail latency—is it addressable?

Setting a target you can't achieve is demoralizing. Setting a target you always meet is not ambitious enough.

Step 3: Consider the full stack

Your latency target for user-facing interactions must budget for the entire stack:

• If your target is 200ms end-to-end
• And network round-trip is 50ms
• And client rendering is 20ms
• Then server processing must complete in 130ms

Work backwards from user-facing target to component budgets.

The Multi-Percentile SLI Pattern

A single latency target is often insufficient. Consider a multi-tier approach:

Primary target (p95): This is your main SLI target. "95% of requests complete within X ms." This sets the expectation for almost all users.

Tail target (p99): A secondary, looser target that bounds the worst case. "99% of requests complete within Y ms (where Y > X)." This prevents severe tail latency.

Example:

• p95 target: 200ms
• p99 target: 1000ms

This says: "Almost everyone (95%) gets a response in 200ms. Even the slowest 5% still get a response in under 1 second."

Why not just p99 for everything?

The stricter the percentile, the harder to maintain. P99 targets are highly sensitive to outliers—one pathological request in 100 can cause an SLO breach. For most services:

• p95 provides the right balance for the primary SLI
• p99 serves as a tail bound to prevent catastrophic degradation
• p99.9 is reserved for extremely high-traffic critical services

Accurate latency measurement is harder than it appears. Several challenges can corrupt your data.

Challenge 1: Clock Skew and Precision

Distributed systems involve multiple clocks, and clocks drift. If you compute latency by subtracting timestamps from different machines, clock skew introduces errors.

Solutions:

• Measure duration on a single machine (start and end times from the same clock)
• Use monotonic clocks (not affected by NTP adjustments) for duration measurement
• Synchronize clocks with NTP but understand remaining drift margins

Challenge 2: Coordinated Omission

Coordinated omission is a measurement error where slow requests are systematically under-counted. It happens when your load generator waits for a response before sending the next request.

If your service is supposed to handle 100 requests/second:

• If request 1 takes 10 seconds, requests 2-1000 don't get sent during that 10 seconds
• Your data shows 1 slow request, not the 999 requests that would have been slow had they been sent
• Result: Tail latencies are massively underestimated

Challenge 3: Survivorship Bias

Timed-out requests often don't appear in latency data because they never complete. If a request times out after 30 seconds:

• It's not in your latency distribution (no final timestamp)
• Your latency data appears better than reality
• The worst user experiences are invisible

Solutions:

• Count timeouts separately and track timeout rate
• Include timeouts in latency data at the timeout threshold value
• For SLO purposes, treat timeouts as failures at maximum latency

Challenge 4: Aggregation Loss

As discussed earlier, percentiles can't be averaged. If you're using pre-aggregated metrics:

• Store histograms (bucketed counts), not pre-computed percentiles
• Use mergeable data structures like HDR Histogram or T-Digest
• Compute percentiles from raw or histogram data at query time

Challenge 5: Sampling Bias

For high-traffic services, you may sample requests for detailed tracing. Sampling must not bias toward fast requests:

• Head-based sampling (decide at request start): May under-sample slow requests that are in-flight during sampling decisions
• Tail-based sampling (decide at request end): Can preferentially capture slow requests, biasing upward
• True random sampling: Unbiased but may miss rare slow requests

For latency SLIs, prefer non-sampled metrics or ensure sampling is unbiased with respect to latency.

In systems with multiple components and dependencies, understanding how latency accumulates is critical for meaningful SLIs.

The Latency Budget Concept

A latency budget allocates your end-to-end latency target across components. For a 300ms user-facing SLI:

This budget makes hidden assumptions explicit and allocates responsibility.

Serial vs. Parallel Latency

How components are arranged affects total latency:

Serial execution: Latencies add up. If A takes 100ms, then B takes 100ms, total is 200ms.

Parallel execution: Latency is the max of parallel paths. If A and B execute in parallel (both 100ms), total is 100ms.

Microservices architectures offer opportunities for parallelization, but dependencies often force serialization. Request fans out to 5 services, but one service depends on another's result—that path is serial.

Latency in fan-out patterns:

When a request fans out to N backends and waits for all:

• Total latency = max(latencies of all N backends)
• This is dominated by the slowest backend
• The more backends, the more likely one is slow (tail latency amplification)

If each backend has p99 = 100ms, and you fan out to 10 backends:

• Probability any single backend is slow: 1%
• Probability at least one is slow with 10 backends: 1 - (0.99)^10 ≈ 10%
• Your effective p90 is now near the backend's p99

Fan-out amplifies tail latency dramatically. This is why services with many dependencies struggle with tail latency despite each dependency being individually fast.

Let's examine concrete latency SLI specifications for real-world systems.

Example 1: Search API Latency SLI

Example 2: End-to-End Page Load Latency SLI

Example 3: Database Query Latency SLI (Internal Component)