Percentiles: p50, p95, p99

Imagine a restaurant that advertises "average wait time: 10 minutes." Sounds reasonable. But what if 90% of customers wait 5 minutes while 10% wait 55 minutes? The average is still 10 minutes—but the experience for that unlucky 10% is terrible. Would you dine there if you knew you had a 1-in-10 chance of waiting nearly an hour?

This is the fundamental problem with averages in performance metrics. They hide the tail, obscure variability, and give a false sense of confidence. In system performance, the tail is often where the most important stories hide—the users who had a terrible experience, the requests that timed out, the edge cases that reveal architectural weaknesses.

Percentiles solve this problem. They describe the entire distribution of performance, letting you understand not just the typical case but every level of the experience spectrum—from the best to the worst.

Before understanding what percentiles offer, we must understand why averages are insufficient—and even dangerous—for performance analysis.

The Mathematical Problem

The arithmetic mean (average) is calculated as:

average = sum(all_values) / count(all_values)

This single number collapses an entire distribution into one point, losing all information about variability, shape, and outliers. Two completely different distributions can have identical averages:

Both distributions show 100ms average latency. But Distribution A is perfectly consistent—every user gets 100ms. Distribution B has 90% of users at 50ms (twice as fast!) but 10% at 550ms (terrible experience). These systems have radically different user experiences despite identical averages.

The Outlier Domination Problem

Averages are sensitive to extreme values. A single outlier can dramatically skew the average:

• 99 requests at 50ms + 1 request at 5 seconds = 99ms average
• Without the outlier: 99 requests at 50ms = 50ms average

One bad request (1% of traffic) doubles the reported average latency. Is the system twice as slow? No—99% of users had an excellent experience. The average hides this.

A percentile (also called quantile) is a value below which a given percentage of observations fall. If the 95th percentile (p95) of latency is 200ms, it means:

• 95% of requests completed in ≤200ms
• 5% of requests took >200ms

Percentile Calculation

To find the p-th percentile:

1. Sort all observations from smallest to largest
2. Find the value at position (p/100) × n, where n is the total count
3. If the position isn't an integer, interpolate between adjacent values

Common Percentiles in System Performance

Why p50 (Median) vs Mean?

The median is the "middle" value—half of observations are above it, half below. Unlike the mean, the median is:

• Robust to outliers: A single 5-second request doesn't move the median
• Representative of typical experience: The median user's actual experience
• Stable over time: Less noisy in monitoring dashboards

For most system performance contexts, the median is a better measure of "typical" than the mean.

Distribution Shape Matters

Percentiles together describe the distribution shape:

• Tight distribution: p50, p90, p99 are close together (e.g., 50ms, 55ms, 70ms)
• Long-tailed: p50 is low, p99 is high (e.g., 50ms, 500ms, 5s)
• Bimodal: Two distinct "humps" in the distribution (cache hits vs. misses)

Seeing p50=50ms, p99=5s tells you the system is highly variable—the tail is 100× worse than the median.

Understanding percentile data requires practice. Let's work through how to read and interpret common percentile presentations.

Percentile Profiles

A percentile profile is a summary of key percentile values. Here's an example API response time profile:

What This Profile Tells Us:

1. The typical experience is good: 45ms median is snappy
2. Most users are happy: 90% get <100ms
3. The tail is long: p99 is 9× the median, p99.9 is 28× the median
4. Mean is misleading: 78ms suggests worse typical performance than the 45ms median

Reading the Gaps

The gaps between percentiles reveal where problems concentrate:

• p50 to p90 (45ms → 98ms): Reasonably tight, 2× degradation for worst 50%→10%
• p90 to p99 (98ms → 423ms): Large gap (4×), significant tail starting here
• p99 to p99.9 (423ms → 1,247ms): 3× worse for the extreme tail

This pattern suggests something is causing occasional severe delays—perhaps GC pauses, lock contention, or cold cache hits. Investigating requests in the p99-p99.9 range would be valuable.

The importance of tail percentiles increases with scale. At low volume, the tail affects few users. At high volume, the tail affects many—and can become the dominant experience.

The Scale Multiplication Effect

At 1,000 requests per day, p99 affects 10 users—probably acceptable. At 1 billion requests per day (Google/Facebook scale), p99 affects 10 million users daily. Suddenly that "only 1%" tail represents a city's worth of unhappy users.

The Session Probability Problem

Consider a user session with 100 API calls (typical for a modern web app). If any single call is slow, the session feels slow. What's the probability of at least one slow call?

P(at least one slow) = 1 - P(all fast)^n
P(at least one slow) = 1 - (1 - p)^100

For different "slow" thresholds (beyond p99):

Modern systems often use fan-out patterns where one request triggers multiple parallel sub-requests. Understanding how percentiles behave in fan-out is critical for realistic latency planning.

The Fan-Out Amplification Effect

When you fan out to N backends in parallel, the aggregate latency is the maximum of all individual latencies (you wait for the slowest). This means tail latencies aggregate badly:

P(aggregate > threshold) = 1 - P(all < threshold)^N
                         = 1 - (percentile/100)^N

Example: Search Fan-Out

A search query fans out to 100 index shards in parallel. Each shard has independent latency with p99=200ms. What's the aggregate p99?

The Shocking Result:

With 100-way fan-out, the individual p99 becomes the aggregate p37! Two-thirds of aggregate requests will exceed what you thought was your p99 target. To achieve aggregate p99, you need individual p99.99 performance.

Implications for System Design:

1. Fan-out multiplies tail requirements: For N parallel calls, target individual p(100-1/N×100) to achieve aggregate p99
2. Reduce fan-out when possible: Fewer parallel calls = less tail amplification
3. Use hedged requests: Send duplicate requests to reduce effective tail
4. Set aggressive per-shard timeouts: Don't let one slow shard delay the entire aggregate
5. Consider partial results: Return best-effort results if some shards timeout

Percentiles are the correct language for Service Level Objectives (SLOs). An SLO should specify a percentile target, not an average target.

SLO Structure

A well-formed percentile SLO has three components:

[Percentage] of [requests/operations] should complete in under [threshold]

Examples:

• 99% of API requests should complete in under 300ms
• 99.9% of database queries should complete in under 100ms
• 95% of page loads should complete in under 2 seconds

Multi-Tier SLOs

Mature systems often have multiple SLO tiers:

Latency SLO Tiers:
- p50: < 100ms (typical experience)
- p90: < 250ms (good experience)
- p99: < 1s   (acceptable experience)
- p99.9: < 5s (maximum tolerable)

Different tiers serve different purposes:

• p50 SLO: Ensures typical user has a good experience
• p99 SLO: Limits worst-case for most users
• p99.9 SLO: Prevents extreme outliers from being too extreme

Computing percentiles at scale presents engineering challenges. Naive approaches that store every observation quickly exhaust memory. Let's examine practical approaches.

Naive Approach: Store Everything

collect all values → sort → extract percentiles

Memory usage: O(n) where n = number of observations

Problems:

• 1M requests/minute × 8 bytes = 8MB/minute = 480MB/hour just for storage
• Sorting for each query is expensive: O(n log n)
• Doesn't scale to streaming or long time windows

Practical Approach: Histograms

Pre-define buckets and count observations in each:

Buckets: [0-10ms, 10-25ms, 25-50ms, 50-100ms, 100-250ms, ...]
Counts:  [15,000,  45,000,  82,000,  54,000,   8,000,    ...]

Memory usage: O(number of buckets) = typically 20-50 values = constant

Advantages:

• Constant memory regardless of observation count
• Mergeable: can combine histograms from multiple servers
• Fast: O(1) per observation, O(buckets) to compute percentile

Disadvantages:

• Bucket boundaries are fixed, limiting precision
• Poor resolution at distribution edges if buckets poorly chosen

Streaming Approach: T-Digest

T-Digest is an algorithm that maintains a compact summary of a distribution with better precision at the tails:

• Stores ~100-200 "centroids" representing data clusters
• Higher precision near p0 and p100 (the tails)
• Mergeable for distributed computation
• Typically <1% error for p99 with few KB of memory

When to Use Which:

• HDR Histogram: Single-server latency measurement with known range
• T-Digest: Distributed systems needing to merge across servers
• Prometheus Histogram: When using Prometheus; uses fixed buckets
• Full Storage: Offline analysis where you need exact values