Performance Improvement Potential

Caching doesn't just make systems a little faster—it can transform them fundamentally. A system that takes 500 milliseconds to respond can become one that responds in 5 milliseconds. A database that struggles at 1,000 queries per second can effectively handle 100,000 requests per second. Infrastructure costs can drop by 90% or more.

These aren't hypothetical improvements. They're what effective caching achieves in production systems every day. But to achieve these gains, you need to understand the potential—how much improvement is possible, under what conditions, and how to predict the impact of caching before you implement it.

In this page, we'll quantify the performance benefits of caching with mathematical precision, explore the models that predict cache behavior, and understand the boundaries of what caching can and cannot achieve.

The most immediate and visible benefit of caching is latency reduction. By serving data from fast storage instead of slow backends, caching can reduce response times by orders of magnitude.

Understanding the latency gap:

The latency difference between cached and uncached access is profound:

Calculating average latency:

With caching, your average latency becomes a weighted average of hit and miss latencies:

Average Latency = (Hit Rate × Hit Latency) + (Miss Rate × Miss Latency)

Example calculation:

Consider a product page that requires fetching product data:

• Cache hit latency: 1 ms (from Redis)
• Cache miss latency: 80 ms (database query + network)
• Cache hit rate: 95%

Average Latency = (0.95 × 1ms) + (0.05 × 80ms)
                = 0.95ms + 4ms
                = 4.95ms

Without caching (100% miss rate): 80ms With 95% hit rate: 4.95ms

That's a 16x improvement in average latency.

The latency distribution shift:

Caching doesn't just improve averages—it fundamentally shifts the latency distribution. Without caching, every request is slow. With caching, most requests are fast, with a tail of slower requests (the misses).

Beyond latency, caching dramatically increases your system's throughput capacity. By serving most requests from cache, your backend can handle the remaining traffic comfortably—even under loads that would otherwise crush it.

The throughput multiplier formula:

If your backend can handle B requests per second without caching, and your cache has hit rate H, your effective system capacity becomes:

Effective Capacity = B ÷ (1 - H)

This is the throughput multiplier effect.

Real-world interpretation:

At 95% hit rate, a database that can handle 1,000 queries per second can effectively support a system receiving 20,000 requests per second. The cache absorbs 19,000 requests; only 1,000 reach the database.

This is why caching is the first technique to consider when scaling read-heavy systems. Before adding read replicas, before sharding, before exotic database architectures—try caching. It's often sufficient and far simpler.

The scaling curve:

Without caching, capacity scales linearly with infrastructure investment:

• 2x traffic → 2x servers
• 10x traffic → 10x servers

With effective caching, capacity scales sub-linearly:

• 2x traffic → ~1.05x additional backend capacity (with 95% hit rate)
• 10x traffic → ~1.5x additional backend capacity

This is because cache infrastructure scales more efficiently than database infrastructure. Adding Redis memory is cheaper than adding database read replicas.

While caching offers dramatic improvements, these improvements have limits. Amdahl's Law, originally formulated for parallel computing, applies equally well to caching optimization.

Amdahl's Law for Caching:

The maximum speedup achievable by caching is limited by the portion of the workload that cannot be cached.

Maximum Speedup = 1 ÷ (1 - Cacheable Fraction)

Where Cacheable Fraction is the portion of requests that can be served from cache.

Example:

If 90% of your requests are cacheable (read queries for static-ish data) and 10% are uncacheable (writes, real-time data):

Maximum Speedup = 1 ÷ (1 - 0.90) = 1 ÷ 0.10 = 10x

No matter how perfect your cache, you cannot achieve more than 10x speedup because 10% of traffic will always experience the uncached latency.

*Assuming perfect cache (100% hit rate on cacheable data, 0ms cache latency)

Practical implications:

Amdahl's Law tells us where to focus optimization efforts:

1. Identify the uncacheable fraction — What percentage of your traffic cannot be cached? Writes? Real-time data? Highly personalized content?

2. Optimize both cached and uncached paths — If 10% is uncacheable and takes 500ms, your average latency floor is 50ms even with perfect caching. You must also optimize the uncacheable path.

3. Expand what's cacheable — Often, data assumed to be uncacheable can be cached with creative approaches:

   • Short TTL caching for 'near-real-time'
   • Write-through caching for read-after-write consistency
   • Layered caching for personalized content (cache common components)

The cache-aside (or look-aside) pattern is the most common caching approach. Understanding its performance characteristics helps predict real-world behavior.

Cache-aside flow:

1. Application checks cache first
2. On hit: return cached data
3. On miss: fetch from database, store in cache, return data

Latency model:

T_hit = T_cache_lookup + T_cache_read
T_miss = T_cache_lookup + T_db_query + T_cache_write

T_average = (H × T_hit) + ((1-H) × T_miss)

Where:

• H = Hit rate
• T_cache_lookup = Time to check cache (typically < 1ms)
• T_cache_read = Time to read from cache (typically < 1ms)
• T_db_query = Database query time (often 10-100ms+)
• T_cache_write = Time to populate cache (typically < 1ms)

Sensitivity analysis:

Different parameters have different impacts on performance:

• Hit rate — Most impactful. Doubling hit rate from 50% to 90%+ transforms performance.
• Database latency — High impact on miss path. Slow DB means misses are very expensive.
• Cache latency — Lower impact. Cache is already fast; making it faster helps marginally.
• Cache write time — Only affects miss path. Usually negligible compared to DB query.

Key insight: Invest effort proportionally to impact. Improving hit rate almost always yields better returns than optimizing cache latency.

Beyond raw latency and throughput, caching provides significant benefits to concurrency and connection management—often overlooked but critically important factors.

The connection pool problem:

Databases have connection limits. Each concurrent query typically requires a connection. Without caching:

• 1,000 concurrent requests → potentially 1,000 database connections
• Connections are expensive (memory, file descriptors, authentication)
• Connection limits are often 100-1000 per database
• Exceeding limits causes connection errors or queuing

With 95% hit rate:

• 1,000 concurrent requests → only 50 database connections needed
• Connection pool utilization drops by 20x
• Headroom for burst traffic
• Reduced risk of connection exhaustion

Little's Law and caching:

Little's Law states: L = λW

Where:

• L = Average number of items in a system (concurrent operations)
• λ = Arrival rate (requests per second)
• W = Average time in system (latency)

For database connections:

Concurrent DB Operations = (Requests/sec × Miss Rate) × DB Query Time

Example:

• 1,000 requests/second
• 95% hit rate (5% miss rate)
• 50ms average database query

Concurrent DB Operations = (1000 × 0.05) × 0.050
                        = 50 × 0.050
                        = 2.5 concurrent queries

With caching, a system handling 1,000 RPS only needs ~3 database connections on average, with some headroom for variance. Without caching (0% hit rate):

Concurrent DB Operations = 1000 × 0.050 = 50 concurrent queries

Caching reduces connection needs by 20x in this example.

Performance improvement directly translates to cost savings. Let's quantify the financial impact of caching.

Cost per request calculation:

The cost of serving a request includes:

• Compute cost (server time)
• Database cost (query execution, I/O)
• Network cost (data transfer)
• Infrastructure overhead (connection management, etc.)

Cached requests have minimal cost compared to uncached requests:

Total cost of ownership model:

Consider a workload of 10 million requests per day:

Without caching:

• Database: 10M queries/day
• Requires: Large DB instance + read replicas
• Monthly DB cost: $5,000
• App servers: 10 instances to handle queue depth
• Monthly compute: $2,000
• Total: $7,000/month

With 95% cache hit rate:

• Database: 500K queries/day (95% reduction)
• Requires: Smaller DB instance, no replicas
• Monthly DB cost: $500
• App servers: 4 instances (faster responses)
• Monthly compute: $800
• Cache (Redis): 1 instance
• Monthly cache: $200
• Total: $1,500/month

Savings: $5,500/month (79% reduction)

Beyond performance and cost, caching provides resilience benefits that improve system reliability and availability. A well-designed cache can keep your system running even when backend components fail.

Caching as a circuit breaker:

When a database becomes slow or unavailable, a cache can:

1. Continue serving cached data (stale but available)
2. Reduce load on struggling backend (fewer requests)
3. Buy time for recovery (users don't notice immediately)
4. Prevent cascading failures (backend doesn't get overwhelmed)

Stale data as a fallback:

When properly configured, caches can serve stale data during backend outages:

• Stale-while-revalidate: Return cached data immediately, refresh in background
• Stale-if-error: Return stale data if backend errors, with appropriate headers
• Grace periods: Allow serving stale data for a period after TTL expires if refresh fails

For many applications, slightly stale data is far preferable to no data at all. A product catalog showing prices from 5 minutes ago is better than an error page.

Traffic absorption during incidents:

During backend incidents, caching provides critical protection:

1. Immediate protection: Cache continues serving regardless of backend state
2. Load reduction: Fewer misses means less pressure on struggling backend
3. Recovery time: Backend can recover without facing full traffic load
4. Gradual rewarming: After recovery, cache gradually repopulates