Cache Hit vs Cache Miss

Every time your system needs data, it asks a simple question: Is this data already in the cache? The answer—yes or no—determines whether you serve the request in microseconds or milliseconds. This binary outcome, multiplied across millions of requests, defines whether your system feels instantaneous or sluggish, whether your infrastructure costs are reasonable or astronomical.

Understanding the difference between cache hits and misses, and more importantly, understanding why hits occur and how to maximize them, is fundamental to building high-performance systems. In this page, we'll dissect the mechanics of cache access, explore the critical concept of hit rate, and examine how hit rate impacts every aspect of system behavior.

A cache hit occurs when requested data is found in the cache and can be served directly without accessing the slower backend data source. This is the happy path—the scenario you want to optimize for.

The cache hit sequence:

1. Request arrives — A component requests data by key
2. Key lookup — Cache performs a hash lookup or index scan to find the key
3. Entry found — The key exists in the cache
4. Validity check — The entry is checked for expiration/staleness
5. Entry valid — The data is within its TTL and usable
6. Data returned — Cached data is deserialized (if necessary) and returned
7. Metadata update — Access time, hit count, etc. are updated

The entire sequence typically completes in microseconds to low milliseconds, depending on whether the cache is in-process (nanoseconds) or network-accessible (sub-millisecond to a few milliseconds).

Why cache hits are fast:

Cache hits are fast for several reinforcing reasons:

• Hardware speed: In-memory access (RAM) is 100,000x+ faster than disk access
• Data locality: Cached data is close to the requester (same machine, same datacenter)
• Optimized data structures: Caches use hash tables for O(1) key lookup
• No computation: The data is pre-computed and ready to serve
• Connection reuse: Cache connections are typically pooled and persistent

The hit's contribution to perceived performance:

Users experience the average response time of your system, which is heavily weighted by cache hits. If 95% of requests hit the cache (0.5ms each) and 5% miss (100ms each), your average response time is:

0.95 × 0.5ms + 0.05 × 100ms = 0.475ms + 5ms = 5.475ms

The 5% of misses dominate the average! This is why even small improvements in hit rate can dramatically improve perceived performance.

A cache miss occurs when requested data is not found in the cache (or is found but stale/invalid). The application must then fetch data from the slower backend source. This is the expensive path that caching aims to minimize.

The cache miss sequence:

1. Request arrives — A component requests data by key
2. Key lookup — Cache performs a hash lookup
3. Entry not found — The key doesn't exist (or has expired)
4. Backend fetch — Application queries the origin data source (database, API, etc.)
5. Origin processing — Database executes query, computes result
6. Response received — Data arrives from the backend
7. Cache population — Data is stored in cache with appropriate TTL
8. Data returned — Data is returned to the requester

This sequence takes significantly longer—typically 10-1000x longer than a cache hit, depending on the backend.

Types of cache misses:

Not all misses are equal. Understanding miss types helps diagnose cache behavior:

1. Compulsory Miss (Cold Miss)

The very first request for a piece of data. No cache can avoid this—the data has never been accessed before. Compulsory misses are inevitable; they represent the cost of populating an empty cache.

2. Capacity Miss

The cache is full and had to evict this entry to make room for other data. The data was previously cached but has been removed. Indicates the cache is undersized for the working set.

3. Conflict Miss

In caches with limited associativity (common in CPU caches), data may be evicted even when the cache isn't full because of hash collisions. Less common in application-level caches.

4. Coherence/Invalidation Miss

The entry was explicitly invalidated due to a data update. The data existed but was removed to maintain correctness. This is an intentional miss.

5. Expiration Miss

The entry's TTL expired. The data was cached but is now considered stale. Indicates TTL may be too short, or this is working as designed.

The cache hit rate is the most important metric for evaluating cache effectiveness. It measures the percentage of requests that are served from the cache without accessing the backend.

Hit Rate Formula:

Hit Rate = (Cache Hits) / (Cache Hits + Cache Misses) × 100%

Alternatively stated:

Hit Rate = (Cache Hits) / (Total Requests) × 100%

A hit rate of 95% means 95 out of every 100 requests are served from cache. Only 5 requests reach the backend.

Miss Rate:

The complement of hit rate is miss rate:

Miss Rate = 100% - Hit Rate
Miss Rate = (Cache Misses) / (Total Requests) × 100%

At 95% hit rate, miss rate is 5%.

Interpreting hit rates:

Hit rate interpretation depends heavily on context:

• CDN serving static assets: Expect 95-99%+. Static content should almost always hit.
• Application cache for database queries: 70-90% is often realistic. Queries vary in cacheability.
• API response cache: Depends on personalization. 50-80% for personalized, 90%+ for shared content.
• Session cache: Near 100% within session lifetime—every session exists once created.

Hit rate over time:

Hit rates aren't static. They change based on:

• Cache temperature: Cold caches have low hit rates; they improve as caches warm
• Traffic patterns: Off-peak traffic may access less common data, lowering hit rate
• Data changes: Bulk updates invalidate entries, temporarily lowering hit rate
• New feature launches: Novel data has no cache history, causing compulsory misses

One of the most important—and counterintuitive—properties of hit rate is that its impact on system performance is non-linear. Small improvements in hit rate at high cache hit rates can have dramatic effects, while the same improvements at lower rates are less impactful.

The backend load calculation:

If your system receives R requests per second and your cache hit rate is H, the requests reaching your backend are:

Backend Load = R × (1 - H)

Consider 100,000 requests per second:

Notice the pattern: moving from 90% to 95% hit rate halves your backend load. Moving from 95% to 99% cuts backend load by 80%. At high hit rates, each percentage point improvement has outsized impact.

The latency distribution effect:

Hit rate also determines your latency distribution. Let's model a system where:

• Cache hit latency: 1ms
• Cache miss latency: 100ms

Important observation: Your p50 (median) latency improves dramatically with hit rate, but your p99 latency remains at the miss latency until you achieve extremely high hit rates. This is why tail latency is often dominated by cache misses even in well-optimized systems.

Capacity planning implications:

Because hit rate determines backend load, capacity planning must account for cache behavior:

• Provisioning: Size your backend for miss traffic, not total traffic
• Cache failures: If cache fails, you must handle 100% of traffic—massive spike
• Cold starts: New cache nodes temporarily increase miss rate
• Hit rate degradation: Even 5% drop in hit rate can significantly increase backend load

Achieving high hit rates requires understanding and optimizing the factors that influence whether requests hit or miss. These factors are often interconnected and must be balanced against each other.

The working set concept:

The working set is the subset of data that is actively accessed during a given time window. Understanding your working set is crucial for cache sizing:

• If cache size > working set: Most requests hit (bounded by TTL)
• If cache size < working set: Capacity misses are common, hit rate suffers
• If cache size ≈ working set: Efficient, but vulnerable to access pattern changes

Measuring working set:

Estimate working set by analyzing:

• Unique keys accessed over time periods (hour, day, week)
• Distribution of access frequency
• Size of cached entries

Example analysis:

- 1 hour: 50,000 unique keys accessed
- 1 day: 200,000 unique keys accessed  
- 1 week: 500,000 unique keys accessed
- Average entry size: 2 KB

1-hour working set: ~100 MB
1-day working set: ~400 MB
1-week working set: ~1 GB

If your cache is 512 MB with a 1-hour TTL, you're likely undersized for the workload.

One of the most dangerous cache-related failure modes is the cache miss storm, also known as the thundering herd problem. This occurs when a large number of requests simultaneously miss the cache and all hit the backend, potentially overwhelming it.

How thundering herds form:

Imagine a popular cached entry—say, the homepage data—that 1,000 requests per second access. At exactly the moment this entry expires:

1. First request finds entry expired → triggers backend fetch
2. Before fetch completes, 50 more requests arrive → all find entry missing
3. All 50 requests independently trigger backend fetches
4. Backend suddenly receives 50 concurrent requests instead of 1
5. Backend may slow down or fail under the load
6. Slow backend means fetches take longer, more requests pile up
7. Cascading failure begins

Scenarios that trigger thundering herds:

• Hot key expiration: Popular cached item TTL expires
• Cache node failure: All keys on that node suddenly miss
• Cache restart: Cold cache, everything misses
• Bulk invalidation: Many entries invalidated simultaneously
• Synchronized expiration: Multiple entries with same TTL expire together

Mitigation strategies:

You can't optimize what you don't measure. Effective cache management requires continuous monitoring of hit rates and related metrics. Let's explore how to instrument and interpret cache metrics.

Alerting on cache metrics:

Set up alerts for cache health:

• Hit rate drops below threshold: Alert if hit rate falls below 80% (adjust based on baseline)
• Eviction rate spikes: May indicate sudden working set growth or cache pressure
• Memory near limit: Proactive warning before cache starts evicting
• Latency increases: Cache performance degrading, possibly due to contention
• Miss rate spikes: Sudden increase may indicate cache failure or invalidation event

Dashboards and visualization:

Effective cache dashboards show:

1. Hit rate over time (line chart) — Primary health indicator
2. Hit/miss rates (stacked area) — Shows absolute volume and ratio
3. Evictions by reason (stacked bar) — Capacity vs. expiration
4. Memory utilization (gauge) — Current vs. limit
5. Latency percentiles (line chart) — Performance trends