Dashboards and Visualization: Bringing Observability to Life

Modern systems generate thousands of metrics. A single Kubernetes cluster might emit tens of thousands of distinct metric series. An application with detailed instrumentation could produce hundreds of metrics per service. Database servers, message queues, load balancers—each component adds to the deluge.\n\nFaced with this abundance, engineers make one of two mistakes: they either display everything (creating overwhelming dashboards that communicate nothing) or they choose metrics arbitrarily (displaying whatever was easy to add rather than what matters).\n\nThe result is dashboards that fail at their fundamental purpose: communicating system health.\n\nThis page addresses the critical question: Of all the metrics we could display, which ones should we actually put on our dashboards? The answer isn't arbitrary—it's grounded in decades of operational experience distilled into principled methodologies.

Before diving into specific frameworks, let's establish the principles that guide metric selection for dashboards.\n\nMetrics Exist to Answer Questions\n\nEvery metric on a dashboard should answer a specific, actionable question. If you cannot articulate what question a metric answers, it doesn't belong on an operational dashboard.\n\nThe questions that matter fall into a hierarchy:

User-Centricity as North Star\n\nThe most important metrics are those that reflect user experience. A database might show 100% healthy on all its internal metrics while queries time out at the application layer. Internal metrics matter, but user-facing metrics are authoritative.\n\nThis principle determines dashboard hierarchy:\n\n1. User-facing metrics (errors users see, latency users experience) get prime position\n2. Service-level metrics (internal but directly impacting user experience) get secondary position\n3. Resource metrics (CPU, memory, network) support investigation\n4. Infrastructure metrics (kernel stats, hardware) are for deep debugging\n\nFewer Metrics, Better Understanding\n\nCognitive research shows that humans make better decisions with less information—if that information is the right information. A dashboard with 5 carefully chosen metrics outperforms one with 50 random metrics.\n\nThe discipline of metric selection is primarily about what to exclude. Every metric added to a dashboard:\n\n- Competes for attention with other metrics\n- Increases cognitive load\n- May distract from more important signals\n\nEach metric must earn its place by answering a question that matters.

The RED method, developed by Tom Wilkie at Grafana Labs, provides a simple framework for monitoring request-driven services. For every service, measure:\n\nR — Rate: How many requests per second is this service handling?\nE — Errors: How many of those requests are failing?\nD — Duration: How long do those requests take?

Why RED Works\n\nRED captures what users care about in a service:\n\n- Rate tells you if the service is being used (and how much)\n- Errors tell you if the service is working\n- Duration tells you if the service is working well\n\nThese three metrics together provide a complete picture of service health from the user's perspective. They're also easy to collect—most HTTP servers and RPC frameworks can emit these automatically.\n\nRED in Practice

RED Method Variations\n\nBy Endpoint: Track RED metrics for each significant endpoint, not just service aggregate. Login latency matters differently than search latency.\n\nBy Customer Tier: If you have premium customers, track their experience separately. A 1% error rate might mean 100% of enterprise customers are affected.\n\nBy Geography: Users in different regions may experience different conditions. Track RED per region for global services.

The USE method, developed by Brendan Gregg, provides a framework for analyzing resource performance. For every resource, measure:\n\nU — Utilization: What percentage of the resource is being used?\nS — Saturation: How much extra work is queued, waiting for the resource?\nE — Errors: How many error events are occurring on this resource?

Applying USE to Common Resources

The Saturation Signal\n\nSaturation is often the most actionable USE metric. High utilization alone doesn't guarantee problems—a CPU running at 90% utilization might be perfectly healthy if there's no queue building. But non-zero saturation means work is waiting, which directly impacts user experience.\n\nPrioritize saturation visibility:\n\n- Run queue length (CPU saturation) is more actionable than CPU percentage\n- I/O wait indicates disk saturation better than disk utilization\n- Connection wait time reveals database saturation before pool exhaustion\n\nSaturation metrics often predict problems before utilization metrics reveal them.

Google's Site Reliability Engineering book introduced the Four Golden Signals—a framework used internally at Google for monitoring user-facing systems. The signals are:\n\n1. Latency — The time it takes to service a request\n2. Traffic — The demand placed on your system\n3. Errors — The rate of failed requests\n4. Saturation — How 'full' your service is

Latency: Time to Serve\n\nLatency measures the time between request receipt and response delivery. Critical nuances:\n\n- Distinguish successful vs. failed request latency — A fast error isn't a success. Track latency for successful requests separately.\n- Track distributions, not averages — The p99 tells you what the worst-off users experience. An average of 100ms might hide a p99 of 3 seconds.\n- Consider user-perceived latency — Time from user action to visual response matters more than server processing time.\n\nTraffic: Demand on the System\n\nTraffic measures workload in domain-appropriate terms:\n\n- Web services: HTTP requests per second\n- Databases: Queries per second, transactions per second\n- Streaming: Messages per second, bytes per second\n- Storage: IOPS, bytes written per second\n\nTraffic provides context for other signals. High error rate at 10x normal traffic tells a different story than high error rate at normal traffic.\n\nErrors: Failed Requests\n\nError tracking must be comprehensive:\n\n- Explicit errors: HTTP 5xx responses, exception throws\n- Implicit errors: Wrong data returned, timeouts\n- Policy violations: Responses that succeed but violate quality standards (e.g., latency SLO violations)\n\nSaturation: Capacity Utilization\n\nSaturation measures how close the service is to capacity:\n\n- Resource saturation: CPU, memory, disk approaching limits\n- Throughput capacity: Requests per second vs. maximum tested\n- Queue depth: Work waiting to be processed

Different system types require different metric emphases. While RED/USE/Golden Signals provide frameworks, application to specific systems requires domain knowledge.\n\nAPI Services and Web Applications

Databases

Message Queues and Streaming Systems

Caches

Metric selection involves avoiding common traps that reduce dashboard effectiveness.

The Cardinality Problem\n\nMetric cardinality—the number of unique time series generated by a metric—can explode when labels have high variability:\n\n| Label Pattern | Risk | Example |\n|---------------|------|---------|\n| User ID as label | Extreme | request_count{user_id="12345"} — Millions of series |\n| Request ID as label | Extreme | request_duration{request_id="abc123"} — Unbounded |\n| URL path with variables | High | /users/12345/posts/67890 — Many unique paths |\n| Reasonable labels | Safe | status_code, method, service, region — Bounded |\n\nHigh-cardinality metrics consume memory, slow queries, and increase storage costs. Design metrics with bounded label values.

SLO-based dashboards require specific metrics that differ from general operational metrics. The focus shifts from instantaneous values to budget consumption.

Essential SLO Metrics\n\nSLI (Service Level Indicator) — The raw measurement underlying the SLO. For an availability SLO, this might be successful_requests / total_requests. For a latency SLO, this might be requests_under_threshold / total_requests.\n\nCurrent SLO Compliance — Whether the SLI currently meets the objective. Displayed as percentage with color coding against target.\n\nError Budget Remaining — How much failure is acceptable before breaching the SLO. Expressed as percentage remaining or time remaining at current burn rate.\n\nBurn Rate — How fast the error budget is being consumed. A burn rate of 1.0x means pace exactly matches the budget; 2.0x means consuming budget twice as fast as sustainable.\n\nTime Window Performance — SLO compliance over the measurement window (30 days, quarter).

SLO Dashboard Layout\n\nA typical SLO dashboard might display:\n\nTop Row: Status Summary\n- SLO compliance status (green/yellow/red) for each service\n- Error budget remaining percentage\n- Days until budget exhaustion at current rate\n\nSecond Row: Current Performance\n- Current SLI value vs. target\n- Burn rate indicator\n- Trend compared to previous period\n\nThird Row: Time Series\n- SLI over time with SLO threshold line\n- Error budget consumption over the window\n- Burn rate over time\n\nDetail Section: Breakdown\n- SLI by endpoint, region, or customer tier\n- Major contributing factors to budget consumption\n- Incident markers correlated with budget consumption

We've covered the frameworks and principles that guide effective metric selection for dashboards. Let's consolidate the key insights:

What's Next:\n\nWith an understanding of which metrics to display, we need to consider who is viewing the dashboard and what they need. The next page explores service-level dashboards—dashboards designed for engineering teams operating specific services.