Alerting Design: Building Effective Alert Systems

At 3:17 AM, an engineer's phone buzzes. Another alert. They glance at it through half-open eyes: 'High CPU usage on web-server-42'. Is this critical? Is the system actually degrading? Or is it just a routine spike that will resolve itself? After years of false alarms, they've learned to ignore most alerts. They roll over and go back to sleep.\n\nTwo hours later, the real incident strikes. A cascading failure brings down the entire service. The alert that could have caught it early was buried among dozens of meaningless notifications that the on-call engineer had learned to tune out.\n\nThis scenario plays out in organizations worldwide, revealing a fundamental truth about alerting: the hardest problem isn't detecting issues—it's deciding what deserves human attention.

Before we discuss what to alert on, we must understand why we alert at all. Alerts exist for one fundamental purpose: to prompt human action when automated systems cannot resolve an issue themselves.\n\nThis seemingly simple statement carries profound implications that many organizations overlook.

The Three Questions Every Alert Must Answer\n\nBefore creating any alert, ask these fundamental questions:\n\n1. What specific action should the responder take? If you cannot clearly articulate the response, the alert is premature.\n\n2. Why does this require human intervention? If automation could handle it, implement automation instead of alerting.\n\n3. Will delaying response by minutes (or hours) cause material harm? If not, this belongs in a daily review dashboard, not an alert.\n\nThe Cost of Every Alert\n\nAlerts are not free. Each one carries costs that must be weighed against benefits:

One of the most fundamental principles in effective alerting is the distinction between symptoms and causes. Understanding this distinction determines whether your alerts detect real problems or merely chase shadows.

Why This Distinction Matters\n\nCause-based alerts suffer from a fundamental problem: they may indicate issues that never manifest as user impact. A server at 90% CPU might be perfectly healthy under load, or it might be about to fall over. The CPU metric alone cannot tell you which.\n\nSymptom-based alerts, by contrast, directly measure what users experience. If error rates are normal and latency is acceptable, the system is healthy—regardless of what internal metrics suggest.\n\nThe Monitoring Hierarchy\n\nThink of monitoring as a diagnostic hierarchy:\n\n\n┌─────────────────────────────────────────────────────────────────┐\n│ USER EXPERIENCE (Alert Here) │\n│ Error rates, latency, availability, throughput │\n├─────────────────────────────────────────────────────────────────┤\n│ APPLICATION METRICS │\n│ Request queues, connection pools, thread states │\n├─────────────────────────────────────────────────────────────────┤\n│ INFRASTRUCTURE METRICS │\n│ CPU, memory, disk, network, container health │\n├─────────────────────────────────────────────────────────────────┤\n│ HARDWARE METRICS │\n│ Temperature, fan speed, disk SMART data │\n└─────────────────────────────────────────────────────────────────┘\n\n\nAlerts should primarily trigger at the top of this hierarchy. Lower-level metrics are invaluable for investigation after an alert fires, but they rarely should trigger alerts themselves.

The most effective alerting strategy centers on a single question: Is the user experiencing degraded service?\n\nThis user-centric approach filters out countless internal metrics that may fluctuate without affecting anyone. It focuses engineering attention on what actually matters: delivering value to customers.

Defining 'User'\n\nThe term 'user' is deliberately broad. It encompasses:\n\n- External customers using your product\n- Internal teams consuming your platform or APIs\n- Automated systems that depend on your services\n- Business processes that require your infrastructure\n\nEach of these user categories has different tolerance thresholds and expectations, which should inform your alerting decisions.

Measuring User Impact\n\nTo alert on user impact, you must measure user impact. This requires instrumentation at every layer where users interact with your system:\n\nRequest-Level Metrics\n- Success rate (percentage of requests returning expected results)\n- Error rate by type (client errors vs. server errors)\n- Latency distributions (p50, p90, p99)\n- Throughput (requests per second)\n\nSession-Level Metrics\n- Login success rate\n- Session duration anomalies\n- User flow abandonment rates\n- Feature adoption metrics\n\nBusiness-Level Metrics\n- Transaction completion rate\n- Revenue per minute/hour\n- Order placement success\n- Critical business process completion

The most sophisticated approach to 'what to alert on' comes from Service Level Objective (SLO) based alerting. Rather than alerting on arbitrary thresholds, you alert when the system is consuming its error budget at an unsustainable rate.\n\nThis approach, pioneered by Google's Site Reliability Engineering practices, fundamentally changes how teams think about alerts.

Understanding Error Budgets\n\nAn error budget is the inverse of your SLO. If you promise 99.9% availability, you have a 0.1% error budget. For a 30-day month, this translates to approximately 43 minutes of acceptable downtime.\n\nWhy This Matters for Alerting\n\nTraditional alerting asks: 'Is something wrong right now?'\n\nSLO-based alerting asks: 'Are we on track to meet our reliability commitments?'\n\nThis shift is profound. A brief spike in error rate that quickly resolves may trigger traditional alerts but wouldn't concern SLO-based alerting—because it barely dented the error budget. Conversely, a slowly increasing error rate that wouldn't trigger percentage-based alerts might be rapidly burning through error budget, warranting immediate attention.

Multiple Burn Rate Windows\n\nEffective SLO-based alerting uses multiple time windows to detect both acute issues and slow burns:\n\n| Window | Burn Rate | Meaning | Response |\n|--------|-----------|---------|----------|\n| 5 min | 14.4x | Severe incident in progress | Page immediately |\n| 1 hour | 6x | Significant issue developing | Page with urgency |\n| 6 hours | 1.5x | Slow burn consuming budget | Ticket for prompt investigation |\n| 3 days | 1x | Gradual degradation | Weekly review item |\n\nThe combination of fast and slow windows catches both sudden failures and gradual degradation that might otherwise slip detection.

Not all alerts are created equal. Understanding the different categories helps you design appropriate responses for each type.

What Deserves a Page?\n\nThe highest bar in alerting is the page—an alert that interrupts someone regardless of time of day. Pages should be reserved for situations where:\n\n1. Users are currently impacted or will be within minutes\n2. Human intervention is required to resolve the issue\n3. Delay will cause escalating harm (more users affected, data loss, revenue impact)\n4. The required action is known and can be performed by on-call\n\nIf any of these conditions isn't met, the alert shouldn't page.

Learning what not to alert on is as important as learning what to alert on. These anti-patterns are commonly found in organizations struggling with alert fatigue.

With principles established, let's construct a practical framework for deciding what to alert on in your systems.

Step 1: Define Your SLOs\n\nStart with what matters to users and the business. Typical SLO categories include:\n\n- Availability: What percentage of requests should succeed?\n- Latency: What response time should users experience?\n- Throughput: What transaction volume must the system support?\n- Correctness: What data accuracy is required?\n\nStep 2: Instrument User-Facing Metrics\n\nEnsure you're measuring what users experience, not just what servers report. Key metrics:\n\n- Requests handled vs. requests failed (as users see it)\n- Time from request receipt to response delivery\n- End-to-end transaction success rates\n- User-reported errors and complaints\n\nStep 3: Calculate Error Budgets\n\nFor each SLO, determine the acceptable error budget and how it's consumed over time. This becomes your alerting baseline.\n\nStep 4: Define Alert Conditions Based on Burn Rate\n\nCreate alerts that trigger when error budget consumption threatens your SLO. Use multiple time windows for different severities.\n\nStep 5: Establish Clear Response Expectations\n\nFor each alert, document:\n- What does this alert mean?\n- What should the responder check first?\n- What are the likely causes?\n- What remediation steps are available?\n- When should the responder escalate?

We've covered the foundational principles for designing effective alerts. Let's consolidate the key insights:

What's Next:\n\nNow that we understand what to alert on, we need to understand how to determine when an alert should fire. The next page explores alert thresholds—the art and science of setting boundaries that catch real problems while avoiding false positives.