What to Alert On

At 3:17 AM, an engineer's phone buzzes. Another alert. They glance at it through half-open eyes: 'High CPU usage on web-server-42'. Is this critical? Is the system actually degrading? Or is it just a routine spike that will resolve itself? After years of false alarms, they've learned to ignore most alerts. They roll over and go back to sleep.

Two hours later, the real incident strikes. A cascading failure brings down the entire service. The alert that could have caught it early was buried among dozens of meaningless notifications that the on-call engineer had learned to tune out.

This scenario plays out in organizations worldwide, revealing a fundamental truth about alerting: the hardest problem isn't detecting issues—it's deciding what deserves human attention.

Before we discuss what to alert on, we must understand why we alert at all. Alerts exist for one fundamental purpose: to prompt human action when automated systems cannot resolve an issue themselves.

This seemingly simple statement carries profound implications that many organizations overlook.

The Three Questions Every Alert Must Answer

Before creating any alert, ask these fundamental questions:

1. What specific action should the responder take? If you cannot clearly articulate the response, the alert is premature.

2. Why does this require human intervention? If automation could handle it, implement automation instead of alerting.

3. Will delaying response by minutes (or hours) cause material harm? If not, this belongs in a daily review dashboard, not an alert.

The Cost of Every Alert

Alerts are not free. Each one carries costs that must be weighed against benefits:

One of the most fundamental principles in effective alerting is the distinction between symptoms and causes. Understanding this distinction determines whether your alerts detect real problems or merely chase shadows.

Why This Distinction Matters

Cause-based alerts suffer from a fundamental problem: they may indicate issues that never manifest as user impact. A server at 90% CPU might be perfectly healthy under load, or it might be about to fall over. The CPU metric alone cannot tell you which.

Symptom-based alerts, by contrast, directly measure what users experience. If error rates are normal and latency is acceptable, the system is healthy—regardless of what internal metrics suggest.

The Monitoring Hierarchy

Think of monitoring as a diagnostic hierarchy:

┌─────────────────────────────────────────────────────────────────┐
│                    USER EXPERIENCE (Alert Here)                  │
│          Error rates, latency, availability, throughput          │
├─────────────────────────────────────────────────────────────────┤
│                   APPLICATION METRICS                            │
│     Request queues, connection pools, thread states              │
├─────────────────────────────────────────────────────────────────┤
│                  INFRASTRUCTURE METRICS                          │
│         CPU, memory, disk, network, container health             │
├─────────────────────────────────────────────────────────────────┤
│                    HARDWARE METRICS                              │
│       Temperature, fan speed, disk SMART data                    │
└─────────────────────────────────────────────────────────────────┘

Alerts should primarily trigger at the top of this hierarchy. Lower-level metrics are invaluable for investigation after an alert fires, but they rarely should trigger alerts themselves.

The most effective alerting strategy centers on a single question: Is the user experiencing degraded service?

This user-centric approach filters out countless internal metrics that may fluctuate without affecting anyone. It focuses engineering attention on what actually matters: delivering value to customers.

Defining 'User'

The term 'user' is deliberately broad. It encompasses:

• External customers using your product
• Internal teams consuming your platform or APIs
• Automated systems that depend on your services
• Business processes that require your infrastructure

Each of these user categories has different tolerance thresholds and expectations, which should inform your alerting decisions.

Measuring User Impact

To alert on user impact, you must measure user impact. This requires instrumentation at every layer where users interact with your system:

Request-Level Metrics

• Success rate (percentage of requests returning expected results)
• Error rate by type (client errors vs. server errors)
• Latency distributions (p50, p90, p99)
• Throughput (requests per second)

Session-Level Metrics

• Login success rate
• Session duration anomalies
• User flow abandonment rates
• Feature adoption metrics

Business-Level Metrics

• Transaction completion rate
• Revenue per minute/hour
• Order placement success
• Critical business process completion

The most sophisticated approach to 'what to alert on' comes from Service Level Objective (SLO) based alerting. Rather than alerting on arbitrary thresholds, you alert when the system is consuming its error budget at an unsustainable rate.

This approach, pioneered by Google's Site Reliability Engineering practices, fundamentally changes how teams think about alerts.

Understanding Error Budgets

An error budget is the inverse of your SLO. If you promise 99.9% availability, you have a 0.1% error budget. For a 30-day month, this translates to approximately 43 minutes of acceptable downtime.

Why This Matters for Alerting

Traditional alerting asks: 'Is something wrong right now?'

SLO-based alerting asks: 'Are we on track to meet our reliability commitments?'

This shift is profound. A brief spike in error rate that quickly resolves may trigger traditional alerts but wouldn't concern SLO-based alerting—because it barely dented the error budget. Conversely, a slowly increasing error rate that wouldn't trigger percentage-based alerts might be rapidly burning through error budget, warranting immediate attention.

Multiple Burn Rate Windows

Effective SLO-based alerting uses multiple time windows to detect both acute issues and slow burns:

The combination of fast and slow windows catches both sudden failures and gradual degradation that might otherwise slip detection.

Not all alerts are created equal. Understanding the different categories helps you design appropriate responses for each type.

What Deserves a Page?

The highest bar in alerting is the page—an alert that interrupts someone regardless of time of day. Pages should be reserved for situations where:

1. Users are currently impacted or will be within minutes
2. Human intervention is required to resolve the issue
3. Delay will cause escalating harm (more users affected, data loss, revenue impact)
4. The required action is known and can be performed by on-call

If any of these conditions isn't met, the alert shouldn't page.

Learning what not to alert on is as important as learning what to alert on. These anti-patterns are commonly found in organizations struggling with alert fatigue.

With principles established, let's construct a practical framework for deciding what to alert on in your systems.

Step 1: Define Your SLOs

Start with what matters to users and the business. Typical SLO categories include:

• Availability: What percentage of requests should succeed?
• Latency: What response time should users experience?
• Throughput: What transaction volume must the system support?
• Correctness: What data accuracy is required?

Step 2: Instrument User-Facing Metrics

Ensure you're measuring what users experience, not just what servers report. Key metrics:

• Requests handled vs. requests failed (as users see it)
• Time from request receipt to response delivery
• End-to-end transaction success rates
• User-reported errors and complaints

Step 3: Calculate Error Budgets

For each SLO, determine the acceptable error budget and how it's consumed over time. This becomes your alerting baseline.

Step 4: Define Alert Conditions Based on Burn Rate

Create alerts that trigger when error budget consumption threatens your SLO. Use multiple time windows for different severities.

Step 5: Establish Clear Response Expectations

For each alert, document:

• What does this alert mean?
• What should the responder check first?
• What are the likely causes?
• What remediation steps are available?
• When should the responder escalate?