Choosing Service Level Indicators

In the realm of Site Reliability Engineering, a fundamental truth often gets obscured by the noise of infrastructure metrics, system dashboards, and technical alerts: reliability is ultimately defined by the user's experience, not by the health of your servers.

Consider a scenario that plays out daily across organizations worldwide: your monitoring dashboards glow green, CPU utilization hovers at a comfortable 40%, memory consumption is well within bounds, network latency between your internal services measures in single-digit milliseconds, and your infrastructure team reports zero incidents. Yet, your customer support channels are flooded with complaints. Users report the application as "slow," "broken," or "unusable." How can this be?

The disconnect stems from a fundamental misalignment between what we measure and what users actually experience. Traditional infrastructure metrics tell you about the health of your systems. User-centric SLIs tell you about the quality of the experience you deliver. These are not the same thing, and conflating them is one of the most common—and costly—mistakes in reliability engineering.

Before diving into specific techniques, we must establish the philosophical foundation that underpins user-centric SLIs. This isn't merely a technical exercise—it represents a fundamental shift in how we conceptualize and measure reliability.

The Traditional Approach: System-Centric Metrics

Historically, operations teams focused on metrics that were easy to collect and directly observable from infrastructure:

• CPU utilization: How busy are our processors?
• Memory consumption: How much RAM is in use?
• Disk I/O: How fast can we read and write data?
• Network bandwidth: How much data flows through our pipes?
• Process health: Are all our services running?

These metrics have value—they're essential for capacity planning, debugging, and understanding system behavior. However, they suffer from a critical limitation: they measure means, not ends.

The User-Centric Paradigm Shift

User-centric SLIs flip this model on its head. Instead of asking "How healthy are our systems?" we ask:

• Can users complete their intended tasks?
• How quickly do users perceive responses?
• How often do users encounter errors or degraded experiences?
• What percentage of user attempts succeed?

This shift has profound implications. It means our SLIs must be derived from user journeys, not system topologies. It means we need to understand user expectations and translate them into measurable quantities. And it means accepting that a "healthy" system causing poor user experience is, by definition, not actually healthy.

The Observability Chain

Think of reliability observability as a chain with multiple links:

1. Infrastructure metrics: CPU, memory, disk, network
2. Service metrics: Request rates, error rates, latency distributions
3. Business metrics: Transactions completed, revenue processed, users served
4. User experience metrics: Perceived performance, task completion, satisfaction

Each link in this chain gets us closer to what ultimately matters. User-centric SLIs position us at the right end of this chain—measuring outcomes rather than intermediaries.

Effective user-centric SLIs begin with a deep understanding of how users interact with your system. This requires mapping user journeys—the sequences of interactions users perform to accomplish their goals.

What Is a User Journey?

A user journey is the complete sequence of steps a user takes to achieve a specific outcome. It encompasses every touchpoint, from initial intent through final confirmation. Consider an e-commerce checkout:

1. User adds items to cart
2. User navigates to checkout
3. User enters shipping information
4. User selects shipping method
5. User enters payment details
6. User reviews order summary
7. User confirms purchase
8. User receives confirmation

Each step in this journey represents an opportunity for the system to succeed or fail from the user's perspective. A user-centric SLI might measure the end-to-end success rate of checkout completions—not just whether individual API endpoints responded.

Identifying Critical User Journeys

Not all user journeys carry equal weight. To prioritize SLI development, categorize journeys by their criticality:

Tier 1 - Revenue-Critical Journeys These directly impact revenue or core business operations. For an e-commerce platform: search, product viewing, checkout, payment processing. For a SaaS product: core workflow completion, data export, API access for paying customers.

Tier 2 - Experience-Critical Journeys These significantly impact user satisfaction and retention but may not directly generate revenue. Examples: account settings modification, preference management, notification delivery, report generation.

Tier 3 - Auxiliary Journeys These support the overall product but are non-essential to core value delivery. Examples: social features, gamification elements, optional integrations, cosmetic customizations.

Your SLI strategy should ensure Tier 1 journeys have the most rigorous measurement, with coverage decreasing through lower tiers.

From Journeys to Measurements

Once you've identified critical user journeys, the next step is translating them into measurable quantities. This requires defining:

1. Success criteria: What constitutes successful completion of this journey?
2. Failure modes: What are the ways this journey can fail?
3. Measurement points: Where in your system can you observe success/failure?
4. Latency expectations: How long should this journey reasonably take?
5. Quality dimensions: Beyond success/failure, what quality aspects matter?

For the e-commerce checkout example:

• Success criteria: Order confirmed, payment captured, confirmation email sent
• Failure modes: Payment declined (not a system failure), timeout during processing, inventory unavailable after cart, system error during confirmation
• Measurement points: Transaction completion event in payment service, confirmation email delivery, order record creation
• Latency expectations: Entire checkout should complete within 30 seconds; payment processing specifically within 5 seconds
• Quality dimensions: Order accuracy, confirmation timeliness, receipt correctness

Understanding how users perceive reliability is essential for choosing SLIs that genuinely reflect experience quality. User perception doesn't perfectly correlate with objective measurements—humans have psychological biases, varying expectations, and contextual interpretations that must be accounted for.

The Psychology of Waiting

Research in human-computer interaction has established that users' perception of time is heavily influenced by context:

• Active waits feel shorter than passive waits: Users watching a progress indicator perceive less time passing than users staring at a blank screen.
• Explained waits feel shorter than unexplained waits: "Processing your payment securely..." feels faster than a generic spinner.
• Uncertain waits feel longer: Not knowing if something is working causes anxiety that stretches perceived time.
• Interrupted waits feel much longer: If a process appears to fail and requires retry, the perceived duration multiplies.

These psychological factors mean that a 3-second operation with good feedback may feel faster than a 2-second operation with no feedback. Your SLIs should account for this—measuring not just raw latency but the quality of the waiting experience.

User Expectations Vary by Context

User tolerance for latency and failure depends heavily on context:

High Stakes = Low Tolerance When users are performing critical actions—financial transactions, medical data access, legal document submission—their tolerance for errors and delays plummets. An error during payment processing is vastly more frustrating than an error loading a news article. Your SLIs should reflect this by having stricter targets for high-stakes journeys.

Frequency Affects Frustration An interaction that's slow once is an annoyance. An interaction that's slow every time becomes infuriating. If users perform an action 50 times per day, even small latency issues compound into significant productivity loss. Frequent interactions often warrant tighter latency SLIs than one-time actions.

Alternatives Change Expectations User expectations are shaped by competitive alternatives. If competing products complete the same action in 200ms, your 2-second latency—even if technically acceptable—becomes a competitive disadvantage. Benchmark SLIs against competitor performance when possible.

Prior Experience Sets Baselines User satisfaction is relative to their expectations. If your service has historically responded in 500ms, users adapt to this baseline. Improvements to 200ms delight users; regressions to 800ms frustrate them. SLIs should protect against regression from established baselines, not just breach of absolute thresholds.

The bridge between subjective user experience and objective SLIs requires careful translation. This section provides frameworks for converting qualitative user expectations into quantitative measurements.

The User Story to SLI Translation Process

Start with user stories that express expectations in natural language, then progressively refine them into measurable indicators:

Stage 1: Qualitative User Statement

"As a customer, I want my search results to appear quickly so I can find products efficiently."

Stage 2: Quantifiable User Expectation

"Search results should feel instant—I shouldn't notice waiting."

Stage 3: Technical Translation

Based on perception research, "feel instant" maps to < 300ms for the first meaningful results.

Stage 4: Measurement Definition

Search latency SLI: Time from user query submission to first 10 results rendered in user's browser.

Stage 5: SLI Specification

95th percentile search latency < 300ms, measured client-side from query keypress to first result render.

Notice how each stage adds precision while maintaining connection to the original user need. The final SLI can be measured objectively, yet it directly traces back to user experience goals.

The Four Golden Signals for User Experience

Google's Site Reliability Engineering book popularized the "Four Golden Signals" for monitoring. Let's reframe these from a user-centric perspective:

1. Latency (User-Centric Framing: Responsiveness) Not just "how long do requests take?" but "how long do users wait for meaningful responses?" This means:

• Measuring time to first byte AND time to interactive
• Separating successful request latency from error latency (users experience slow errors differently)
• Weighting by user-facing criticality

2. Traffic (User-Centric Framing: User Activity) Not just "requests per second" but "how many users are successfully engaging?" This means:

• Tracking unique user sessions, not just raw requests
• Distinguishing organic traffic from synthetic monitoring
• Understanding traffic composition (new vs. returning users, premium vs. free)

3. Errors (User-Centric Framing: Success Rate) Not just "what percentage of HTTP requests return 5xx?" but "what percentage of user attempts succeed?" This means:

• Including client errors caused by poor UX (400 errors from confusing forms)
• Treating timeouts as errors (users waited and got nothing)
• Considering partial failures (page loaded but core function broken)

4. Saturation (User-Centric Framing: Capacity Headroom) Not just "how utilized are our resources?" but "how close are we to degrading user experience?" This means:

• Defining saturation in terms of latency impact, not resource percentage
• Identifying the resource that will degrade user experience first
• Building capacity models that predict user impact

The location where you measure dramatically affects what your SLI captures. The closer you measure to the user, the more accurately your SLI reflects true user experience—but client-side measurement also introduces complexity. Understanding this tradeoff is essential.

The Measurement Spectrum

Imagine a typical web request flowing through your infrastructure:

User Browser → CDN → Load Balancer → API Gateway → Application Server → Database

Each point along this path offers a potential measurement location, with different tradeoffs:

Real User Monitoring (RUM) Deep Dive

Real User Monitoring instruments actual user browsers to capture performance data as users experience it. This provides unparalleled accuracy but introduces implementation challenges:

Advantages of RUM for SLIs:

• Captures true end-to-end latency including rendering
• Reflects geographic distribution of your user base
• Accounts for device and network variability
• Includes client-side JavaScript execution time
• Reflects actual usage patterns, not synthetic scenarios

Challenges with RUM:

• Not all users have JavaScript enabled or functional
• Data is sampled, not comprehensive (cost/performance tradeoff)
• Client clock skew can affect timing accuracy
• Data may be blocked by ad blockers or privacy settings
• Requires careful aggregation to be meaningful

Key RUM Metrics for SLIs:

• First Contentful Paint (FCP): When users first see content
• Largest Contentful Paint (LCP): When main content is visible
• Time to Interactive (TTI): When users can meaningfully interact
• First Input Delay (FID): Responsiveness of first user interaction
• Cumulative Layout Shift (CLS): Visual stability during loading

These Web Vitals metrics map directly to user experience goals and should form the basis of user-centric SLIs.

Synthetic Monitoring as a Complement

While RUM captures real user experience, synthetic monitoring provides consistent baselines:

• Run identical tests from consistent locations and conditions
• Detect issues before real users encounter them
• Enable meaningful comparison over time (control for user variability)
• Provide 24/7 coverage even during low-traffic periods

Best practice: Use synthetic monitoring to detect availability issues and regressions quickly, use RUM to understand true user experience and set SLO targets.

Even with the best intentions, teams often fall into patterns that undermine the user-centricity of their SLIs. Recognizing these anti-patterns helps you avoid them.

Anti-Pattern 1: Measuring Averages Instead of Percentiles

Average latency is deeply misleading. Consider two scenarios:

• Scenario A: 99 requests at 100ms, 1 request at 10,000ms → Average: 199ms
• Scenario B: 100 requests all at 199ms → Average: 199ms

Both have identical averages, but Scenario A has a catastrophically bad experience for 1% of users. If you're processing 1 million requests daily, that's 10,000 users per day having a terrible experience—invisible in your average.

The Fix: Always use percentiles. The 50th percentile (median) shows "typical" experience. The 95th and 99th percentiles reveal what your worst-affected users experience. For user-centric SLIs, the 99th percentile often matters most.

Anti-Pattern 2: The "Available Means Pingable" Trap

Many teams define availability as "the load balancer health check passes." This dramatically overstates availability from a user perspective. A service might be pingable while:

• Database connections are exhausted
• Critical downstream services are failing
• Authentication is broken
• Core workflows are throwing errors
• Performance is degraded beyond usability

The Fix: Define availability in terms of user capability. "Available" means "users can accomplish their primary goals." This often requires synthetic transactions that exercise full user workflows, not just ping health endpoints.

Anti-Pattern 3: Excludingowing Exclusions

Teams often exclude various failures from SLI calculations:

• "That was a dependency failure, not our fault"
• "Those errors were from a bad deploy we rolled back"
• "That outage was during our maintenance window"
• "Those timeouts were users with slow connections"

From the user's perspective, all of these were failures. User-centric SLIs should include everything users experience as failures.

The Fix: Include all failures in your SLI. If you have legitimate exclusions (planned maintenance with notice, user-caused issues), document them explicitly and require approval. The default should be inclusion, not exclusion.

Anti-Pattern 4: Over-Aggregation

A single "availability" SLI for your entire platform obscures problems in specific journeys. If product search is failing but checkout is working, an aggregated availability metric might still look acceptable—while users trying to find products are having a terrible experience.

The Fix: Create distinct SLIs for distinct user journeys. Roll these up into summary metrics for executive dashboards, but preserve the granular signals for operational use.

With theoretical foundations established, let's walk through a practical framework for implementing user-centric SLIs in a real system.

Step 1: Enumerate User Journeys

Start by creating a comprehensive inventory of user journeys. For each journey, document:

• Journey name and description
• User persona(s) who perform this journey
• Business criticality (Tier 1/2/3)
• Expected frequency of use
• User-facing success criteria
• Known pain points or historical issues

Step 2: Define Success and Failure

For each critical journey, precisely define what constitutes success and failure from the user's perspective. Be explicit about edge cases:

Example: E-Commerce Product Search

• Success: User receives relevant search results within 1 second of query submission
• Failure - Error: Zero results due to system error (not empty results from no matches)
• Failure - Timeout: User waited > 5 seconds with no response
• Failure - Degraded: Results rendered but pagination or filters non-functional
• Excluded: Empty results from legitimate query with no matching products

Step 3: Establish Measurement Infrastructure

Implement the instrumentation to collect your defined SLIs:

1. Client-side instrumentation: Add Real User Monitoring to capture page load performance, interaction latency, and client-side errors. Use the Web Vitals API for standardized metrics.

2. Server-side observability: Ensure structured logging and distributed tracing are in place to correlate client observations with server behavior.

3. Synthetic probes: Deploy synthetic monitoring that exercises critical user journeys from multiple geographic locations.

4. Data pipeline: Build aggregation pipelines that compute SLI values from raw metrics at appropriate intervals (typically 1-5 minute windows).

Step 4: Validate Against User Feedback

Before finalizing SLI targets, validate that your measurements correlate with actual user satisfaction:

• Compare SLI values during periods of known user complaints to baseline periods
• Correlate SLI degradation with support ticket volume
• Survey users about their experience and cross-reference with SLI data
• Review NPS trends against SLI performance over time

If your SLI shows good performance during periods users report poor experience, your SLI isn't measuring what matters—refine it.

Step 5: Document and Communicate

User-centric SLIs are most valuable when understood beyond the SRE team:

• Publish SLI definitions in a central wiki accessible to product and engineering teams
• Include SLIs in regular product review meetings
• Create dashboards that show SLI status alongside business metrics
• Train product managers to interpret SLI data for roadmap prioritization

User-centric SLIs represent a fundamental shift from measuring what's convenient to measuring what matters. Let's consolidate the key principles:

What's Next

With the philosophical foundation of user-centric measurement established, we're ready to explore specific categories of SLIs. In the next page, we'll dive deep into Availability SLIs—how to measure and reason about whether your service is truly available to users, including the subtle distinctions between uptime, reachability, and functionality.