Incident Response Process — Incident Management | OneNoughtOne

0/107

System Design (HLD)SLOs, SLIs & Incident Management

Incident Management

LevelAdvanced

Duration90 mins

TopicSLOs, SLIs & Incident Management

2 / 5

Incident Response Process

When Chaos Meets Structure

It's 3:17 AM. Your phone buzzes with a PagerDuty alert: "CRITICAL: Payment Service Error Rate > 5%." You check the dashboard—error rate is climbing, now at 8%. Customers can't complete purchases. Your company loses roughly $50,000 per minute when checkout is broken. What happens next determines whether this incident resolves in 15 minutes or 3 hours. Whether it's a controlled response or a chaotic scramble. Whether your team learns from the experience or repeats the same mistakes. The difference isn't luck or individual heroics—it's process . Organizations that handle incidents well have internalized a structured response framework that channels the urgency of a crisis into coordinated, effective action. This page explores that framework in depth.

What You Will Learn

By the end of this page, you will understand the complete incident response lifecycle: the phases from detection through resolution and beyond, the roles that participate in response, the coordination mechanisms that enable effective parallel work, and the documentation practices that capture learning for the future. You'll be able to design and implement incident response processes that scale from five-person startups to thousand-person enterprises.

The Incident Response Lifecycle

Every incident, regardless of severity or duration, follows a predictable lifecycle. Understanding this lifecycle is fundamental to effective response—it provides a mental model for where you are, what should happen next, and what success looks like at each stage. The Five Phases of Incident Response Detection & Triage : Recognizing an incident exists and determining its initial scope Escalation & Mobilization : Engaging the right people and resources Investigation & Diagnosis : Understanding root cause and impact Mitigation & Resolution : Stopping the bleeding and restoring service Recovery & Learning : Post-incident activities including follow-up and prevention

Converting Mermaid diagram...

Phase Transitions Transitions between phases aren't always linear. Investigation may reveal the need for additional escalation. A mitigation attempt may fail, cycling back to investigation. Resolution may be partial, requiring iteration. Experienced responders recognize these phase transitions and communicate them clearly: "We've triaged this as SEV-1. Escalating to full incident response." "Mitigation attempt failed. We're back in investigation mode." "Service is partially restored. Entering recovery phase for the remaining issues." Naming the phase explicitly helps the team maintain shared understanding of where they are in the process.

The Golden Hour Concept

Borrowed from emergency medicine: the first 60 minutes of incident response are disproportionately impactful. Actions taken (or not taken) in this window often determine whether resolution takes 2 hours or 12 hours. Prioritize getting the right people engaged and stabilizing the situation over perfect diagnosis during this critical window.

Incident Response Roles

Effective incident response requires clear role definition. Without explicit roles, you get either paralysis (everyone waiting for someone else) or chaos (everyone doing everything, stepping on each other). Mature incident response frameworks define several distinct roles: Core Incident Response Roles

Incident Response Roles and Responsibilities
Role	Primary Focus	Key Responsibilities	Required Skills
Incident Commander (IC)	Overall coordination	Owns the incident, coordinates work, makes decisions, drives toward resolution	Leadership, communication, technical breadth, calm under pressure
Technical Lead	Technical investigation	Leads diagnosis, proposes mitigations, validates fixes, coordinates SMEs	Deep system knowledge, debugging skills, architectural understanding
Communications Lead	Stakeholder updates	Updates status page, notifies stakeholders, manages external messaging	Clear writing, stakeholder awareness, timing judgment
Scribe	Documentation	Records timeline, actions, decisions; maintains incident channel log	Attention to detail, fast typing, synthesis skills
Subject Matter Expert	Domain expertise	Provides deep knowledge on specific systems, answers technical questions	Specialized system expertise
Customer Liaison	Customer impact	Monitors support channels, escalates customer reports, coordinates response	Customer empathy, support knowledge, communication

The Incident Commander Role The Incident Commander (IC) is the most critical role in incident response. This person is responsible for the incident as a whole—not for solving the technical problem personally, but for ensuring the incident progresses toward resolution. IC Responsibilities: Maintain Situational Awareness : Know current status, who's working on what, and what's pending Coordinate Parallel Workstreams : Assign investigation tasks and prevent duplication Make Decisions : When the team is stuck deciding between options, the IC decides Manage Timeline : Ensure regular status updates and progress checks Escalate When Needed : Bring in additional expertise or leadership involvement Protect the Team : Shield responders from distractions and interruptions Drive to Closure : Push for action over analysis paralysis; know when to declare resolution IC Anti-Patterns: Getting pulled into hands-on-keyboard debugging (delegate technical work) Making unilateral technical decisions without consulting SMEs Disappearing from the incident channel for extended periods Failing to delegate documentation and communication Continuing incident mode after the problem is resolved

Role Flexibility

For smaller incidents, one person may fill multiple roles. For major incidents, each role might have a dedicated person or even a sub-team. The key is explicit assignment: everyone should know who's doing what. Never assume—always state clearly: 'I'll take IC' or 'Can someone volunteer as scribe?'

Incident Declaration and Triage

Not every alert is an incident. The triage phase determines whether a detected issue warrants incident response activation. This decision has significant implications: declaring an incident too readily leads to chaos and fatigue; hesitating to declare leads to delayed response and extended impact. Incident Declaration Criteria An incident should generally be declared when one or more of the following are true: User Impact : Customers are experiencing degraded service or outage Revenue Impact : Business transactions are failing or significantly delayed Data Risk : Data integrity, availability, or confidentiality is threatened Regulatory Risk : Compliance or legal obligations may be violated Reputational Risk : The issue may become publicly visible Escalating Scope : The problem is growing worse or spreading to other systems Complex Resolution : Multiple teams or extended effort will be required

incident-triage-decision.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
/**
 * Incident Triage Decision Framework
 * 
 * This framework helps responders systematically evaluate
 * whether an issue warrants incident declaration and what
 * severity level to assign.
 */
 
interface TriageInput {
  // Impact assessment
  userImpact: 'none' | 'degraded' | 'partial-outage' | 'full-outage';
  affectedUserPercentage: number;
  revenueImpact: 'none' | 'minor' | 'significant' | 'major';
  
  // Scope assessment
  affectedServices: string[];
  isEscalating: boolean;
  estimatedResolutionTime: 'minutes' | 'hours' | 'uncertain';
  
  // Risk assessment
  dataAtRisk: boolean;
  securityImplications: boolean;
  regulatoryImplications: boolean;
  
  // Context
  recentDeployments: boolean;
  knownIssue: boolean;
  existingIncident: boolean;
}
 
interface TriageDecision {
  declareIncident: boolean;
  recommendedSeverity: 'SEV-1' | 'SEV-2' | 'SEV-3' | 'SEV-4' | null;
  rationale: string;
  immediateActions: string[];
  escalationTargets: string[];
}
 
function triageIssue(input: TriageInput): TriageDecision {
  // Critical path: Full outage with user/revenue impact
  if (input.userImpact === 'full-outage' && input.revenueImpact !== 'none') {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-1',
      rationale: 'Complete service outage affecting revenue',
      immediateActions: [
        'Page all on-call responders for affected services',
        'Open incident bridge/channel immediately',
        'Begin customer communication preparation',
        'Alert leadership within 5 minutes',
      ],
      escalationTargets: [
        'Primary and secondary on-call',
        'Engineering leadership',
        'Customer success leadership',
      ],
    };
  }
 
  // Security or data breach
  if (input.securityImplications || input.dataAtRisk) {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-1',
      rationale: 'Security incident or data at risk requires immediate response',
      immediateActions: [
        'Engage security team immediately',
        'Consider isolation/containment measures',
        'Begin evidence preservation',
        'Alert legal/compliance if data breach suspected',
      ],
      escalationTargets: [
        'Security on-call',
        'CISO or security leadership',
        'Legal/compliance if needed',
      ],
    };
  }
 
  // Partial outage or significant degradation
  if (
    input.userImpact === 'partial-outage' ||
    (input.userImpact === 'degraded' && input.affectedUserPercentage > 25)
  ) {
    const severity = input.isEscalating || input.revenueImpact === 'significant' 
      ? 'SEV-1' 
      : 'SEV-2';
    
    return {
      declareIncident: true,
      recommendedSeverity: severity,
      rationale: 'Significant user impact exceeds acceptable thresholds',
      immediateActions: [
        'Page primary on-call for affected services',
        'Open incident channel',
        'Begin initial investigation',
        'Prepare customer status update',
      ],
      escalationTargets: [
        'Primary on-call for affected services',
        'Engineering manager if escalating',
      ],
    };
  }
 
  // Multi-service impact
  if (input.affectedServices.length >= 3) {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-2',
      rationale: 'Widespread impact across multiple services',
      immediateActions: [
        'Page on-call for all affected services',
        'Open incident channel for coordination',
        'Look for common cause (shared dependency)',
      ],
      escalationTargets: input.affectedServices.map(s => `${s} on-call`),
    };
  }
 
  // Known issue with workaround
  if (input.knownIssue && input.userImpact === 'degraded') {
    return {
      declareIncident: false,
      recommendedSeverity: null,
      rationale: 'Known issue with understood impact and workaround',
      immediateActions: [
        'Monitor for escalation',
        'Ensure workaround documentation is current',
        'Track toward permanent resolution',
      ],
      escalationTargets: [],
    };
  }
 
  // Minor degradation
  if (input.userImpact === 'degraded' && input.affectedUserPercentage <= 5) {
    return {
      declareIncident: input.estimatedResolutionTime !== 'minutes',
      recommendedSeverity: 'SEV-3',
      rationale: 'Minor impact, investigate without full incident response unless prolonged',
      immediateActions: [
        'Primary team investigate',
        'Monitor for escalation',
        'Create ticket for tracking',
      ],
      escalationTargets: ['Primary team on-call'],
    };
  }
 
  // Default: Don't declare, but investigate
  return {
    declareIncident: false,
    recommendedSeverity: null,
    rationale: 'Issue does not meet incident criteria; handle through normal operations',
    immediateActions: [
      'Create ticket for tracking',
      'Assign to appropriate team',
      'Monitor for changes in scope',
    ],
    escalationTargets: [],
  };
}
 
// Usage example
const triageResult = triageIssue({
  userImpact: 'partial-outage',
  affectedUserPercentage: 30,
  revenueImpact: 'significant',
  affectedServices: ['checkout', 'payment-gateway'],
  isEscalating: true,
  estimatedResolutionTime: 'uncertain',
  dataAtRisk: false,
  securityImplications: false,
  regulatoryImplications: false,
  recentDeployments: true,
  knownIssue: false,
  existingIncident: false,
});
 
console.log('Triage Decision:', triageResult);
// Output: SEV-1 incident, page all responders, recent deployment likely cause

The Cost of Hesitation

When in doubt, declare the incident. An over-declared incident can be quickly downgraded or closed. An under-declared incident results in delayed response, extended impact, and frustrated teams who weren't informed early enough. It's much easier to scale down than to catch up.

Investigation and Diagnosis

Once an incident is declared, the team must understand what's happening before they can fix it. Investigation is the phase where hypotheses are formed and tested, data is gathered, and root cause is identified. Effective investigation is methodical rather than chaotic—even under pressure. The Investigation Framework Systematic investigation follows a structured approach: What changed? Start with recent changes: deployments, config changes, infrastructure updates, traffic patterns What's affected? Map the blast radius: which services, users, regions, or features are impacted What's the pattern? Look for correlations: timing, user segments, error types, specific endpoints What do the observability signals show? Metrics, logs, and traces tell the story—follow them What's the simplest explanation? Apply Occam's Razor before pursuing exotic theories

Effective Investigation Patterns

•Parallel Paths: Divide investigation across responders (one checks recent deploys, another checks logs)
•Time Boxing: Spend 5-10 minutes on a hypothesis before switching approaches if no progress
•Progressive Narrowing: Start broad (which service?), then narrow (which endpoint? which user type?)
•Seek Contradictions: Look for evidence that disproves your hypothesis, not just confirms it
•Document as You Go: Log findings in the incident channel so nothing is lost
•Think Out Loud: Share reasoning so others can validate or redirect

Investigation Anti-Patterns

•Tunnel Vision: Fixating on the first hypothesis without considering alternatives
•Silent Investigation: Working in isolation without sharing findings
•Tool Distraction: Spending excessive time building queries rather than interpreting results
•Blame Seeking: Focusing on 'who did this' instead of 'what's happening'
•Analysis Paralysis: Pursuing perfect understanding before taking any mitigation action
•Deployment Obsession: Blaming deploys without evidence when other causes are equally likely

The 'Recent Changes' Checklist The majority of incidents are caused by changes. When triaging, quickly review: Code Deployments : What was deployed in the last 24 hours? The last 2 hours? Configuration Changes : Any feature flags, environment variables, or config updates? Infrastructure Changes : New instances, network changes, certificate rotations? Dependency Updates : Did any upstream services or libraries change? Traffic Patterns : Is there unusual traffic volume, source, or composition? External Factors : Is there third-party maintenance, regional outages, or attack patterns? A deployment timeline correlated with the incident onset time often reveals the trigger immediately.

The Five Whys Technique

For complex incidents, apply the 'Five Whys' iteratively: • Why are users seeing errors? Because the API is timing out. • Why is the API timing out? Because database queries are slow. • Why are queries slow? Because the table lacks an index on the new column. • Why is the column unindexed? Because migration didn't include index creation. • Why wasn't the index included? Because our migration process doesn't review for performance. This reveals root causes beyond the immediate symptom.

Mitigation and Resolution

Mitigation is the phase where action replaces investigation. The goal shifts from understanding the problem to stopping the bleeding. In many incidents, mitigation can and should begin before diagnosis is complete—if there's a safe action that might help, take it. The Mitigation Mindset Key principles for effective mitigation: Restore First, Root Cause Later : Don't delay customer recovery to achieve perfect understanding. Rollback now; post-mortem later. Reversible Actions First : Prefer actions you can undo. A rollback is easily reversed; a database schema change is not. Small Blast Radius Before Large : If trying a fix in production, limit exposure: one region, one pod, one percent of traffic. Communicate Before Acting : Announce mitigation actions in the incident channel: "Rolling back the 2:15 PM deploy now." Verify After Acting : Confirm the mitigation worked: "Error rate dropping. Now at 2%, down from 12%."

Common Mitigation Strategies
Strategy	When to Use	Risks/Considerations	Reversibility
Rollback Deployment	Recent deploy correlated with issue onset	Ensure rollback process is tested; may need to rollback database migrations first	Usually reversible (re-deploy)
Disable Feature Flag	New feature causing problems	Fast and surgical; ensure flag controls the suspected code path	Easily reversible
Scale Up Resources	Capacity exhaustion (CPU, memory, connections)	May mask the symptom; doesn't address root cause	Easily reversible
Failover to Backup	Primary system unhealthy	Verify backup is current and functional before switching	Usually reversible (fail back)
Block Bad Traffic	DDoS or abuse pattern identified	May block legitimate users if criteria too broad; may block legitimate traffic	Easily reversible
Restart Services	Suspected memory leak, deadlock, or corrupted state	Rolling restart preferred; verify graceful draining is working	N/A (one-time action)
Shed Load	System overloaded beyond capacity	Proactively reject some requests to protect overall stability	Reversible (remove limiting)

The Mitigation Decision Tree When facing multiple mitigation options, prioritize by: Confidence : How likely is this to work? (Higher confidence = try first) Speed : How fast can we execute? (Faster is better when customers are impacted) Reversibility : Can we undo it if it makes things worse? (Reversible is safer) Side Effects : Will this action cause other problems? (Minimal side effects preferred) Blast Radius : How much is affected if this goes wrong? (Smaller is safer) Resolution vs. Mitigation Mitigation restores service; resolution fixes the root cause. They're related but distinct: Mitigation : Rolling back a bad deploy to restore service Resolution : Fixing the bug, deploying the corrected code, and confirming it works Incident response should achieve mitigation quickly. Resolution may come later, even after the incident is officially closed, through follow-up work items.

The Rollback Hesitation Anti-Pattern

Teams often hesitate to rollback because they fear losing work or causing other issues. This hesitation extends outages. Establish a cultural norm: rollback is always acceptable when customer impact is ongoing. The engineering work isn't lost—it's in version control waiting to be fixed and redeployed.

Coordination Mechanics

Multiple people working on an incident isn't automatically better than one person—it's only better if they're coordinated. Effective coordination transforms individual efforts into collective capability. Poor coordination creates confusion, duplication, and conflict. Communication Channels Most organizations use multiple channels during incidents: Incident Channel (Slack/Teams) : Primary async communication; all responders join; all updates logged here Bridge Call (Zoom/Meet) : For real-time synchronous communication during active work; optional for smaller incidents Status Page : Customer-facing communication; managed by Communications Lead Stakeholder Channel : Executive/leadership updates; separate from technical response War Room (physical or virtual): For extended, complex incidents requiring intense collaboration

Incident Channel Best Practices

•
Prefix Messages: Use consistent prefixes to categorize updates
- [STATUS] Current state of the incident
- [ACTION] Something someone is doing or has done
- [FINDING] An observation or discovery
- [THEORY] A hypothesis being considered
- [DECISION] A choice being made by IC or team
•Keep It in the Channel: Avoid side conversations; if it's said on the bridge, post a summary in the channel
•Use Threads Sparingly: Keep main channel focused on primary updates; use threads for detailed technical debugging that would overwhelm the channel
•Regular Heartbeats: IC posts status updates every 10-15 minutes, even if just "Still investigating; no update."
•Pin Important Info: Pin key links (dashboard, runbook, status page) to the channel for easy access
•Bot Automation: Use bots to capture timeline, manage role assignments, and generate post-mortem templates

incident-channel-example.txt
#inc-2024-0142-payment-failures
 
📌 Pinned
- Dashboard: https://grafana.example.com/d/payments-overview
- Runbook: https://runbooks.example.com/payments/high-failure-rate
- Status Page: https://status.example.com
 
---
 
[10:17] 🤖 Incident Bot: Incident created by alert "PaymentFailureRateCritical"
                        Severity: SEV-1 | Commander: unassigned
 
[10:18] @sarah.chen: Taking IC. This is SEV-1, full incident response.
 
[10:18] @sarah.chen: 
[STATUS] Payment failure rate at 12%, up from baseline 0.5%
[STATUS] Started approximately 10:15 AM PST
[ACTION] Paging payments team and platform on-call
 
[10:19] @mike.johnson: Joining as Tech Lead. Looking at recent deploys.
 
[10:20] @sarah.chen: 
[STATUS] Bridge call started: meet.google.com/abc-defg-hij
Need SMEs on: payments gateway, database team
 
[10:22] @mike.johnson:
[FINDING] Deploy at 10:12 AM to payment-service v2.47.3
[ACTION] Pulling up the change list now
 
[10:23] @lisa.wong: Database on-call here. Checking for query performance issues.
 
[10:25] @mike.johnson:
[FINDING] v2.47.3 changed the retry logic for gateway timeouts
[THEORY] Might be creating amplified load on downstream gateway
 
[10:26] @sara.chen:
[DECISION] Let's prepare rollback to v2.47.2 while continuing investigation
@mike.johnson can you stage the rollback?
@lisa.wong any database anomalies?
 
[10:27] @lisa.wong:
[FINDING] No database issues - queries normal latency
 
[10:28] @mike.johnson:
[ACTION] Rollback staged, awaiting confirmation to execute
 
[10:29] @sarah.chen:
[DECISION] Execute rollback. Gateway amplification theory is plausible.
[ACTION] @mike.johnson proceed with rollback to v2.47.2
 
[10:30] @mike.johnson:
[ACTION] Rollback initiated - payment-service v2.47.2 deploying
 
[10:33] @mike.johnson:
[STATUS] Rollback complete. New pods healthy.
 
[10:34] @sarah.chen:
[STATUS] Error rate dropping. 8%... 5%... 3%...
 
[10:37] @sarah.chen:
[STATUS] Error rate at 0.6%, returning to baseline
[STATUS] Monitoring for stability. Will close incident if holds for 15 min.
 
[10:52] @sarah.chen:
[STATUS] Stable at baseline for 15 min. Closing incident.
[ACTION] Will schedule post-mortem for tomorrow 11 AM
[ACTION] @mike.johnson please create follow-up ticket for retry logic fix

The Power of the Scribe Role

A dedicated scribe dramatically improves incident quality. They capture details that would otherwise be lost, create the foundation for post-mortems, and allow technical responders to focus entirely on diagnosis and mitigation. For SEV-1 incidents, always assign a scribe.

Incident Closure and Handoffs

Knowing when and how to close an incident is as important as knowing how to respond to one. Premature closure leads to recurrence; delayed closure wastes resources and creates responder fatigue. Incident Closure Criteria An incident should be considered for closure when: Service Restored : User-facing impact has ended and metrics confirm normal operation Stability Verified : The system has been stable for a defined period (typically 15-30 minutes for major incidents) Immediate Actions Complete : All urgent follow-ups have been addressed or documented Stakeholders Notified : Status page updated, customer communications sent Responders Released : On-call and SMEs can safely return to normal duties Closure \neq Complete Closing an incident means the immediate crisis has passed—not that all work is done. Closure should include: Follow-up Items : Documented in tickets, assigned to owners with due dates Post-Mortem Scheduled : For significant incidents (SEV-1, SEV-2), schedule within 48-72 hours Incident Report Draft : Initial timeline and summary captured while memory is fresh Metrics Capture : Final impact numbers (duration, affected users, revenue impact) recorded

Incident Closure Checklist

•✅ Service metrics have returned to normal baseline for 15+ minutes
•✅ No new error spikes or alerts related to the incident
•✅ Immediate mitigation actions verified as effective
•✅ Status page updated to 'Resolved' with final summary
•✅ Customer communication sent (if applicable)
•✅ Incident channel summary posted with key details
•✅ Follow-up tickets created with owners and due dates
•✅ Post-mortem meeting scheduled (for SEV-1/SEV-2)
•✅ Incident record updated with final impact metrics
•✅ Responders formally released: "Incident closed, thank you everyone"

Handoff Between Responders For extended incidents spanning shift changes, formal handoff is essential: Handoff Protocol: Synchronous Briefing : Outgoing IC briefs incoming IC (10-15 min call minimum) Written Summary : Current status, recent actions, pending items, open questions Context Transfer : Key findings, ruled-out theories, stakeholder communications sent Role Assignment : Explicitly confirm role transitions in the incident channel Questions Encouraged : Incoming responders should ask until they feel confident The worst handoffs happen when: A brief message in Slack replaces a real conversation Outgoing responder disappears before incoming is truly oriented Tacit knowledge ("I noticed something weird but didn't log it") is lost Critical context lives only in someone's head

The 'Soft Close' Pattern

For incidents where you're confident but not certain of resolution: close the incident but keep the channel open and monitoring elevated for 24-48 hours. Announce: 'Soft-closing this incident. Keeping channel active for observation. Will hard-close tomorrow if stable.' This prevents premature confident closure while not keeping responders actively engaged.

System Design (HLD)SLOs, SLIs & Incident Management

Incident Management

LevelAdvanced

Duration90 mins

TopicSLOs, SLIs & Incident Management

2 / 5

Incident Response Process

When Chaos Meets Structure

It's 3:17 AM. Your phone buzzes with a PagerDuty alert: "CRITICAL: Payment Service Error Rate > 5%." You check the dashboard—error rate is climbing, now at 8%. Customers can't complete purchases. Your company loses roughly $50,000 per minute when checkout is broken. What happens next determines whether this incident resolves in 15 minutes or 3 hours. Whether it's a controlled response or a chaotic scramble. Whether your team learns from the experience or repeats the same mistakes. The difference isn't luck or individual heroics—it's process . Organizations that handle incidents well have internalized a structured response framework that channels the urgency of a crisis into coordinated, effective action. This page explores that framework in depth.

What You Will Learn

By the end of this page, you will understand the complete incident response lifecycle: the phases from detection through resolution and beyond, the roles that participate in response, the coordination mechanisms that enable effective parallel work, and the documentation practices that capture learning for the future. You'll be able to design and implement incident response processes that scale from five-person startups to thousand-person enterprises.

The Incident Response Lifecycle

Every incident, regardless of severity or duration, follows a predictable lifecycle. Understanding this lifecycle is fundamental to effective response—it provides a mental model for where you are, what should happen next, and what success looks like at each stage. The Five Phases of Incident Response Detection & Triage : Recognizing an incident exists and determining its initial scope Escalation & Mobilization : Engaging the right people and resources Investigation & Diagnosis : Understanding root cause and impact Mitigation & Resolution : Stopping the bleeding and restoring service Recovery & Learning : Post-incident activities including follow-up and prevention

Converting Mermaid diagram...

Phase Transitions Transitions between phases aren't always linear. Investigation may reveal the need for additional escalation. A mitigation attempt may fail, cycling back to investigation. Resolution may be partial, requiring iteration. Experienced responders recognize these phase transitions and communicate them clearly: "We've triaged this as SEV-1. Escalating to full incident response." "Mitigation attempt failed. We're back in investigation mode." "Service is partially restored. Entering recovery phase for the remaining issues." Naming the phase explicitly helps the team maintain shared understanding of where they are in the process.

The Golden Hour Concept

Borrowed from emergency medicine: the first 60 minutes of incident response are disproportionately impactful. Actions taken (or not taken) in this window often determine whether resolution takes 2 hours or 12 hours. Prioritize getting the right people engaged and stabilizing the situation over perfect diagnosis during this critical window.

Incident Response Roles

Effective incident response requires clear role definition. Without explicit roles, you get either paralysis (everyone waiting for someone else) or chaos (everyone doing everything, stepping on each other). Mature incident response frameworks define several distinct roles: Core Incident Response Roles

Incident Response Roles and Responsibilities
Role	Primary Focus	Key Responsibilities	Required Skills
Incident Commander (IC)	Overall coordination	Owns the incident, coordinates work, makes decisions, drives toward resolution	Leadership, communication, technical breadth, calm under pressure
Technical Lead	Technical investigation	Leads diagnosis, proposes mitigations, validates fixes, coordinates SMEs	Deep system knowledge, debugging skills, architectural understanding
Communications Lead	Stakeholder updates	Updates status page, notifies stakeholders, manages external messaging	Clear writing, stakeholder awareness, timing judgment
Scribe	Documentation	Records timeline, actions, decisions; maintains incident channel log	Attention to detail, fast typing, synthesis skills
Subject Matter Expert	Domain expertise	Provides deep knowledge on specific systems, answers technical questions	Specialized system expertise
Customer Liaison	Customer impact	Monitors support channels, escalates customer reports, coordinates response	Customer empathy, support knowledge, communication

The Incident Commander Role The Incident Commander (IC) is the most critical role in incident response. This person is responsible for the incident as a whole—not for solving the technical problem personally, but for ensuring the incident progresses toward resolution. IC Responsibilities: Maintain Situational Awareness : Know current status, who's working on what, and what's pending Coordinate Parallel Workstreams : Assign investigation tasks and prevent duplication Make Decisions : When the team is stuck deciding between options, the IC decides Manage Timeline : Ensure regular status updates and progress checks Escalate When Needed : Bring in additional expertise or leadership involvement Protect the Team : Shield responders from distractions and interruptions Drive to Closure : Push for action over analysis paralysis; know when to declare resolution IC Anti-Patterns: Getting pulled into hands-on-keyboard debugging (delegate technical work) Making unilateral technical decisions without consulting SMEs Disappearing from the incident channel for extended periods Failing to delegate documentation and communication Continuing incident mode after the problem is resolved

Role Flexibility

For smaller incidents, one person may fill multiple roles. For major incidents, each role might have a dedicated person or even a sub-team. The key is explicit assignment: everyone should know who's doing what. Never assume—always state clearly: 'I'll take IC' or 'Can someone volunteer as scribe?'

Incident Declaration and Triage

Not every alert is an incident. The triage phase determines whether a detected issue warrants incident response activation. This decision has significant implications: declaring an incident too readily leads to chaos and fatigue; hesitating to declare leads to delayed response and extended impact. Incident Declaration Criteria An incident should generally be declared when one or more of the following are true: User Impact : Customers are experiencing degraded service or outage Revenue Impact : Business transactions are failing or significantly delayed Data Risk : Data integrity, availability, or confidentiality is threatened Regulatory Risk : Compliance or legal obligations may be violated Reputational Risk : The issue may become publicly visible Escalating Scope : The problem is growing worse or spreading to other systems Complex Resolution : Multiple teams or extended effort will be required

incident-triage-decision.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
/**
 * Incident Triage Decision Framework
 * 
 * This framework helps responders systematically evaluate
 * whether an issue warrants incident declaration and what
 * severity level to assign.
 */
 
interface TriageInput {
  // Impact assessment
  userImpact: 'none' | 'degraded' | 'partial-outage' | 'full-outage';
  affectedUserPercentage: number;
  revenueImpact: 'none' | 'minor' | 'significant' | 'major';
  
  // Scope assessment
  affectedServices: string[];
  isEscalating: boolean;
  estimatedResolutionTime: 'minutes' | 'hours' | 'uncertain';
  
  // Risk assessment
  dataAtRisk: boolean;
  securityImplications: boolean;
  regulatoryImplications: boolean;
  
  // Context
  recentDeployments: boolean;
  knownIssue: boolean;
  existingIncident: boolean;
}
 
interface TriageDecision {
  declareIncident: boolean;
  recommendedSeverity: 'SEV-1' | 'SEV-2' | 'SEV-3' | 'SEV-4' | null;
  rationale: string;
  immediateActions: string[];
  escalationTargets: string[];
}
 
function triageIssue(input: TriageInput): TriageDecision {
  // Critical path: Full outage with user/revenue impact
  if (input.userImpact === 'full-outage' && input.revenueImpact !== 'none') {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-1',
      rationale: 'Complete service outage affecting revenue',
      immediateActions: [
        'Page all on-call responders for affected services',
        'Open incident bridge/channel immediately',
        'Begin customer communication preparation',
        'Alert leadership within 5 minutes',
      ],
      escalationTargets: [
        'Primary and secondary on-call',
        'Engineering leadership',
        'Customer success leadership',
      ],
    };
  }
 
  // Security or data breach
  if (input.securityImplications || input.dataAtRisk) {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-1',
      rationale: 'Security incident or data at risk requires immediate response',
      immediateActions: [
        'Engage security team immediately',
        'Consider isolation/containment measures',
        'Begin evidence preservation',
        'Alert legal/compliance if data breach suspected',
      ],
      escalationTargets: [
        'Security on-call',
        'CISO or security leadership',
        'Legal/compliance if needed',
      ],
    };
  }
 
  // Partial outage or significant degradation
  if (
    input.userImpact === 'partial-outage' ||
    (input.userImpact === 'degraded' && input.affectedUserPercentage > 25)
  ) {
    const severity = input.isEscalating || input.revenueImpact === 'significant' 
      ? 'SEV-1' 
      : 'SEV-2';
    
    return {
      declareIncident: true,
      recommendedSeverity: severity,
      rationale: 'Significant user impact exceeds acceptable thresholds',
      immediateActions: [
        'Page primary on-call for affected services',
        'Open incident channel',
        'Begin initial investigation',
        'Prepare customer status update',
      ],
      escalationTargets: [
        'Primary on-call for affected services',
        'Engineering manager if escalating',
      ],
    };
  }
 
  // Multi-service impact
  if (input.affectedServices.length >= 3) {
    return {
      declareIncident: true,
      recommendedSeverity: 'SEV-2',
      rationale: 'Widespread impact across multiple services',
      immediateActions: [
        'Page on-call for all affected services',
        'Open incident channel for coordination',
        'Look for common cause (shared dependency)',
      ],
      escalationTargets: input.affectedServices.map(s => `${s} on-call`),
    };
  }
 
  // Known issue with workaround
  if (input.knownIssue && input.userImpact === 'degraded') {
    return {
      declareIncident: false,
      recommendedSeverity: null,
      rationale: 'Known issue with understood impact and workaround',
      immediateActions: [
        'Monitor for escalation',
        'Ensure workaround documentation is current',
        'Track toward permanent resolution',
      ],
      escalationTargets: [],
    };
  }
 
  // Minor degradation
  if (input.userImpact === 'degraded' && input.affectedUserPercentage <= 5) {
    return {
      declareIncident: input.estimatedResolutionTime !== 'minutes',
      recommendedSeverity: 'SEV-3',
      rationale: 'Minor impact, investigate without full incident response unless prolonged',
      immediateActions: [
        'Primary team investigate',
        'Monitor for escalation',
        'Create ticket for tracking',
      ],
      escalationTargets: ['Primary team on-call'],
    };
  }
 
  // Default: Don't declare, but investigate
  return {
    declareIncident: false,
    recommendedSeverity: null,
    rationale: 'Issue does not meet incident criteria; handle through normal operations',
    immediateActions: [
      'Create ticket for tracking',
      'Assign to appropriate team',
      'Monitor for changes in scope',
    ],
    escalationTargets: [],
  };
}
 
// Usage example
const triageResult = triageIssue({
  userImpact: 'partial-outage',
  affectedUserPercentage: 30,
  revenueImpact: 'significant',
  affectedServices: ['checkout', 'payment-gateway'],
  isEscalating: true,
  estimatedResolutionTime: 'uncertain',
  dataAtRisk: false,
  securityImplications: false,
  regulatoryImplications: false,
  recentDeployments: true,
  knownIssue: false,
  existingIncident: false,
});
 
console.log('Triage Decision:', triageResult);
// Output: SEV-1 incident, page all responders, recent deployment likely cause

The Cost of Hesitation

When in doubt, declare the incident. An over-declared incident can be quickly downgraded or closed. An under-declared incident results in delayed response, extended impact, and frustrated teams who weren't informed early enough. It's much easier to scale down than to catch up.

Investigation and Diagnosis

Once an incident is declared, the team must understand what's happening before they can fix it. Investigation is the phase where hypotheses are formed and tested, data is gathered, and root cause is identified. Effective investigation is methodical rather than chaotic—even under pressure. The Investigation Framework Systematic investigation follows a structured approach: What changed? Start with recent changes: deployments, config changes, infrastructure updates, traffic patterns What's affected? Map the blast radius: which services, users, regions, or features are impacted What's the pattern? Look for correlations: timing, user segments, error types, specific endpoints What do the observability signals show? Metrics, logs, and traces tell the story—follow them What's the simplest explanation? Apply Occam's Razor before pursuing exotic theories

Effective Investigation Patterns

•Parallel Paths: Divide investigation across responders (one checks recent deploys, another checks logs)
•Time Boxing: Spend 5-10 minutes on a hypothesis before switching approaches if no progress
•Progressive Narrowing: Start broad (which service?), then narrow (which endpoint? which user type?)
•Seek Contradictions: Look for evidence that disproves your hypothesis, not just confirms it
•Document as You Go: Log findings in the incident channel so nothing is lost
•Think Out Loud: Share reasoning so others can validate or redirect

Investigation Anti-Patterns

•Tunnel Vision: Fixating on the first hypothesis without considering alternatives
•Silent Investigation: Working in isolation without sharing findings
•Tool Distraction: Spending excessive time building queries rather than interpreting results
•Blame Seeking: Focusing on 'who did this' instead of 'what's happening'
•Analysis Paralysis: Pursuing perfect understanding before taking any mitigation action
•Deployment Obsession: Blaming deploys without evidence when other causes are equally likely

The 'Recent Changes' Checklist The majority of incidents are caused by changes. When triaging, quickly review: Code Deployments : What was deployed in the last 24 hours? The last 2 hours? Configuration Changes : Any feature flags, environment variables, or config updates? Infrastructure Changes : New instances, network changes, certificate rotations? Dependency Updates : Did any upstream services or libraries change? Traffic Patterns : Is there unusual traffic volume, source, or composition? External Factors : Is there third-party maintenance, regional outages, or attack patterns? A deployment timeline correlated with the incident onset time often reveals the trigger immediately.

The Five Whys Technique

For complex incidents, apply the 'Five Whys' iteratively: • Why are users seeing errors? Because the API is timing out. • Why is the API timing out? Because database queries are slow. • Why are queries slow? Because the table lacks an index on the new column. • Why is the column unindexed? Because migration didn't include index creation. • Why wasn't the index included? Because our migration process doesn't review for performance. This reveals root causes beyond the immediate symptom.

Mitigation and Resolution

Mitigation is the phase where action replaces investigation. The goal shifts from understanding the problem to stopping the bleeding. In many incidents, mitigation can and should begin before diagnosis is complete—if there's a safe action that might help, take it. The Mitigation Mindset Key principles for effective mitigation: Restore First, Root Cause Later : Don't delay customer recovery to achieve perfect understanding. Rollback now; post-mortem later. Reversible Actions First : Prefer actions you can undo. A rollback is easily reversed; a database schema change is not. Small Blast Radius Before Large : If trying a fix in production, limit exposure: one region, one pod, one percent of traffic. Communicate Before Acting : Announce mitigation actions in the incident channel: "Rolling back the 2:15 PM deploy now." Verify After Acting : Confirm the mitigation worked: "Error rate dropping. Now at 2%, down from 12%."

Common Mitigation Strategies
Strategy	When to Use	Risks/Considerations	Reversibility
Rollback Deployment	Recent deploy correlated with issue onset	Ensure rollback process is tested; may need to rollback database migrations first	Usually reversible (re-deploy)
Disable Feature Flag	New feature causing problems	Fast and surgical; ensure flag controls the suspected code path	Easily reversible
Scale Up Resources	Capacity exhaustion (CPU, memory, connections)	May mask the symptom; doesn't address root cause	Easily reversible
Failover to Backup	Primary system unhealthy	Verify backup is current and functional before switching	Usually reversible (fail back)
Block Bad Traffic	DDoS or abuse pattern identified	May block legitimate users if criteria too broad; may block legitimate traffic	Easily reversible
Restart Services	Suspected memory leak, deadlock, or corrupted state	Rolling restart preferred; verify graceful draining is working	N/A (one-time action)
Shed Load	System overloaded beyond capacity	Proactively reject some requests to protect overall stability	Reversible (remove limiting)

The Mitigation Decision Tree When facing multiple mitigation options, prioritize by: Confidence : How likely is this to work? (Higher confidence = try first) Speed : How fast can we execute? (Faster is better when customers are impacted) Reversibility : Can we undo it if it makes things worse? (Reversible is safer) Side Effects : Will this action cause other problems? (Minimal side effects preferred) Blast Radius : How much is affected if this goes wrong? (Smaller is safer) Resolution vs. Mitigation Mitigation restores service; resolution fixes the root cause. They're related but distinct: Mitigation : Rolling back a bad deploy to restore service Resolution : Fixing the bug, deploying the corrected code, and confirming it works Incident response should achieve mitigation quickly. Resolution may come later, even after the incident is officially closed, through follow-up work items.

The Rollback Hesitation Anti-Pattern

Teams often hesitate to rollback because they fear losing work or causing other issues. This hesitation extends outages. Establish a cultural norm: rollback is always acceptable when customer impact is ongoing. The engineering work isn't lost—it's in version control waiting to be fixed and redeployed.

Coordination Mechanics

Multiple people working on an incident isn't automatically better than one person—it's only better if they're coordinated. Effective coordination transforms individual efforts into collective capability. Poor coordination creates confusion, duplication, and conflict. Communication Channels Most organizations use multiple channels during incidents: Incident Channel (Slack/Teams) : Primary async communication; all responders join; all updates logged here Bridge Call (Zoom/Meet) : For real-time synchronous communication during active work; optional for smaller incidents Status Page : Customer-facing communication; managed by Communications Lead Stakeholder Channel : Executive/leadership updates; separate from technical response War Room (physical or virtual): For extended, complex incidents requiring intense collaboration

Incident Channel Best Practices

•
Prefix Messages: Use consistent prefixes to categorize updates
- [STATUS] Current state of the incident
- [ACTION] Something someone is doing or has done
- [FINDING] An observation or discovery
- [THEORY] A hypothesis being considered
- [DECISION] A choice being made by IC or team
•Keep It in the Channel: Avoid side conversations; if it's said on the bridge, post a summary in the channel
•Use Threads Sparingly: Keep main channel focused on primary updates; use threads for detailed technical debugging that would overwhelm the channel
•Regular Heartbeats: IC posts status updates every 10-15 minutes, even if just "Still investigating; no update."
•Pin Important Info: Pin key links (dashboard, runbook, status page) to the channel for easy access
•Bot Automation: Use bots to capture timeline, manage role assignments, and generate post-mortem templates

incident-channel-example.txt
#inc-2024-0142-payment-failures
 
📌 Pinned
- Dashboard: https://grafana.example.com/d/payments-overview
- Runbook: https://runbooks.example.com/payments/high-failure-rate
- Status Page: https://status.example.com
 
---
 
[10:17] 🤖 Incident Bot: Incident created by alert "PaymentFailureRateCritical"
                        Severity: SEV-1 | Commander: unassigned
 
[10:18] @sarah.chen: Taking IC. This is SEV-1, full incident response.
 
[10:18] @sarah.chen: 
[STATUS] Payment failure rate at 12%, up from baseline 0.5%
[STATUS] Started approximately 10:15 AM PST
[ACTION] Paging payments team and platform on-call
 
[10:19] @mike.johnson: Joining as Tech Lead. Looking at recent deploys.
 
[10:20] @sarah.chen: 
[STATUS] Bridge call started: meet.google.com/abc-defg-hij
Need SMEs on: payments gateway, database team
 
[10:22] @mike.johnson:
[FINDING] Deploy at 10:12 AM to payment-service v2.47.3
[ACTION] Pulling up the change list now
 
[10:23] @lisa.wong: Database on-call here. Checking for query performance issues.
 
[10:25] @mike.johnson:
[FINDING] v2.47.3 changed the retry logic for gateway timeouts
[THEORY] Might be creating amplified load on downstream gateway
 
[10:26] @sara.chen:
[DECISION] Let's prepare rollback to v2.47.2 while continuing investigation
@mike.johnson can you stage the rollback?
@lisa.wong any database anomalies?
 
[10:27] @lisa.wong:
[FINDING] No database issues - queries normal latency
 
[10:28] @mike.johnson:
[ACTION] Rollback staged, awaiting confirmation to execute
 
[10:29] @sarah.chen:
[DECISION] Execute rollback. Gateway amplification theory is plausible.
[ACTION] @mike.johnson proceed with rollback to v2.47.2
 
[10:30] @mike.johnson:
[ACTION] Rollback initiated - payment-service v2.47.2 deploying
 
[10:33] @mike.johnson:
[STATUS] Rollback complete. New pods healthy.
 
[10:34] @sarah.chen:
[STATUS] Error rate dropping. 8%... 5%... 3%...
 
[10:37] @sarah.chen:
[STATUS] Error rate at 0.6%, returning to baseline
[STATUS] Monitoring for stability. Will close incident if holds for 15 min.
 
[10:52] @sarah.chen:
[STATUS] Stable at baseline for 15 min. Closing incident.
[ACTION] Will schedule post-mortem for tomorrow 11 AM
[ACTION] @mike.johnson please create follow-up ticket for retry logic fix

The Power of the Scribe Role

A dedicated scribe dramatically improves incident quality. They capture details that would otherwise be lost, create the foundation for post-mortems, and allow technical responders to focus entirely on diagnosis and mitigation. For SEV-1 incidents, always assign a scribe.

Incident Closure and Handoffs

Knowing when and how to close an incident is as important as knowing how to respond to one. Premature closure leads to recurrence; delayed closure wastes resources and creates responder fatigue. Incident Closure Criteria An incident should be considered for closure when: Service Restored : User-facing impact has ended and metrics confirm normal operation Stability Verified : The system has been stable for a defined period (typically 15-30 minutes for major incidents) Immediate Actions Complete : All urgent follow-ups have been addressed or documented Stakeholders Notified : Status page updated, customer communications sent Responders Released : On-call and SMEs can safely return to normal duties Closure \neq Complete Closing an incident means the immediate crisis has passed—not that all work is done. Closure should include: Follow-up Items : Documented in tickets, assigned to owners with due dates Post-Mortem Scheduled : For significant incidents (SEV-1, SEV-2), schedule within 48-72 hours Incident Report Draft : Initial timeline and summary captured while memory is fresh Metrics Capture : Final impact numbers (duration, affected users, revenue impact) recorded

Incident Closure Checklist

•✅ Service metrics have returned to normal baseline for 15+ minutes
•✅ No new error spikes or alerts related to the incident
•✅ Immediate mitigation actions verified as effective
•✅ Status page updated to 'Resolved' with final summary
•✅ Customer communication sent (if applicable)
•✅ Incident channel summary posted with key details
•✅ Follow-up tickets created with owners and due dates
•✅ Post-mortem meeting scheduled (for SEV-1/SEV-2)
•✅ Incident record updated with final impact metrics
•✅ Responders formally released: "Incident closed, thank you everyone"

Handoff Between Responders For extended incidents spanning shift changes, formal handoff is essential: Handoff Protocol: Synchronous Briefing : Outgoing IC briefs incoming IC (10-15 min call minimum) Written Summary : Current status, recent actions, pending items, open questions Context Transfer : Key findings, ruled-out theories, stakeholder communications sent Role Assignment : Explicitly confirm role transitions in the incident channel Questions Encouraged : Incoming responders should ask until they feel confident The worst handoffs happen when: A brief message in Slack replaces a real conversation Outgoing responder disappears before incoming is truly oriented Tacit knowledge ("I noticed something weird but didn't log it") is lost Critical context lives only in someone's head

The 'Soft Close' Pattern

For incidents where you're confident but not certain of resolution: close the incident but keep the channel open and monitoring elevated for 24-48 hours. Announce: 'Soft-closing this incident. Keeping channel active for observation. Will hard-close tomorrow if stable.' This prevents premature confident closure while not keeping responders actively engaged.

Incident Response Process

When Chaos Meets Structure

The Incident Response Lifecycle

Incident Response Roles

Incident Declaration and Triage

Investigation and Diagnosis

Mitigation and Resolution

Coordination Mechanics

Incident Closure and Handoffs

There’s more to understand.

Incident Response Process

When Chaos Meets Structure

The Incident Response Lifecycle

Incident Response Roles

Incident Declaration and Triage

Investigation and Diagnosis

Mitigation and Resolution

Coordination Mechanics

Incident Closure and Handoffs

There’s more to understand.