Learning from Failures

In 2003, NASA's Space Shuttle Columbia disintegrated during re-entry, killing all seven crew members. The immediate technical cause was foam debris striking the shuttle's thermal protection tiles during launch. But the deeper finding of the Columbia Accident Investigation Board was devastating: NASA had failed to learn from previous incidents.

Foam strikes had occurred on nearly every previous shuttle mission. They had been discussed, investigated, and normalized. The organization had the information to prevent Columbia's loss—it simply failed to transform that information into effective action. The lessons were documented but not learned.

The post-mortem is not learning. The post-mortem creates the potential for learning. Actual learning happens when insights change behavior, when knowledge spreads beyond the incident team, when patterns are recognized across incidents, and when the organization fundamentally improves its capacity to prevent and respond to failures.

This page is about closing that gap—transforming incidents from isolated events into drivers of organizational improvement.

When an incident occurs and a post-mortem is conducted, who learns? At minimum, the individuals directly involved learn something—they gain firsthand experience of a failure mode and its resolution. But individual learning is fragile and limited:

• Individuals leave organizations, taking knowledge with them
• Individuals forget details over time
• Individuals may not encounter similar situations again
• Individual learning doesn't scale—the same lesson must be learned repeatedly by different people

Organizational learning occurs when the organization itself changes: its systems, processes, documentation, training, tooling, and culture. These changes persist beyond any individual and apply to everyone who encounters similar situations.

The Learning Hierarchy:

Learning operates at multiple levels, each with increasing impact and persistence:

1. Individual memory — Lowest level; fragile, doesn't scale
2. Team knowledge — Shared understanding within a team; lost when team changes
3. Documentation — Written records that persist; requires discovery and reading
4. Process change — New procedures that apply to everyone; requires compliance
5. Automation/tooling — Systemic changes that apply automatically; highest persistence

Goal: Push learning up the hierarchy. Whenever possible, transform individual lessons into automation or tooling. When automation isn't feasible, encode in process. When process is insufficient, at least document.

A post-mortem document sitting in a folder helps no one if teams in similar situations don't know it exists. Dissemination is the process of spreading incident learnings beyond the immediate team to everyone who might benefit.

Dissemination channels:

Targeting dissemination:

Not every post-mortem needs maximum distribution. Consider audience relevance:

• Team-level incidents (limited impact, localized cause) — Share within team; summarize to broader org
• Cross-team incidents (involving multiple services) — Share with all involved teams; summarize broadly
• Organization-wide incidents (major outages, novel failure modes) — Full readout to all engineering; executive summary to company
• Industry-relevant incidents (novel patterns, useful to others) — Consider publishing externally (blog posts, conferences)

Individual incidents reveal local failure modes. Patterns across incidents reveal systemic issues. An organization that analyzes incidents only in isolation misses critical insights that emerge from aggregation.

Types of patterns to watch for:

Implementing pattern recognition:

1. Incident tagging/categorization

Develop a taxonomy of incident attributes:

• Component(s) involved
• Root cause category (e.g., configuration, capacity, bug, dependency)
• Trigger type (e.g., deployment, traffic spike, external event)
• Detection method (e.g., monitoring, customer report, internal discovery)
• Impact category (e.g., availability, latency, data integrity)

Apply tags consistently to every post-mortem.

2. Periodic pattern analysis

Quarterly review of all incidents with aggregate analysis:

• Volume and trends
• Distribution by category
• Emerging themes
• Repeat offenders (services, teams, causes)

3. Threshold-triggered investigation

Automatic alerts when thresholds are crossed:

• 'Service X has had 3+ incidents this quarter'
• 'Root cause Y has appeared in 5+ post-mortems'
• 'Customer detection rate is above 30%'

Threshold crossing triggers a focused investigation into the systemic issue.

While individual post-mortems focus on specific incidents, the Learning Review is a periodic examination of incident patterns and organizational improvement trends. It answers: Are we actually getting better?

Learning Review structure:

Frequency and attendees:

• Engineering-wide Learning Review: Quarterly, attended by all team leads + reliability/SRE + engineering leadership. 60-90 minutes.
• Team-level Learning Review: Monthly, attended by team members. 30 minutes.

Sample agenda for quarterly Learning Review:

Not all failures cause incidents. Sometimes an error is made, but recovery happens before customers are affected. A misconfiguration is deployed but caught by monitoring and rolled back in seconds. A database failover is triggered but completes seamlessly.

These near-misses are goldmines of learning—they reveal vulnerabilities without incurring damage. But they're easy to overlook. Since no harm occurred, there's little urgency to investigate.

High-reliability organizations actively cultivate near-miss reporting.

Building a near-miss program:

1. Create a low-ceremony reporting mechanism — A Slack channel, a simple form, or a dedicated email alias. The goal is to minimize friction so people actually report.

2. Explicitly encourage and thank reporters — Public recognition for near-miss reports reinforces the behavior. 'Thanks for catching this before it became an incident.'

3. Regular near-miss review — Weekly or biweekly review of near-miss reports to identify patterns and prioritize investigations.

4. Elevated near-misses — When a near-miss reveals a serious vulnerability, escalate to a full post-mortem. Just because nothing bad happened doesn't mean it couldn't have.

5. Psychological safety — Near-miss reporting only works if people don't fear punishment for admitting they almost caused problems. Blameless culture is a prerequisite.

The Heinrich Ratio: Industrial safety research suggests that for every major incident, there are approximately 30 minor incidents and 300 near-misses. If you're only learning from incidents, you're missing 99% of your learning opportunities.

You don't have to experience every failure firsthand to learn from it. Organizations across the industry regularly publish incident reports, post-mortems, and reliability case studies. Learning from others' failures is vastly cheaper than learning from your own.

Sources of external incident learning:

Applying external learnings:

1. Regular reading/review — Assign engineers to periodically review external incident reports and summarize relevant lessons.

2. Defensive analysis — When reading about an incident at another company, ask: 'Could this happen to us?' If yes, what would prevent or detect it?

3. Architecture review input — When designing new systems, review relevant external incidents to inform design decisions.

4. Tabletop exercises — Use external incident scenarios for tabletop exercises: 'If this happened to us, how would we respond?'

5. Knowledge sharing — Circulate notable external post-mortems to relevant teams. 'Cloudflare had an outage due to X—here's how our system handles this.'

Incident learnings must be stored somewhere accessible. A knowledge repository is the organizational memory for incident experience—searchable, browsable, and integrated into engineering workflows.

Repository requirements:

Tool options:

• Dedicated incident management platforms (Blameless, FireHydrant, incident.io) — Built-in post-mortem management with search, tagging, and tracking
• Documentation platforms (Notion, Confluence, GitHub Wiki) — Flexible, integrated with other engineering docs, but requires discipline for consistency
• Custom internal tools — Some organizations build bespoke incident management systems tailored to their processes

Integration points:

• Link from on-call handoffs to relevant post-mortems
• Include 'Related Incidents' section in service runbooks
• Surface relevant post-mortems during incident response (based on affected service or symptoms)
• Connect action items to originating post-mortems bidirectionally