Root Cause Analysis

After every significant incident, someone asks: 'What was the root cause?' The question seems simple, almost administrative—we need something to write in the incident report, something to tell stakeholders, something to check off before moving on.

But this question, and especially our approach to answering it, determines whether we genuinely prevent future incidents or merely document past failures. Root cause analysis (RCA) is the systematic process of identifying the fundamental factors that contributed to an incident. Done well, RCA reveals actionable insights that improve system reliability. Done poorly, it produces superficial explanations that change nothing.

The challenge is that complex systems fail in complex ways. There is rarely a single 'root cause' in any meaningful sense. Multiple factors combine—technical, organizational, environmental—to produce an outcome that no single factor would have caused alone. Effective RCA must navigate this complexity rather than suppress it.

Before diving into techniques, we must address a fundamental misconception: the notion that every incident has a single 'root cause' that, if eliminated, would have prevented the failure.

This belief—sometimes called the 'root cause fallacy'—is deeply embedded in engineering culture, inherited from simpler domains where linear cause-and-effect relationships hold. If a machine breaks, there's often a specific component that failed. Find the component, replace it, problem solved.

Complex sociotechnical systems don't work this way. Consider a typical modern production incident:

Which of these is 'the' root cause? All of them contributed, and none alone is sufficient. If any one of them had been different, the incident might not have occurred—or might have been detected and resolved before causing user impact.

The term 'root cause' is so entrenched that abandoning it is impractical, but we must reframe it. Root causes are the set of systemic factors that, if addressed, significantly reduce the probability or impact of similar incidents. There are always multiple, and the goal of RCA is to identify the ones most amenable to improvement.

The Five Whys is the most widely-known RCA technique, originating from the Toyota Production System. The method is deceptively simple: when a problem occurs, ask 'Why?' successively (traditionally five times, though the number is not fixed) to peel back layers of causation until reaching an addressable root level.

Example application:

Strengths of Five Whys:

• Simplicity — Requires no specialized training or tools
• Accessibility — Anyone can participate, promoting democratic analysis
• Speed — Can be completed quickly in a meeting
• Depth — Naturally pushes past surface-level explanations

When to use Five Whys:

Five Whys works best for relatively simple incidents with a dominant causal chain. For complex, multi-factor incidents, use it as a starting point but complement with more rigorous methods like Fault Trees or Ishikawa diagrams.

Best practices:

• Apply Five Whys to multiple starting points, not just one
• Don't stop at five if the answer isn't actionable
• Avoid stopping at 'human error'; always ask why the human action led to failure
• Capture branching causes, not just a single chain
• Validate each 'why' answer with evidence from logs, traces, or other data

Ishikawa diagrams, also called fishbone or cause-and-effect diagrams, address the Five Whys' limitation of linear causation. Created by Kaoru Ishikawa in the 1960s, this technique visually maps multiple categories of contributing factors to an incident.

The diagram structure resembles a fish skeleton: the 'head' is the incident/problem, and 'bones' branch off the spine representing categories of causes. Within each category, specific factors are listed.

Common category frameworks for software systems:

The 6 Ms (adapted from manufacturing):

• Methods — Processes, procedures, runbooks
• Machines — Hardware, software, infrastructure
• Materials — Data, configurations, dependencies
• Measurements — Monitoring, alerting, observability
• Mother Nature — Environmental factors, time pressure, external events
• People — Training, experience, cognitive load

Alternative: 4 Ps (production-focused):

• Policies — Organizational rules and constraints
• Procedures — Specific operational steps
• People — Human factors
• Plant — Infrastructure and tooling

Fault Tree Analysis (FTA) is a deductive, top-down technique that maps the logical relationships between a top-level failure (the incident) and its underlying causes. Unlike Ishikawa diagrams, which are primarily taxonomic, fault trees model how causes combine using Boolean logic.

FTA originated in aerospace and nuclear industries where rigorous failure analysis is mandatory. It's particularly powerful for complex incidents involving multiple interacting failures.

Key concepts in Fault Tree Analysis:

• Top Event — The incident being analyzed (e.g., 'Service unavailable for 15 minutes')
• Intermediate Events — Contributing failures that combine to cause the top event
• Basic Events — Root-level failures that cannot be decomposed further (e.g., 'Unit tests missing')
• AND Gate — All child events must occur for the parent event to occur
• OR Gate — Any child event is sufficient to cause the parent event

Analyzing the tree:

The Boolean structure reveals critical insights:

• Minimal Cut Sets — The smallest combination of basic events that would cause the top event. These identify the most critical vulnerabilities.
• Single Points of Failure — Events connected by OR gates where one failure alone causes the incident.
• Redundancy Gaps — Places where AND gates could be introduced to require multiple failures.

Clear terminology is essential for effective RCA. Different levels of causation require different responses:

Proximate Cause: The immediate trigger of the incident—the final action or event in the causal chain. Example: 'The engineer ran the migration script against production.'

Proximate causes are often human actions. While they're the 'cause' in a technical sense, stopping here is almost always insufficient for prevention.

Contributing Factors: Conditions that increased the probability or severity of the incident but weren't direct causes by themselves. Example: 'The migration was scheduled for 2:47 AM when the on-call engineer was fatigued.'

Contributing factors often involve organizational, environmental, or contextual elements. They represent the 'latent conditions' in James Reason's Swiss Cheese model.

Root Causes: The fundamental systemic flaws that, if addressed, would prevent the entire class of incident. Example: 'The deployment system does not distinguish between production and staging environments.'

Root causes typically involve system design, processes, tooling, or organizational structures. They're 'root' not because they're first chronologically, but because they're the deepest level at which intervention would be effective.

Even with good intentions and structured techniques, RCA frequently produces inadequate results. Awareness of common pitfalls allows teams to avoid them:

Quality indicators for root cause analysis:

✓ Root causes are systemic, not individual ✓ Multiple contributing factors are identified ✓ Each root cause is actionable (leads to a specific improvement) ✓ The analysis explains how, not just what ✓ Timeline events are supported by data, not memory ✓ Analysis accounts for what was knowable at decision points ✓ Uncomfortable truths about systems/processes are not avoided

Rigorous RCA requires evidence, not narrative reconstruction from memory. Human memory is notoriously unreliable, especially for stressful incidents. Within hours, recollections become contaminated by hindsight, conversation, and the natural tendency to construct coherent stories.

Evidence sources for RCA:

Preserving evidence during incidents:

The time to gather evidence is immediately during and after the incident—not days later during the post-mortem meeting. Establish a practice of:

1. Taking screenshots of dashboards and alerts at key moments
2. Noting the exact times of significant actions
3. Preserving the state of anomalous systems (logs, metrics) before cleanup
4. Creating a dedicated channel/thread for incident discussion that can be referenced later
5. Running automated evidence collection scripts that snapshot logs, metrics, and alerts