The Consensus Problem

In 1985, three researchers—Michael Fischer, Nancy Lynch, and Michael Paterson—published a paper that sent shockwaves through the distributed systems community. Their result, now known as the FLP impossibility theorem, proved something startling: deterministic consensus is impossible in an asynchronous system if even a single process can fail.

This wasn't a limitation of known algorithms or a gap in current research—it was a fundamental, mathematical impossibility. No matter how clever the protocol, no matter how many messages are exchanged, there exists an execution in which the protocol never terminates. The FLP result closed the door on an entire approach to distributed consensus.

But rather than being a dead end, FLP became the foundation for understanding what consensus protocols can do. Every practical consensus algorithm—Paxos, Raft, PBFT, and beyond—works by carefully sidestepping FLP's assumptions. Understanding this impossibility is essential for understanding why consensus protocols are designed the way they are.

Let's state the FLP result clearly before diving into its meaning:

FLP Impossibility Theorem (1985):

In an asynchronous system with reliable message delivery, no deterministic protocol can guarantee consensus if even a single process may fail (crash).

Breaking this down:

• Asynchronous system: No bounds on message delay or process execution speed
• Reliable message delivery: Messages that are sent are eventually delivered (no loss)
• Deterministic protocol: Given the same inputs and message orderings, the protocol always behaves the same way
• Consensus: The three properties: Agreement, Validity, Termination
• Single process failure: Even one crash failure is enough to break things

What FLP Does NOT Say:

It's equally important to understand what FLP does not claim:

The Practical Implication:

FLP shows that for any deterministic consensus protocol, an adversarial scheduler (or just bad luck with timing) can construct an execution where the protocol never decides. This doesn't mean the protocol usually fails—it means there's always a pathological case.

Think of it like this: for any deterministic strategy you devise, I can always find a sequence of message delays and process schedules that makes your protocol run forever without deciding. The protocol might work 99.999% of the time in practice, but I can always construct that 0.001% failure case.

The FLP proof is intricate, but its core intuition is accessible. The argument relies on two key concepts: bivalent states and indistinguishability.

Bivalent States:

A configuration (state of all processes and messages in transit) is called:

• 0-valent: From this state, only 0 can be decided (all reachable decisions are 0)
• 1-valent: From this state, only 1 can be decided (all reachable decisions are 1)
• Bivalent: From this state, both 0 and 1 are still possible (the decision isn't determined yet)

Think of bivalent as 'undecided'—the system hasn't committed to any outcome yet.

The FLP Proof Strategy (Intuition):

The proof proceeds in three parts:

Part 1: There exists an initial bivalent state.

With different processes proposing 0 and 1, there must be some initial configuration from which both outcomes are reachable. If all initial states were univalent (committed to one outcome), then the decision would depend only on the initial proposals, violating the requirement that the protocol must work despite failures.

Part 2: From any bivalent state, we can reach another bivalent state.

This is the heart of the proof. Given a bivalent state C and any message m that could be delivered, the proof shows there exists a schedule that:

• Delivers m
• Keeps the system bivalent

This is done by carefully considering what happens if m takes the system to a 0-valent state vs a 1-valent state, and showing that a small change in the schedule (like delaying one process's step) can flip the outcome—meaning neither outcome is determined yet.

Part 3: By infinitely repeating Part 2, we never decide.

If we can always find a way to stay bivalent, we can construct an infinite execution that never commits to any decision.

Let's make FLP concrete with a simplified example. Consider three processes deciding between 0 and 1.

Setup:

• Process P proposes 0
• Process Q proposes 1
• Process R has no initial preference

Suppose the protocol works roughly like: 'Wait for two values, then decide on the majority.'

The Adversarial Schedule:

The Critical Step Analysis:

Now consider two scenarios from this bivalent state:

Scenario A: R receives P's message first.

• R sees: 0 from P
• R might wait for more, or might lean toward 0
• If R then crashes before receiving from Q...
• P and Q might see R's 0-leaning response and decide 0

Scenario B: R receives Q's message first.

• R sees: 1 from Q
• R might wait for more, or might lean toward 1
• If R then crashes before receiving from P...
• P and Q might see R's 1-leaning response and decide 1

The Indistinguishability Trap:

Here's the key insight: if R crashes at just the right moment (before sending any response), P and Q cannot distinguish:

• 'R crashed' from 'R is just slow'
• 'R received 0 first' from 'R received 1 first' from 'R received nothing'

This indistinguishability means P and Q don't know which way R would have voted. If they wait forever, they violate termination. If they proceed without R, they might make a decision that R (if it recovers) would have prevented.

The Infinite Loop:

The adversary's strategy is to always crash (or delay) the 'critical' process—the one whose next step would determine the outcome. By repeatedly delaying or crashing this process, the adversary keeps the system bivalent forever.

FLP isn't magic—it relies on specific assumptions. Understanding these assumptions reveals the escape routes practical protocols use.

The Critical Assumptions of FLP:

Deep Dive: Why Asynchrony Is Essential to FLP

The asynchrony assumption is the linchpin of FLP. In an asynchronous system:

1. A message delayed 1 second looks like a message delayed 1 year
2. A slow process looks like a crashed process
3. No timeout is 'safe'—any timeout might incorrectly suspect a correct process

This creates the impossibility: the protocol cannot distinguish slow from dead, so it cannot safely proceed without the slow process (might be a valid vote) or safely wait forever (violates termination).

Deep Dive: Why Determinism Is Essential

A deterministic protocol always makes the same choice given the same information. This means the adversary can predict what the protocol will do and schedule messages to prevent commitment.

With randomization, the adversary can't predict the protocol's choices. Even if they control message timing, the protocol might randomly make a decision they didn't anticipate.

Deep Dive: Why Even One Failure Matters

You might think, 'Surely we can tolerate failures if we have enough redundancy?' But FLP shows that even the possibility of one failure is enough. The protocol must be designed to handle the failure, and that design creates the vulnerability the adversary exploits.

If deterministic consensus is impossible in async systems, how do Paxos, Raft, and other protocols work in practice? They circumvent FLP by violating (or relaxing) one of its assumptions.

The Three Escape Routes:

Paxos and Raft: The Partial Synchrony Approach

Most practical consensus protocols, including Paxos and Raft, take the partial synchrony route:

1. Safety always holds: Agreement and validity are never violated, regardless of timing. Even during network partitions or message delays, decided values are never contradicted.

2. Liveness requires eventual synchrony: The protocol may stall during async periods (fast leader changes, split votes), but once the network stabilizes and messages arrive within bounds, progress is made.

Why This Works in Practice:

Real networks are 'mostly synchronous'—they have transient periods of high latency or partitions, but usually messages arrive in reasonable time. By designing for partial synchrony:

• During normal operation (synchronous), the protocol makes quick progress
• During instability (asynchronous), the protocol might stall but never corrupts
• Once stability returns, progress resumes

The 'stalling' is precisely what FLP predicts—there's always a potential execution that doesn't terminate. But in practice, real networks don't behave like a malicious adversary optimizing for non-termination.

FLP isn't just a theoretical curiosity—it profoundly shapes how we design distributed systems. Understanding this influence helps you appreciate why protocols are structured the way they are.

Design Implications of FLP:

CAP Theorem Through the FLP Lens:

FLP helps explain the CAP theorem too. During a partition:

• The system is effectively asynchronous (cannot distinguish slow from dead)
• FLP applies—cannot guarantee both safety and liveness
• CAP forces the choice: availability (liveness) or consistency (safety)

CP systems (like Paxos-based databases) choose consistency, accepting potential unavailability. AP systems choose availability, accepting potential inconsistency. FLP tells us this choice is unavoidable.

Testing Implications:

FLP also shapes how we test distributed systems:

• Chaos engineering: Inject failures to find the executions that cause problems
• Jepsen testing: Specifically look for safety violations under partition
• Adversarial testing: Simulate the FLP adversary—delay exactly the messages that would prevent progress

Knowing that pathological executions exist means we must actively search for them.

FLP is the most famous impossibility result in distributed computing, but it's part of a family of results that define the limits of what's possible. Understanding this landscape provides a fuller picture of distributed systems theory.

Related Impossibility Results:

The Consensus Hierarchy (Herlihy 1991):

Maurice Herlihy proved that shared objects can be ranked by their 'consensus power'—the number of processes for which they can solve consensus:

• Consensus number 1: Read-write registers (most basic memory)
• Consensus number 2: Test-and-set, fetch-and-add
• Consensus number ∞: Compare-and-swap (CAS), load-linked/store-conditional

This shows that CAS is 'universal'—given CAS, you can implement any concurrent object for any number of processes. This is why modern CPUs provide CAS instructions.

The Weakest Failure Detector:

Chandra, Hadzilacos, and Toueg showed that Ω (eternal leader) is the weakest failure detector for solving consensus. This means:

• Any failure detector that solves consensus is at least as powerful as Ω
• If you can implement Ω, you can solve consensus
• Paxos and Raft implicitly implement Ω through their leader election

We've explored the famous FLP impossibility result—the theoretical cornerstone that shapes how we think about and design consensus protocols. Let's consolidate our understanding:

What's Next:

We've now completed our exploration of the consensus problem—what it is, why it's hard, the different failure models, and the fundamental impossibility that constrains our solutions. In the following modules, we'll study the concrete protocols that solve consensus within these constraints: Paxos, Raft, and ZAB. Armed with this theoretical foundation, you'll understand not just how these protocols work, but why they're designed the way they are.