S073 · Consensus — Paxos & Raft Intuition

Module M08: Distributed Systems · Session 73 of 130 · Track: SYS 🗳️

What you'll be able to do after this session

• Explain Consensus to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S072 — Consistency Models — Linearizable, Sequential, Eventual

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Raft Explained (TLA+ / Ian Cooper) — 12-minute animation that makes leader election finally click.
• 🎥 Deep dive (30–60 min) · Designing for Understandability: Raft (Diego Ongaro, Stanford) — the co-author's own 60-minute talk.
• 🎥 Hands-on demo (10–20 min) · The Secret Lives of Data — Raft Visualisation — companion video for the classic interactive Raft demo.
• 📖 Canonical article · In Search of an Understandable Consensus Algorithm — Raft paper (Ongaro & Ousterhout) — the paper, ~18 pages, unusually readable.
• 📖 Book chapter / tutorial · The Secret Lives of Data — Raft (interactive) — click through leader election and log replication step by step.
• 📖 Engineering blog · etcd — Why the Raft Consensus Algorithm — how etcd (the brain of Kubernetes) actually uses Raft in production.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How machines agree.

Imagine five friends deciding where to eat, but they can only text each other and any text might get lost or delayed forever. How do they guarantee everyone ends up at the same restaurant, even if two friends' phones die mid-conversation? That is the consensus problem, and it is the single hardest thing in distributed systems.

Consensus exists because the moment you have more than one machine, one of them will be slow, crashed, partitioned, or lying — and yet the group has to agree on things like "who is the leader?", "what was the next log entry?", or "was that transaction committed?". Without agreement, you get split brain: two nodes think they're leader, both accept writes, and your database silently diverges.

Paxos (Leslie Lamport, 1989) proved this is possible — but the paper is famously incomprehensible. Raft (Ongaro & Ousterhout, 2014) is functionally equivalent but designed for humans to understand. Raft breaks consensus into three tidy sub-problems: leader election, log replication, and safety. Every modern distributed system you'll touch — etcd, Consul, CockroachDB, TiDB, MongoDB replica sets, Kafka's new KRaft mode — uses Raft or a Raft-shaped protocol.

Gotcha to carry forever: consensus needs a majority (quorum) to make progress. A 3-node cluster tolerates 1 failure; a 5-node cluster tolerates 2. Adding an even number of nodes doesn't help fault tolerance and adds latency — always run 3, 5, or 7 nodes, never 2, 4, or 6.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Raft's normal-case operation:

Leader election worked example. Nodes have terms (a monotonically increasing number, like an era). Each node is in one of three states: follower, candidate, leader.

Safety rule that saves you. A candidate can only win an election if its log is at least as up-to-date as the majority's — this prevents a stale node from becoming leader and truncating committed entries. This one rule is why Raft never loses a committed write.

Where Paxos differs. Basic Paxos agrees on one value; Multi-Paxos strings agreements together; Fast Paxos skips a round in the common case. All of them work; none of them are as easy to implement correctly as Raft, which is why Raft won the mindshare battle.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Run a real 3-node Raft cluster (etcd) locally, watch leader election, then kill the leader.

#!/usr/bin/env bash
# raft-demo.sh · needs docker
set -euo pipefail
 
docker network create raft-net 2>/dev/null || true
TOKEN=demo-cluster
INITIAL="etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380"
 
for i in 1 2 3; do
  docker run -d --name etcd$i --network raft-net \
    -p $((2378+i)):2379 \
    quay.io/coreos/etcd:v3.5.11 \
    /usr/local/bin/etcd \
    --name etcd$i \
    --initial-advertise-peer-urls http://etcd$i:2380 \
    --listen-peer-urls http://0.0.0.0:2380 \
    --listen-client-urls http://0.0.0.0:2379 \
    --advertise-client-urls http://etcd$i:2379 \
    --initial-cluster-token $TOKEN \
    --initial-cluster "$INITIAL" \
    --initial-cluster-state new
done
sleep 5
 
echo '--- current leader ---'
docker exec etcd1 etcdctl --endpoints=etcd1:2379,etcd2:2379,etcd3:2379 endpoint status --write-out=table
 
echo '--- write a key ---'
docker exec etcd1 etcdctl put /demo/hello "world (term 1)"
docker exec etcd2 etcdctl get /demo/hello
 
echo '--- kill current leader ---'
LEADER=$(docker exec etcd1 etcdctl --endpoints=etcd1:2379,etcd2:2379,etcd3:2379 \
   endpoint status --write-out=json | grep -o '"IsLeader":true' -B1 | head -1 || true)
docker stop etcd1  # (adjust if leader is elsewhere)
sleep 3
 
echo '--- new leader after election ---'
docker exec etcd2 etcdctl --endpoints=etcd2:2379,etcd3:2379 endpoint status --write-out=table
docker exec etcd2 etcdctl --endpoints=etcd2:2379,etcd3:2379 get /demo/hello

Observe (checklist):

1. endpoint status shows exactly one IS_LEADER=true — never zero, never two.
2. A put succeeds through any node (it's forwarded to the leader).
3. When you stop the leader, endpoint status briefly errors for ~1–2 s (election in progress), then a new leader appears with a higher raft_term.
4. The key you wrote is still readable — no data lost.
5. Stop a second node — the remaining single node can no longer form a majority, and both reads and writes fail. This is the quorum requirement in action.

Try this modification: re-run with 5 nodes and confirm the cluster still serves traffic after killing any 2. Then try to kill 3 — the cluster halts. That's the difference between f=1 and f=2 fault tolerance.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Every widely deployed strongly consistent system today is a Raft implementation or a Raft variant.

Kubernetes. The entire cluster state lives in etcd, a Raft cluster of 3 or 5 nodes. If etcd loses quorum, kubectl stops working and pods can't be scheduled — the cluster keeps running but you can't change anything. Real outages: several public postmortems of Kubernetes clusters where etcd disks filled up and consensus stalled.

CockroachDB & TiDB. Split the key space into thousands of ranges, each range is its own Raft group. Millions of Raft groups across a cluster — the engineering work is in making Raft cheap enough per range.

Kafka's KRaft mode (2022+) replaced ZooKeeper (a Paxos variant, ZAB) with a self-hosted Raft quorum controller — because operating two consensus systems (ZK + Kafka) was the #1 pain point for Kafka operators.

Common bugs & pitfalls:

• Even-node clusters. Someone runs a 2-node "HA" etcd. It survives zero failures — losing either breaks quorum. Always run odd numbers.
• Cross-region Raft = pain. Every commit waits for a majority-round-trip. Put your Raft cluster in one region, or accept ~100 ms commit latency.
• The "learner" role. Adding a node to a Raft group requires it to catch up first; if you make it a full voter immediately, you can accidentally break quorum. etcd, TiKV, and CockroachDB all support learners — non-voting members that catch up first.
• Disk stalls kill Raft. Raft assumes fsync completes; a slow SAN or full disk causes the leader to lose its own election. Monitor fsync p99 on every Raft node.
• Split-brain "impossible" — but re-joined old leaders. When a partitioned old leader rejoins, it must step down before serving stale reads. Bugs here are subtle; use battle-tested libraries (etcd/raft, hashicorp/raft, tikv/raft) rather than rolling your own.

Rule of thumb from ops teams: don't implement Raft, use an existing implementation. The paper is readable, the corner cases are not.

(e) Quiz + exercise · 10 min

1. Why does a Raft cluster need an odd number of nodes?
2. What are Raft's three sub-problems, and what does each solve?
3. In one sentence, why is Raft considered easier to understand than Paxos?
4. What is the "election timeout" and why is it randomised?
5. If a leader commits an entry to its log but crashes before informing followers, is the entry considered committed? Why?

Stretch (connects to S072 · Consistency Models): Raft gives you linearizable writes for free, but linearizable reads actually require an extra step. Explain what a "read-index" or "lease read" is and why it's necessary, referring to the definition of linearizability from S072.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Consensus? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S074): Sharding & Partitioning Strategies
• Previous (S072): Consistency Models — Linearizable, Sequential, Eventual
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.