S071 · Replication — Leader/Follower, Multi-Leader, Leaderless
Three ways to keep copies of your data in sync — leader/follower (Postgres, MySQL), multi-leader (active-active), leaderless (Dynamo, Cassandra). Sync vs async, replication lag, and the split-brain problem.
🎯 Pick a replication topology (leader/follower, multi-leader, leaderless) for a workload and reason about failover, split-brain, and replication lag.
Why this session exists
Replication is the mechanism behind every ‘we survive a datacentre failure’ story you'll ever hear. It's also the source of split-brain, stale reads, and dropped writes if you get it wrong. Three topologies dominate the industry; this session gives you working intuition + a code demo for each so you can read a doc page (‘we use multi-leader with LWW conflict resolution’) and immediately know what to worry about.
- Contrast leader/follower, multi-leader, and leaderless replication in one sentence each.
- Reason about sync vs async replication and their durability vs latency trade-offs.
- Explain read-your-writes, monotonic-read, and eventual-consistency guarantees at replica level.
- Diagnose split-brain scenarios and describe the two mainstream defences (quorum + fencing).
Prerequisites
- S042 · Transactions & ACID — durability and isolation semantics.
- S070 · CAP & PACELC — the theoretical frame we'll instantiate.
- S062 · Load balancers — read replicas often sit behind an LB.
(a) Intuition · 5 min
One editor (leader/follower): the editor writes every story. Copies are printed at 500 newsstands. Readers can pick up any newsstand copy, but every change goes through the editor. If the editor is sick, no new stories go out until someone is promoted.
Multiple editors (multi-leader): New York and London both have editors who can independently publish. Occasionally they publish contradictory stories at the same time; the papers must be reconciled after the fact ("we're going with New York's version").
No editor (leaderless): Any writer can drop off a story at any newsstand. Newsstands gossip changes among themselves. To make sure YOUR story is really out, you drop it off at three newsstands. To be sure you have the LATEST story, you check three newsstands and take the newest.
Leader/follower (Postgres, MySQL, Redis, MongoDB, most SQL DBs): one write leader; N read followers. Simple mental model, but leader failover is a serious operation.
Multi-leader (some active-active MySQL, CouchDB, historical MongoDB shards): writes accepted anywhere; conflicts inevitable and must be resolved.
Leaderless (Dynamo, Cassandra, Riak, Scylla): any replica can accept any operation; correctness comes from quorum (write to W of N, read from R of N; if W + R > N you're guaranteed at least one replica has the latest write).
The three topologies at a glance
- Leader/follower — one node accepts writes; N replicas stream changes. Simplest to reason about; failover is the hard problem.
- Multi-leader — multiple leaders accept writes independently, replicate to each other. Great for latency at write time; conflict resolution is the price.
- Leaderless — all replicas are equal peers. Client writes/reads to a quorum. No leader = no failover pain; conflict + repair mechanics move onto the client + background.
The two orthogonal knobs you'll turn
A quick history so the ecosystem makes sense
- 1980‘Notes on Distributed Databases’ · IBMEarly formalization of replication schemes. Most ideas we use today were already sketched.
- 1996MySQL replication (binlog + slave)The pattern that trained a generation. ‘Slave’ was later renamed to ‘replica’.
- 2007Dynamo paper · AmazonLeaderless + quorum reads/writes + vector clocks. The blueprint for Cassandra, Riak, Voldemort.
- 2010Postgres streaming replication + hot standbyModern leader/follower Postgres emerges. Sync + async modes; replicas readable.
- 2014Raft published (Ongaro & Ousterhout)‘In Search of an Understandable Consensus Algorithm.’ Powers etcd, Consul, CockroachDB — the modern CP replicated log.
- 2020Cloud-native primary-primary DBsAurora, Cosmos, Spanner, CockroachDB — hide replication mechanics behind managed APIs.
(b) Visual walkthrough · 15 min
Leader/follower — the classic
Multi-leader — writes accepted in multiple regions
Leaderless — quorum reads/writes
The three topologies compared
Postgres · MySQL · MongoDB
- Writes to leader only; reads from any replica
- Simple mental model; SQL semantics preserved
- Failover is a manual (or Orchestrator-managed) event
- Async replication → potential data loss on leader crash
- Fit: OLTP, ‘standard SQL app’
CouchDB · Cassandra (per-DC) · some MySQL
- Multiple write regions → local write latency in each
- Conflicts on same key → app or DB must resolve
- Standard for multi-DC or offline-first apps
- Wins on latency, loses on ‘just SQL semantics’
- Fit: global collaboration, offline mobile sync
Dynamo · Cassandra · Riak · Scylla
- Every replica peer; client picks W and R per request
- If W + R > N → guaranteed to see latest write
- No failover event — nodes come/go, quorum absorbs it
- Reads may return multiple versions; app picks
- Fit: high write throughput, high availability
Sync vs async — the durability/latency dial
Every topology can be tuned along this axis
"Adding read replicas scales reads linearly. Ten replicas means ten times the read capacity."
Every replica must apply the full write stream from the primary, so each one does 100% of the write work regardless of how many reads it serves. Replicas scale read throughput; they do nothing for write throughput and they do not reduce per-node write load. On a write-heavy workload, adding replicas adds cost and replication lag without adding meaningful capacity — the write stream is the ceiling and every replica sits under it.
The myth is sticky because it is nearly true for read-heavy workloads, which is the case most people meet first and the case vendors demonstrate. It also matches the intuition that more machines means more capacity, which holds for stateless tiers. It breaks precisely where the workload becomes write-heavy: replicas then spend most of their capacity replaying the log, and the marginal read capacity per replica falls as write volume rises.
Measure the fraction of each replica's capacity consumed by replication before adding another one:
-- Postgres: how far behind is the replica, and is it falling?
select client_addr,
pg_wal_lsn_diff(sent_lsn, replay_lsn) as replay_bytes_behind
from pg_stat_replication;
-- if replay_bytes_behind grows under load, the replica cannot
-- keep up with writes alone -- adding another will not help,
-- and every replica shows the same number for the same reason.Why does read-your-own-writes break with asynchronous replicas, and why is the obvious fix — route that user to the primary — not the real answer?
- 1A write is acknowledged once the primary has durably recorded it, before replicas have applied it.forced by · waiting for replicas would make write latency depend on the slowest replica, which is the whole point of asynchronous replication
- 2A subsequent read may be routed to any replica, and the router has no information about which writes that replica has applied.forced by · load balancers route on connection or load, not on causal history, and they have no notion of a session's writes
- 3Therefore a user can write successfully and then read stale data, observing their own change disappear — which users interpret as data loss, not as eventual consistency.forced by · the user's causal expectation is violated even though the system is behaving exactly as designed
- 4Pinning the user to the primary fixes it, but concentrates all reads-after-write on the single node you were trying to protect, and the pin must persist for at least the replication lag.forced by · you have converted a consistency problem into a capacity problem on the least scalable component
- 5The general fix is to make the read carry a token — the log position of the user's last write — and route to any replica that has caught up to it, waiting briefly if none has.forced by · consistency is needed only relative to that session's own writes, so the guarantee should be scoped to the session rather than to the whole system
Therefore read-your-writes is a session-scoped guarantee, and implementing it correctly means propagating a position token, not pinning to a node. That is why databases expose log sequence numbers to clients at all.
And note what this predicts: the same mechanism gives you monotonic reads — never seeing time go backwards — because carrying the highest position you have observed prevents routing to a replica that is behind it. Both anomalies have one cause and one fix. It also predicts why "just wait a second before reading" appears in so many codebases: it is an unreliable approximation of a token, and it fails exactly when lag exceeds the guess, which is under load.
The primary writes an ordered log of changes. Every replica replays that log in order. A replica's state is entirely determined by how far through the log it has got, so "lag" is a position difference, not a vague slowness.
Every replication question reduces to two: how far behind is a replica allowed to be before we acknowledge a write, and what happens to log entries the primary had but never sent when it dies.
- Single-leader is the default because it makes conflicts impossible by construction: one node decides the order. Multi-leader and leaderless buy write availability and locality, and pay for it with conflict resolution you must design yourself.
- Replication lag is a queue, and queues under sustained overload grow without bound. If the primary's write rate exceeds a replica's apply rate even slightly, lag grows forever — so alert on lag trend, not just on an absolute threshold.
- Semi-synchronous replication — wait for one replica, not all — is the practical middle: it bounds data loss to zero for a single-node failure while keeping latency at one local round trip rather than the slowest replica's.
- Failover has two distinct dangers: promoting a replica that is behind loses the writes it never received, and failing to fence the old primary produces two nodes accepting writes. The second is worse, because the divergence is silent and both sides believe they are correct.
Fire this model when you see: a user reporting their save "didn't take" · replication lag climbing during peak hours · a report showing different numbers on refresh · a failover that lost recent transactions · two nodes both claiming to be primary.
Do writes wait for replica acknowledgement before returning success?
Semi-synchronous within a region, asynchronous across regions. This matches the failure distribution: single-node failures are common and must lose nothing, regional failures are rare and can accept a bounded window.
The number that must be known, monitored and agreed is replication lag, because it is your recovery point objective expressed as a live metric. Not a target in a document — the actual current value. And whatever you configure, verify the failover behaviour by testing it: the difference between a failover design and a failover that works is discovered only by running one, ideally on a schedule rather than during an incident.
(c) Hands-on · 25 min
Two experiments: (1) run Postgres primary + replica in Docker and see async replication lag; (2) simulate leaderless quorum reads/writes in Python and demonstrate the W + R > N guarantee.
And the leaderless quorum simulator:
"""s071_quorum_sim.py — leaderless replication with W/R tunable per request.
Demonstrates:
* W + R > N guarantees at least one read sees the latest write.
* W + R <= N allows stale reads.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import random, time
@dataclass
class Replica:
name: str
store: dict[str, tuple[str, int]] = field(default_factory=dict) # value, version(ts)
class Cluster:
def __init__(self, n: int = 3):
self.replicas = [Replica(f"r{i}") for i in range(n)]
def _rand(self, k: int): # pick k replicas at random
return random.sample(self.replicas, k)
def write(self, key: str, value: str, w: int) -> bool:
ts = time.time_ns()
acks = 0
for r in self._rand(len(self.replicas)):
# simulate 30% of replicas being slow / dropping this write
if random.random() < 0.3:
continue
r.store[key] = (value, ts)
acks += 1
if acks >= w:
return True
return False
def read(self, key: str, r: int) -> str | None:
results = []
for rep in self._rand(len(self.replicas)):
if key in rep.store and len(results) < r:
results.append(rep.store[key])
if not results:
return None
# Take the highest version (LWW)
return max(results, key=lambda vt: vt[1])[0]
if __name__ == "__main__":
random.seed(42)
N = 3
print(f"N={N} replicas")
for W, R in [(1, 1), (2, 2), (3, 1), (1, 3), (2, 1)]:
cluster = Cluster(N)
# write ‘v1’, then ‘v2’ — both with same W
cluster.write("k", "v1", w=W)
cluster.write("k", "v2", w=W)
# read
val = cluster.read("k", r=R)
guarantee = "GUARANTEED latest" if W + R > N else "MAY be stale"
print(f" W={W} R={R} W+R={W+R} read={val!r} ({guarantee})")What each block does
Anatomy of the two experiments
Recreate the primary with wal_keep_size=1MB and push many INSERTs. If the replica lags more than 1MB of WAL, the primary recycles the segment and the replica can no longer catch up — you'll see errors like requested WAL segment ... has already been removed. Recovery = destroy the replica and re-run pg_basebackup, or use replication slots (Postgres 9.4+) which force the primary to keep WAL until the replica has it. In production, ALWAYS use replication slots for critical replicas.
(d) Production reality · 15 min
GitHub runs MySQL in a leader/follower setup with Orchestrator managing failover. A 43-second network partition triggered a failover to the east-coast cluster. Some writes had committed to the original primary but not yet replicated. When the west-coast leader was promoted, those writes were on the wrong side of a split-brain.
Manual reconciliation took 24 hours; some data required customer coordination to recover.
Common failure modes to internalise
- Split-brain — two nodes both think they're leader after a partition. Both accept writes. Reconciliation is manual, often lossy. Defence: fencing (STONITH) + quorum-based leader election.
- Replication lag on read replicas — user reads their own write and sees the old version. Defence: primary reads for N seconds after write, or session tokens.
- Async replication + leader crash — writes acked to the client but never replicated. Data loss. Defence: semi-sync replication or a consensus protocol.
- Replica falls off due to WAL recycling — high write load + small wal_keep_size = replica can't catch up. Defence: replication slots (Postgres) or larger WAL retention.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- Give one system for each of leader/follower, multi-leader, leaderless.
- What's the difference between sync and async replication, and when do you pick each?
- What is split-brain, and how does quorum prevent it?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.