Search Tech Journey

Find topics, journeys and posts

6-month learning plan71 / 130
back to blog
systemsintermediate 55m read

S071 · Replication — Leader/Follower, Multi-Leader, Leaderless

Three ways to keep copies of your data in sync — leader/follower (Postgres, MySQL), multi-leader (active-active), leaderless (Dynamo, Cassandra). Sync vs async, replication lag, and the split-brain problem.

⚙️SystemsM08 · Distributed Systems· Session 071 of 130 90 min

🎯 Pick a replication topology (leader/follower, multi-leader, leaderless) for a workload and reason about failover, split-brain, and replication lag.

Why this session exists

Replication is the mechanism behind every ‘we survive a datacentre failure’ story you'll ever hear. It's also the source of split-brain, stale reads, and dropped writes if you get it wrong. Three topologies dominate the industry; this session gives you working intuition + a code demo for each so you can read a doc page (‘we use multi-leader with LWW conflict resolution’) and immediately know what to worry about.

You will be able to
  • Contrast leader/follower, multi-leader, and leaderless replication in one sentence each.
  • Reason about sync vs async replication and their durability vs latency trade-offs.
  • Explain read-your-writes, monotonic-read, and eventual-consistency guarantees at replica level.
  • Diagnose split-brain scenarios and describe the two mainstream defences (quorum + fencing).

Prerequisites

  • S042 · Transactions & ACID — durability and isolation semantics.
  • S070 · CAP & PACELC — the theoretical frame we'll instantiate.
  • S062 · Load balancers — read replicas often sit behind an LB.


(a) Intuition · 5 min

A newspaper with one editor, many editors, or none
🌍 Real world

One editor (leader/follower): the editor writes every story. Copies are printed at 500 newsstands. Readers can pick up any newsstand copy, but every change goes through the editor. If the editor is sick, no new stories go out until someone is promoted.

Multiple editors (multi-leader): New York and London both have editors who can independently publish. Occasionally they publish contradictory stories at the same time; the papers must be reconciled after the fact ("we're going with New York's version").

No editor (leaderless): Any writer can drop off a story at any newsstand. Newsstands gossip changes among themselves. To make sure YOUR story is really out, you drop it off at three newsstands. To be sure you have the LATEST story, you check three newsstands and take the newest.

💻 Code world

Leader/follower (Postgres, MySQL, Redis, MongoDB, most SQL DBs): one write leader; N read followers. Simple mental model, but leader failover is a serious operation.

Multi-leader (some active-active MySQL, CouchDB, historical MongoDB shards): writes accepted anywhere; conflicts inevitable and must be resolved.

Leaderless (Dynamo, Cassandra, Riak, Scylla): any replica can accept any operation; correctness comes from quorum (write to W of N, read from R of N; if W + R > N you're guaranteed at least one replica has the latest write).

The three topologies at a glance

One-line summary each
  • Leader/follower — one node accepts writes; N replicas stream changes. Simplest to reason about; failover is the hard problem.
  • Multi-leader — multiple leaders accept writes independently, replicate to each other. Great for latency at write time; conflict resolution is the price.
  • Leaderless — all replicas are equal peers. Client writes/reads to a quorum. No leader = no failover pain; conflict + repair mechanics move onto the client + background.

The two orthogonal knobs you'll turn

A quick history so the ecosystem makes sense

  1. 1980
    ‘Notes on Distributed Databases’ · IBM
    Early formalization of replication schemes. Most ideas we use today were already sketched.
  2. 1996
    MySQL replication (binlog + slave)
    The pattern that trained a generation. ‘Slave’ was later renamed to ‘replica’.
  3. 2007
    Dynamo paper · Amazon
    Leaderless + quorum reads/writes + vector clocks. The blueprint for Cassandra, Riak, Voldemort.
  4. 2010
    Postgres streaming replication + hot standby
    Modern leader/follower Postgres emerges. Sync + async modes; replicas readable.
  5. 2014
    Raft published (Ongaro & Ousterhout)
    ‘In Search of an Understandable Consensus Algorithm.’ Powers etcd, Consul, CockroachDB — the modern CP replicated log.
  6. 2020
    Cloud-native primary-primary DBs
    Aurora, Cosmos, Spanner, CockroachDB — hide replication mechanics behind managed APIs.

(b) Visual walkthrough · 15 min

Leader/follower — the classic

Multi-leader — writes accepted in multiple regions

Leaderless — quorum reads/writes

The three topologies compared

Leader / Follower

Postgres · MySQL · MongoDB

  • Writes to leader only; reads from any replica
  • Simple mental model; SQL semantics preserved
  • Failover is a manual (or Orchestrator-managed) event
  • Async replication → potential data loss on leader crash
  • Fit: OLTP, ‘standard SQL app’
Multi-Leader

CouchDB · Cassandra (per-DC) · some MySQL

  • Multiple write regions → local write latency in each
  • Conflicts on same key → app or DB must resolve
  • Standard for multi-DC or offline-first apps
  • Wins on latency, loses on ‘just SQL semantics’
  • Fit: global collaboration, offline mobile sync
Leaderless (Quorum)

Dynamo · Cassandra · Riak · Scylla

  • Every replica peer; client picks W and R per request
  • If W + R > N → guaranteed to see latest write
  • No failover event — nodes come/go, quorum absorbs it
  • Reads may return multiple versions; app picks
  • Fit: high write throughput, high availability

Sync vs async — the durability/latency dial

Every topology can be tuned along this axis

Sync (safe, slow)
Leader waits for AT LEAST ONE replica ack before returning success. If leader crashes, no data lost. Cost: writer's latency = leader disk + slowest replica ack.
safe
Semi-sync / async-with-quorum
Wait for majority (or N-1) replicas — the sweet spot. Postgres synchronous_commit=on with quorum_synchronous_names; MongoDB w:majority; Cassandra QUORUM.
middle
Async (fast, risky)
Leader acks the write once its own disk is fsynced; replicas catch up later. If leader dies before replication, those writes are LOST. Postgres default is close to this.
risky
Per-request tuning
Cassandra: ONE / QUORUM / ALL per read/write. Postgres: synchronous_commit=off inside a transaction. DynamoDB: strongly-consistent read flag. Match durability to importance.
flexible

Common misconception
✗ What most people think

"Adding read replicas scales reads linearly. Ten replicas means ten times the read capacity."

✓ What is actually true

Every replica must apply the full write stream from the primary, so each one does 100% of the write work regardless of how many reads it serves. Replicas scale read throughput; they do nothing for write throughput and they do not reduce per-node write load. On a write-heavy workload, adding replicas adds cost and replication lag without adding meaningful capacity — the write stream is the ceiling and every replica sits under it.

Why the myth is so sticky

The myth is sticky because it is nearly true for read-heavy workloads, which is the case most people meet first and the case vendors demonstrate. It also matches the intuition that more machines means more capacity, which holds for stateless tiers. It breaks precisely where the workload becomes write-heavy: replicas then spend most of their capacity replaying the log, and the marginal read capacity per replica falls as write volume rises.

Prove it to yourself

Measure the fraction of each replica's capacity consumed by replication before adding another one:

-- Postgres: how far behind is the replica, and is it falling?
select client_addr,
       pg_wal_lsn_diff(sent_lsn, replay_lsn) as replay_bytes_behind
from pg_stat_replication;

-- if replay_bytes_behind grows under load, the replica cannot
-- keep up with writes alone -- adding another will not help,
-- and every replica shows the same number for the same reason.
From first principles
Start with the question

Why does read-your-own-writes break with asynchronous replicas, and why is the obvious fix — route that user to the primary — not the real answer?

  1. 1
    A write is acknowledged once the primary has durably recorded it, before replicas have applied it.
    forced by · waiting for replicas would make write latency depend on the slowest replica, which is the whole point of asynchronous replication
  2. 2
    A subsequent read may be routed to any replica, and the router has no information about which writes that replica has applied.
    forced by · load balancers route on connection or load, not on causal history, and they have no notion of a session's writes
  3. 3
    Therefore a user can write successfully and then read stale data, observing their own change disappear — which users interpret as data loss, not as eventual consistency.
    forced by · the user's causal expectation is violated even though the system is behaving exactly as designed
  4. 4
    Pinning the user to the primary fixes it, but concentrates all reads-after-write on the single node you were trying to protect, and the pin must persist for at least the replication lag.
    forced by · you have converted a consistency problem into a capacity problem on the least scalable component
  5. 5
    The general fix is to make the read carry a token — the log position of the user's last write — and route to any replica that has caught up to it, waiting briefly if none has.
    forced by · consistency is needed only relative to that session's own writes, so the guarantee should be scoped to the session rather than to the whole system
⇒ Therefore

Therefore read-your-writes is a session-scoped guarantee, and implementing it correctly means propagating a position token, not pinning to a node. That is why databases expose log sequence numbers to clients at all.

And note what this predicts: the same mechanism gives you monotonic reads — never seeing time go backwards — because carrying the highest position you have observed prevents routing to a replica that is behind it. Both anomalies have one cause and one fix. It also predicts why "just wait a second before reading" appears in so many codebases: it is an unreliable approximation of a token, and it fails exactly when lag exceeds the guess, which is under load.

Mental modelOne log, many players

The primary writes an ordered log of changes. Every replica replays that log in order. A replica's state is entirely determined by how far through the log it has got, so "lag" is a position difference, not a vague slowness.

Every replication question reduces to two: how far behind is a replica allowed to be before we acknowledge a write, and what happens to log entries the primary had but never sent when it dies.

  • Single-leader is the default because it makes conflicts impossible by construction: one node decides the order. Multi-leader and leaderless buy write availability and locality, and pay for it with conflict resolution you must design yourself.
  • Replication lag is a queue, and queues under sustained overload grow without bound. If the primary's write rate exceeds a replica's apply rate even slightly, lag grows forever — so alert on lag trend, not just on an absolute threshold.
  • Semi-synchronous replication — wait for one replica, not all — is the practical middle: it bounds data loss to zero for a single-node failure while keeping latency at one local round trip rather than the slowest replica's.
  • Failover has two distinct dangers: promoting a replica that is behind loses the writes it never received, and failing to fence the old primary produces two nodes accepting writes. The second is worse, because the divergence is silent and both sides believe they are correct.
🔔 Fires when you see

Fire this model when you see: a user reporting their save "didn't take" · replication lag climbing during peak hours · a report showing different numbers on refresh · a failover that lost recent transactions · two nodes both claiming to be primary.

The tradeoff

Do writes wait for replica acknowledgement before returning success?

Fully asynchronous
+ you gain write latency is purely local, replicas cannot slow the primary down, and a failing or slow replica has no effect on write availability at all. Maximum throughput and the simplest operational behaviour.
− you pay a primary failure loses every write not yet sent. The size of that loss equals the replication lag at the moment of failure — which is largest precisely under heavy load, which is when failures are most likely.
pick when when the data can be reconstructed from another source, or when the business has explicitly accepted a data loss window measured in seconds
Semi-synchronous (wait for one replica)
+ you gain an acknowledged write exists on at least two nodes, so a single-node failure loses nothing. Latency cost is one local round trip, which is small on a datacenter network.
− you pay if the acknowledging replica becomes slow or unreachable, writes stall until a timeout or a fallback to asynchronous — so a replica problem becomes a primary latency problem. Configuring the fallback correctly is subtle and often gets it wrong in one direction or the other.
pick when the sensible default for production transactional systems within a single region
Fully synchronous (wait for all replicas)
+ you gain every replica is guaranteed current, so any of them can be promoted with no data loss and reads from any replica are consistent without tokens or pinning.
− you pay write latency equals the slowest replica's, and write availability requires every replica to be up. Each additional replica therefore reduces availability — the opposite of what people expect from adding redundancy.
pick when a very small number of replicas where losing a single acknowledged write is genuinely unacceptable, and write availability is less important than durability
What a senior engineer actually does

Semi-synchronous within a region, asynchronous across regions. This matches the failure distribution: single-node failures are common and must lose nothing, regional failures are rare and can accept a bounded window.

The number that must be known, monitored and agreed is replication lag, because it is your recovery point objective expressed as a live metric. Not a target in a document — the actual current value. And whatever you configure, verify the failover behaviour by testing it: the difference between a failover design and a failover that works is discovered only by running one, ideally on a schedule rather than during an incident.


(c) Hands-on · 25 min

Two experiments: (1) run Postgres primary + replica in Docker and see async replication lag; (2) simulate leaderless quorum reads/writes in Python and demonstrate the W + R > N guarantee.

#!/usr/bin/env bash# s071-pg-replica.sh Postgres 16 primary + streaming replica in Docker.set -euo pipefail DIR="$HOME/projects/learning/s071"mkdir -p "$DIR" && cd "$DIR"log() { printf "\033[1;36m %s\033[0m\n" "$*"; } log "1/4 Bring up a primary Postgres with replication user + WAL config"docker rm -f pg-primary pg-replica >/dev/null 2>&1 ||

And the leaderless quorum simulator:

"""s071_quorum_sim.py — leaderless replication with W/R tunable per request.
 
Demonstrates:
  * W + R > N guarantees at least one read sees the latest write.
  * W + R <= N allows stale reads.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import random, time
 
 
@dataclass
class Replica:
    name: str
    store: dict[str, tuple[str, int]] = field(default_factory=dict)  # value, version(ts)
 
 
class Cluster:
    def __init__(self, n: int = 3):
        self.replicas = [Replica(f"r{i}") for i in range(n)]
 
    def _rand(self, k: int):  # pick k replicas at random
        return random.sample(self.replicas, k)
 
    def write(self, key: str, value: str, w: int) -> bool:
        ts = time.time_ns()
        acks = 0
        for r in self._rand(len(self.replicas)):
            # simulate 30% of replicas being slow / dropping this write
            if random.random() < 0.3:
                continue
            r.store[key] = (value, ts)
            acks += 1
            if acks >= w:
                return True
        return False
 
    def read(self, key: str, r: int) -> str | None:
        results = []
        for rep in self._rand(len(self.replicas)):
            if key in rep.store and len(results) < r:
                results.append(rep.store[key])
        if not results:
            return None
        # Take the highest version (LWW)
        return max(results, key=lambda vt: vt[1])[0]
 
 
if __name__ == "__main__":
    random.seed(42)
    N = 3
 
    print(f"N={N} replicas")
    for W, R in [(1, 1), (2, 2), (3, 1), (1, 3), (2, 1)]:
        cluster = Cluster(N)
        # write ‘v1’, then ‘v2’ — both with same W
        cluster.write("k", "v1", w=W)
        cluster.write("k", "v2", w=W)
        # read
        val = cluster.read("k", r=R)
        guarantee = "GUARANTEED latest" if W + R > N else "MAY be stale"
        print(f"  W={W} R={R}  W+R={W+R}  read={val!r}   ({guarantee})")

What each block does

Anatomy of the two experiments

wal_level=replica + max_wal_senders
Postgres primary settings that enable streaming to hot standbys. WAL = the write-ahead log, the source of truth for replication.
pg-primary
pg_basebackup -R
Creates a physically-identical initial copy of the primary's data + writes a standby.signal / primary_conninfo. The -R flag makes it start up as a follower.
pg-bootstrap
pg_stat_replication
The primary's view of its followers: bytes written / flushed / replayed. write_lag / flush_lag / replay_lag are your monitoring dashboards.
monitoring
Quorum sim · random drop
30% of writes simulate a slow/failed replica. With W=2 out of N=3, the client still succeeds as long as 2 of 3 accept.
leaderless
W + R &gt; N guarantee
Simple pigeonhole: if writes went to W replicas and reads sample R replicas, and W + R > N, some replica MUST be in both sets — you read at least one copy of the latest write.
quorum
Try itBreak replication by starving the primary of WAL

Recreate the primary with wal_keep_size=1MB and push many INSERTs. If the replica lags more than 1MB of WAL, the primary recycles the segment and the replica can no longer catch up — you'll see errors like requested WAL segment ... has already been removed. Recovery = destroy the replica and re-run pg_basebackup, or use replication slots (Postgres 9.4+) which force the primary to keep WAL until the replica has it. In production, ALWAYS use replication slots for critical replicas.

💡 Hint · Reduce wal_keep_size to 1MB and hammer the primary with writes. Replica falls off; you must re-bootstrap.

(d) Production reality · 15 min

War story GitHub· 201824-hour degraded service · 22 GB data desync
🔥 What broke

GitHub runs MySQL in a leader/follower setup with Orchestrator managing failover. A 43-second network partition triggered a failover to the east-coast cluster. Some writes had committed to the original primary but not yet replicated. When the west-coast leader was promoted, those writes were on the wrong side of a split-brain.

Manual reconciliation took 24 hours; some data required customer coordination to recover.

🧯 The fix
Post-incident: GitHub added stricter fencing on failover, longer replication-lag ceilings before allowing promotion, and now writes are rejected if the primary lost quorum. They wrote one of the best post-mortems in tech.
🎓 Lesson to steal
Failover is a data-integrity operation, not a routing operation. If your primary can accept writes that haven't been replicated, a poorly-timed failover WILL lose data. Solutions: semi-sync replication, fencing, or a consensus protocol (Raft) at the DB level.
Post-mortem
War story Common failure mode · async replication + zero-downtime deploy assumption· 2024widespread
🔥 What broke
Application team assumes ‘we have a read replica, so we can serve reads from it during deploys’. During a big backfill job, replica lag balloons to 5 minutes. Users create a comment on the primary, get redirected to a page that reads from the replica, see ‘comment not found’, refresh, appears, disappears (LB reroutes), reappears. Support tickets explode.
🧯 The fix
Sticky reads: for N seconds after a user's write, route their reads to the primary (or a strongly-consistent replica). This is what most cloud DBs (Aurora, Cosmos) implement transparently now. If you're rolling your own, use a session cookie or the ‘lsn’ (log sequence number) pattern — client tells the replica ‘I need at least LSN=X’, replica waits or forwards to primary.
🎓 Lesson to steal
Read-your-own-writes is a first-class product requirement, not an implementation detail. Design for it or your users will experience ghost updates.
War story Cassandra · community failure mode· 2023repeatable pattern
🔥 What broke
Team runs Cassandra with default consistency LOCAL_ONE for both reads and writes. Under normal ops it's fast and fine. During a node failure, some reads occasionally return stale data — the ‘please stop calling databases CP or AP’ post applies. Users see intermittent inconsistency and blame the app.
🧯 The fix
Use LOCAL_QUORUM for both reads and writes on important data (accounts, orders). LOCAL_ONE for non-critical high-volume reads (metrics). Explicitly document the level per query/table; treat it as important as index choice.
🎓 Lesson to steal
Leaderless DBs move consistency into the API. Choosing the wrong level per query is a silent bug. Codify the choice in your data-access layer; make LOCAL_QUORUM the default for anything with a business-integrity requirement.

Common failure modes to internalise

  • Split-brain — two nodes both think they're leader after a partition. Both accept writes. Reconciliation is manual, often lossy. Defence: fencing (STONITH) + quorum-based leader election.
  • Replication lag on read replicas — user reads their own write and sees the old version. Defence: primary reads for N seconds after write, or session tokens.
  • Async replication + leader crash — writes acked to the client but never replicated. Data loss. Defence: semi-sync replication or a consensus protocol.
  • Replica falls off due to WAL recycling — high write load + small wal_keep_size = replica can't catch up. Defence: replication slots (Postgres) or larger WAL retention.

Where this shows up in the rest of the plan

Replication is the foundation of every distributed data topic ahead
S070 · CAP & PACELC
The theoretical frame; this session is the mechanics.
S072 · Consistency models
Linearizable, causal, session — all defined against a replicated log.
S073 · Consensus (Paxos, Raft)
The algorithms that make leader/follower + leader election safe.
S074 · Sharding & partitioning
Data is split (partitioned) AND each shard is replicated.
S089 · System design · Twitter feed
Followers read from replicas; tweets fan out via a replicated log.
S095 · Multi-region deploys
Cross-region replication is the multi-region resilience story.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

  1. Give one system for each of leader/follower, multi-leader, leaderless.
  2. What's the difference between sync and async replication, and when do you pick each?
  3. What is split-brain, and how does quorum prevent it?

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.