Search Tech Journey

Find topics, journeys and posts

6-month learning plan72 / 130
back to blog
systemsintermediate 55m read

S072 · Consistency Models — Linearizable, Sequential, Eventual

The hierarchy of promises a distributed system makes about what you'll read after you write. Pick the wrong rung and you either lose money or lose latency.

⚙️SystemsM08 · Distributed Systems· Session 072 of 130 90 min

🎯 Rank the six consistency models from strongest to weakest, name a real system at each rung, and pick the correct one for a shopping cart, a leaderboard, and a bank ledger.

Why this session exists

In a single-node database every write is followed by a read that sees it. In a distributed database that is a promise the system has to actively pay for — with latency, with dollars, with availability during a partition. Consistency models are the menu of promises. If you don't know the menu you either overpay (linearizable everything, 300 ms writes) or under-promise (eventual everything, "why does my balance flicker?"). Every senior backend interview lives in this space, and every real outage in a distributed database traces back to a mismatch between the model the code assumed and the model the store actually provided.

You will be able to
  • Order the six consistency models from strongest to weakest and give one real datastore that sits at each rung.
  • Draw the timing diagram that distinguishes linearizable from sequential from causal consistency.
  • Explain why 'strong consistency' is a marketing term and what to ask instead ('linearizable? per-key? cross-shard?').
  • Pick the right model for a shopping cart, a like counter, a bank ledger, and a chat app — and defend the choice against a PM who wants 'always strong'.
  • Recognise the three canonical anomalies (stale read, non-monotonic read, lost update) in a bug report.

Prerequisites

  • S071 · CAP & PACELC — the trade space this session picks a point in.
  • S068 · Replication basics — you must know what a follower is before you can reason about what it lags by.
  • S044 · Transactions & isolation levels — same shape of problem, one node scope. Consistency models are the multi-node cousin.


(a) Intuition · 5 min

A group chat vs. a live scoreboard
🌍 Real world

Imagine you and three friends are in a group chat, all typing at once. Two things could be true. Version A: everyone sees every message in the exact same order at the exact same moment — as if there is one shared television screen we all watch. Version B: each phone shows messages in the order they arrived on that phone, and if your friend's message took 2 s longer over LTE you saw hers second while everyone else saw hers first.

Version A is expensive. Someone has to coordinate all four phones on every message. Version B is cheap but weird — the "conversation" you replay from your phone is not the one your friend replays from hers.

💻 Code world

Version A is linearizability. Version B is eventual consistency. Every other model on the menu is a compromise between "expensive but obvious" and "cheap but weird". The choice is not a bug — it is a knob you turn per feature.

A stock exchange must be Version A: two "buy" orders at the same instant cannot both win. A YouTube view counter is happily Version B: your view might not show in the count for 30 s and no one cares.

The six-rung ladder — memorise these names

From strongest (most expensive) to weakest (cheapest)
  • Linearizable (aka strong / atomic) — every operation appears to take effect instantaneously at some point between its call and its return. There is a single, real-time-respecting order.
  • Sequential — all clients see operations in the same order, but that order does not have to match real time. Lamport's original definition.
  • Causal — if A happened-before B (you replied to a message), everyone sees A before B. Concurrent operations can be seen in any order.
  • Read-your-writes (aka session) — you always see your own writes. Someone else might not, yet.
  • Monotonic reads — successive reads never go backwards in time. You'll never see a value 'un-update' itself.
  • Eventual — if writes stop, all replicas eventually converge. No guarantee about when or in what order in the meantime.

A quick history — where the models came from

  1. 1979
    Lamport · sequential consistency
    Defined for multiprocessors. Every processor sees the same interleaving; real time doesn't matter.
  2. 1990
    Herlihy & Wing · linearizability
    Added the real-time constraint. This is what you get from a single-node SQL database.
  3. 2007
    Dynamo paper · Amazon
    Chose availability + eventual consistency over linearizability. Shopping cart never says 'sorry, retry'.
  4. 2011
    CAP re-explained · Gilbert & Lynch
    Formal proof that during a partition you cannot have both linearizability and availability.
  5. 2012
    Spanner · Google
    External consistency (= linearizable) across continents, using atomic clocks (TrueTime) to bound uncertainty.
  6. 2017
    Cosmos DB · Azure
    Ships five consistency levels as a per-request knob. First mainstream cloud DB to expose the ladder directly.

(b) Visual walkthrough · 15 min

The ladder as a diagram

The timing anomaly that separates the top three rungs

The four decisions inside "which consistency model?"

1scope
Per-key or across keys?

Linearizable per single key is cheap. Linearizable across two keys (a transaction) needs consensus — 10-100× more expensive.

2direction
Reads or writes or both?

Some systems (DynamoDB) offer 'strongly consistent read' as an opt-in flag on individual reads. Writes are always leader-serialised.

3geography
Inside a region or global?

Linearizable inside one AZ: single Raft group, ~1 ms. Linearizable across continents: needs Paxos rounds crossing oceans, ~200 ms. Physics is not negotiable.

4failure
On the happy path or during a partition?

CAP: during a partition you pick consistency OR availability. Every system takes a side. Know which side yours takes before the incident, not during.

Layers where each model shows up

Where consistency models live in your stack

L7 · Application
'When user clicks Buy, do they see the new stock count?' — this is a product decision that dictates the model below.
product
L6 · SDK / driver
DynamoDB SDK's `ConsistentRead=true` flag; MongoDB's `readConcern: 'linearizable'`; Cassandra's `LOCAL_QUORUM`. The knob lives here.
client
L5 · Query router
Router decides: send to leader (linearizable) or nearest replica (eventual). Cosmos DB literally routes based on the level you asked for.
coordinator
L4 · Replication protocol
Raft/Paxos gives linearizability. Async log shipping gives eventual. Chain replication gives sequential. The protocol IS the model.
engine
L3 · Storage engine
Single-node LSM/BTree is already linearizable — this problem only exists once you replicate.
local

Comparing three real systems side by side

Spanner (Google)

Linearizable, globally

  • TrueTime API: atomic-clock-backed timestamps with bounded uncertainty (typically <7 ms)
  • Every commit waits out the uncertainty interval before returning
  • Cost: ~50-200 ms cross-region writes — you pay for the guarantee
  • Use when: money is on the line and eventual would mean double-spend
DynamoDB (Amazon)

Eventual by default, linearizable per-item on request

  • Default reads: eventual, ~5 ms, cheap
  • ConsistentRead=true: linearizable, 2× the read cost, still ~5 ms in-region
  • No cross-item linearizability without Transactions API (which is slower again)
  • Use when: item-scoped invariants (a single user's balance) suffice
Cassandra

Tunable per-request via QUORUM

  • You pick W (write quorum) and R (read quorum); if W+R > N you get 'strong' reads
  • No real-time guarantee — sequential-ish at best
  • During partition: chooses AP — writes succeed to any reachable replica
  • Use when: massive write throughput matters more than exact ordering

The mental model to hold


Common misconception
✗ What most people think

"Eventual consistency means the data will be correct after a short delay. It's just strong consistency with lag."

✓ What is actually true

Eventual consistency guarantees only that replicas converge if writes stop. It says nothing about how long, and — crucially — nothing about what intermediate states clients observe. Without additional guarantees, a client can read a new value then an old one, see two clients' writes in different orders, or observe a state that never existed on any single replica. The delay is not the problem; the absence of ordering guarantees is.

Why the myth is so sticky

The myth is sticky because in a healthy system convergence usually takes milliseconds, so the observed behaviour genuinely looks like "strong consistency with lag". The distinction surfaces only in the cases that matter: during a partition, under load, or when an application makes a decision from a read. A counter incremented by two clients concurrently does not converge to a delayed-but-correct value — it converges to a wrong one, permanently, unless the data type is designed for it.

Prove it to yourself

Distinguish the two by asking what a sequence of reads can look like:

write(x = 1)      -- acknowledged

read(x) -> 1      -- replica A, caught up
read(x) -> 0      -- replica B, behind
read(x) -> 1      -- replica A again

Under eventual consistency this is LEGAL.
Monotonic reads forbids the second line.
Linearizability forbids it and much more.

If your application would misbehave given that sequence, "eventually consistent" is not a sufficient contract for it.

From first principles
Start with the question

Why is linearizability so expensive, when it sounds like it should just require replicas to agree?

  1. 1
    Linearizability requires that every operation appears to take effect instantaneously at some point between its invocation and its response, in one global order consistent with real time.
    forced by · that is precisely what "behaves like a single copy" means to a client
  2. 2
    For a read to return the latest value, the responding node must know that no more recent write exists anywhere in the system.
    forced by · returning a stale value would place the read before a write that had already completed in real time, violating the order
  3. 3
    A node cannot know this from local state, because a write could have been accepted by another node microseconds ago.
    forced by · knowledge of remote events requires communication, and there is no shared clock precise enough to substitute for it
  4. 4
    Therefore every linearizable operation requires communication with a quorum of nodes, and that communication cannot be skipped even for reads.
    forced by · a majority intersects every other majority, which is the only way to guarantee overlap with the most recent write
  5. 5
    So each operation costs at least one round trip to a quorum, and its latency is bounded below by the distance to the furthest node in that quorum.
    forced by · the slowest member of the required majority determines when the operation can complete
⇒ Therefore

Therefore linearizability's cost is not implementation inefficiency; it is the mandatory cost of establishing shared knowledge across a network. Latency is bounded by geography.

And note what this predicts: a linearizable system spanning continents has a hard latency floor of tens of milliseconds per operation, and no engineering removes it. It also predicts why "fast strongly consistent reads" appear in some systems only with extra machinery — a lease held by the leader, or tightly bounded clocks — because both are ways of establishing in advance that no other node could have accepted a write, thus removing the round trip from the read path. Once you see the derivation, those mechanisms stop being magic and become the only two available answers.

Mental modelA ladder of guarantees, each with a price

Consistency models form a ladder. At the top, linearizability: one copy, real-time order, quorum round trip per operation. Below it, sequential and causal consistency: a shared order, but not tied to real time. Near the bottom, eventual consistency: convergence only, no ordering. Every rung down removes a coordination requirement and buys latency and availability.

You do not pick one for your system. You pick one per operation, and the correct rung is the weakest one your application logic can tolerate.

  • Causal consistency is the sweet spot for most applications: it preserves the orderings users can actually perceive — a reply after its message, a read after your own write — without requiring global agreement. It is achievable without a quorum round trip and it eliminates the anomalies users complain about.
  • The session guarantees are what users experience: read-your-writes, monotonic reads, monotonic writes, writes-follow-reads. Most "eventual consistency is confusing" complaints are actually one of these four being violated, and they can be provided far more cheaply than linearizability.
  • Convergence requires a deterministic conflict resolution rule. Last-writer-wins is simple and silently discards data; CRDTs converge without loss but constrain what operations you can express. Choosing not to decide means last-writer-wins by default, chosen by your database rather than by you.
  • Read-modify-write is the operation that eventual consistency cannot support safely. Any logic that reads a value, computes from it and writes it back needs either a compare-and-swap, a lock, or a commutative data type — otherwise concurrent updates silently overwrite each other.
🔔 Fires when you see

Fire this model when you see: a counter that drifts from its true value · a user seeing their update vanish then reappear · two clients disagreeing about the order of events · check-then-act logic against a replicated store · a design document that says "eventually consistent" with no further detail.

The tradeoff

An inventory system must not oversell. Do you enforce that with strong consistency, or with an eventually consistent store and compensation?

Strong consistency on the stock count
+ you gain overselling is impossible by construction. The logic is simple to write and simple to reason about, and there are no compensating flows, apology emails, or reconciliation processes to build and operate.
− you pay every purchase serialises on a quorum operation, so throughput is bounded and latency includes a coordination round trip. A popular item becomes a contention hotspot, and during a partition the minority side cannot sell at all.
pick when low-volume, high-value items where a single oversell is expensive — event tickets, limited editions, anything where the customer relationship is damaged by cancellation
Eventual consistency plus compensation
+ you gain writes are local and fast, throughput is essentially unbounded, and sales continue during a partition. The system stays available under exactly the conditions that generate the most demand.
− you pay you will oversell sometimes, and you must build the entire compensation path: detection, cancellation, refund, customer communication. That path is business logic, not infrastructure, and it needs ongoing ownership.
pick when high-volume commodity goods where the cost of an occasional cancellation is far below the revenue lost to unavailability or added latency
Reservation with a buffer
+ you gain partition the stock into per-region allocations and sell against a local allocation without coordination. Coordination happens only when an allocation is nearly exhausted, so the common path is fast and the rare path is correct.
− you pay stock is stranded in allocations that are not selling while another region runs out, and the rebalancing logic is a real distributed systems problem in its own right. Utilisation of the last units is poor.
pick when medium-to-high volume with meaningful stock depth, where the buffer size can be tuned so exhaustion is rare
What a senior engineer actually does

Ask what a violation actually costs the business, in money and in customer trust, before choosing. Engineers overwhelmingly default to strong consistency because it is easier to reason about, then spend the following year optimising around contention that a weaker model would never have created.

The reframing that resolves most of these arguments: overselling is not a bug to be eliminated at any cost, it is a business risk to be priced. Physical retail has run on eventually consistent inventory forever and handles the exceptions with a process. If the compensation flow is cheaper than the coordination, build the compensation flow — and if nobody can say what an oversell costs, that is the question to answer before writing any code.


(c) Hands-on · 25 min

Let's see the models with a tiny simulator. We'll model three replicas with variable network delay, apply a workload, and check which histories are legal under each model. No external dependencies — pure Python.

#!/usr/bin/env python3
"""consistency_lab.py — a toy replicated store to feel the six models.
 
Run:  python consistency_lab.py
Deps: standard library only. Tested on 3.10+.
 
We simulate 3 replicas with async replication. A "client" library
lets you write to any replica; replication happens after a random delay.
We then check whether the observed read history is legal under
each consistency model.
"""
from __future__ import annotations
 
import random
import threading
import time
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Callable
 
random.seed(42)
 
 
@dataclass
class Op:
    """A single client operation: write(k,v) or read(k) -> observed value."""
    kind: str           # "W" or "R"
    key: str
    value: str | None   # written value, OR observed value for reads
    ts_call: float      # wall time the client called
    ts_ret: float       # wall time the client saw a return
    client: str
 
 
@dataclass
class Replica:
    """One replica of a key-value store."""
    name: str
    store: dict[str, str] = field(default_factory=dict)
    # Peers to forward writes to (async).
    peers: list["Replica"] = field(default_factory=list)
    # Simulated network delay range in seconds.
    delay: tuple[float, float] = (0.01, 0.05)
 
    def write(self, key: str, value: str) -> None:
        self.store[key] = value
        # Fire-and-forget replication with a delay per peer.
        for peer in self.peers:
            d = random.uniform(*self.delay)
            threading.Timer(d, peer._apply, args=(key, value)).start()
 
    def _apply(self, key: str, value: str) -> None:
        # Last-writer-wins — the simplest (and weakest) merge policy.
        self.store[key] = value
 
    def read(self, key: str) -> str | None:
        return self.store.get(key)
 
 
def run_workload() -> list[Op]:
    """Two clients, three replicas, some writes, some reads."""
    r1, r2, r3 = Replica("r1"), Replica("r2"), Replica("r3")
    r1.peers, r2.peers, r3.peers = [r2, r3], [r1, r3], [r1, r2]
 
    history: list[Op] = []
 
    def op(client: str, kind: str, key: str,
           fn: Callable[[], str | None], value: str | None = None) -> None:
        t0 = time.time()
        result = fn()
        t1 = time.time()
        history.append(Op(kind, key, value if kind == "W" else result,
                          t0, t1, client))
 
    # t=0    C1 writes x=1 to r1
    op("C1", "W", "x", lambda: r1.write("x", "1"), value="1")
    time.sleep(0.005)
    # t=5ms  C2 reads x from r3 — probably still None (write hasn't replicated)
    op("C2", "R", "x", lambda: r3.read("x"))
    time.sleep(0.100)
    # t=105ms C2 reads x again from r3 — now sees "1"
    op("C2", "R", "x", lambda: r3.read("x"))
    # C1 writes x=2 then reads its own write from r1 — must be "2"
    op("C1", "W", "x", lambda: r1.write("x", "2"), value="2")
    op("C1", "R", "x", lambda: r1.read("x"))
    time.sleep(0.150)  # let everyone converge
    op("C2", "R", "x", lambda: r3.read("x"))
    return history
 
 
def check_linearizable(h: list[Op]) -> bool:
    """Real-time order must be respected: if op A returns before op B calls,
    then A appears before B in any valid serialisation."""
    # Simplified: check that every read sees the most recent write
    # whose ts_ret < read's ts_call.
    for op in h:
        if op.kind != "R":
            continue
        latest = None
        for w in h:
            if w.kind == "W" and w.key == op.key and w.ts_ret < op.ts_call:
                if latest is None or w.ts_ret > latest.ts_ret:
                    latest = w
        expected = latest.value if latest else None
        if op.value != expected:
            return False
    return True
 
 
def check_read_your_writes(h: list[Op]) -> bool:
    """After a client writes v to k, its own subsequent reads of k must return
    v or a later value from the same client."""
    latest_own: dict[tuple[str, str], str] = {}
    for op in h:
        if op.kind == "W":
            latest_own[(op.client, op.key)] = op.value  # type: ignore[assignment]
        else:
            expected = latest_own.get((op.client, op.key))
            if expected is not None and op.value != expected:
                return False
    return True
 
 
def check_monotonic_reads(h: list[Op]) -> bool:
    """A client's successive reads of the same key never see an older write."""
    by_client: dict[str, list[Op]] = defaultdict(list)
    for op in h:
        if op.kind == "R":
            by_client[op.client].append(op)
    # For simplicity we treat any transition value -> None as non-monotonic.
    for _c, reads in by_client.items():
        seen: set[str] = set()
        for r in reads:
            if r.value is None and seen:
                return False
            if r.value is not None:
                seen.add(r.value)
    return True
 
 
def check_eventual(h: list[Op], convergence_window: float = 0.5) -> bool:
    """After the last write, given enough time, all reads should agree."""
    last_write = max((o for o in h if o.kind == "W"), key=lambda o: o.ts_ret,
                     default=None)
    if last_write is None:
        return True
    for op in h:
        if op.kind == "R" and op.ts_call > last_write.ts_ret + convergence_window:
            if op.value != last_write.value:
                return False
    return True
 
 
def main() -> None:
    h = run_workload()
    print(f"\n{'time (ms)':>10}  {'client':>6}  op                    result")
    t0 = h[0].ts_call
    for op in h:
        rel = int((op.ts_call - t0) * 1000)
        detail = (f"W({op.key}={op.value})" if op.kind == "W"
                  else f"R({op.key})->{op.value!r}")
        print(f"{rel:>10}  {op.client:>6}  {detail:<20}")
    print()
    print("Legal under linearizable?     ", check_linearizable(h))
    print("Legal under read-your-writes? ", check_read_your_writes(h))
    print("Legal under monotonic reads?  ", check_monotonic_reads(h))
    print("Legal under eventual?         ", check_eventual(h))
 
 
if __name__ == "__main__":
    main()

What each block does

Anatomy of the script

Replica.write + Timer
Applies write locally, then fires threading.Timer to replicate to peers after 10-50 ms — this is exactly how async replication looks in the real world.
replication
_apply · last-writer-wins
The naive merge policy every young distributed store uses. Great for demo, terrible for concurrent writes to the same key — you lose one.
conflict
run_workload
Hand-crafted so C2's first read hits r3 before the write has replicated — a stale read is guaranteed. This is what 'eventual' looks like in a diagram.
scenario
check_linearizable
The most rigorous checker. For every read, find the latest write whose return-time precedes the read's call-time; the read must equal that value.
checker
check_read_your_writes
Weaker: only checks each client sees its OWN latest write. Cross-client staleness is allowed.
checker
check_monotonic_reads
Weakest useful check: values never regress. The 'un-liked post' bug lives here — a monotonic-reads violation.
checker
check_eventual
The floor: given enough silent time, everyone agrees. Fails only if replication itself is broken.
checker
Try itBreak sequential consistency, then decide whether it matters

Modify Replica.__init__ to accept delay_matrix: dict[str, tuple[float,float]], then make the delay from r1→r2 fast (1 ms) and r1→r3 slow (100 ms). Repeat for the reverse writes. Add a third client that reads from r2 and r3 during the window and print both timelines. When they disagree on the order, you have a sequential-consistency violation. This is what the Cassandra "read repair" mechanism exists to paper over.

💡 Hint · Add a third client C3 that reads x from r2 in between C1's two writes. If C2 sees the writes in order [1, 2] and C3 sees them in order [2, 1], sequential consistency is violated. In your simulator this can't happen because writes fan out from a single origin — try replacing the async Timer with a delay dictionary keyed by (source, dest) pair so different peers see different orders.

(d) Production reality · 15 min

War story GitHub· 201824-hour degraded service, 5+ M users affected
🔥 What broke

A 43-second network partition between GitHub's East and West coast data centres caused the MySQL orchestration layer to promote a stale replica to primary in one region while the original primary was still accepting writes in the other. When the partition healed, the two primaries had diverged.

The system tried to reconcile — for hours. Reads returned inconsistent data (a repo would show 47 stars, refresh, show 42) because different app servers hit different sides of the split.

🧯 The fix
Freeze writes globally, manually pick a source-of-truth primary, replay writes from the other side into it by hand. Took 24 hours. Long-term fix: adopt a consensus-backed metadata store (Raft) for topology decisions, so promotion cannot happen without a quorum agreeing.
🎓 Lesson to steal
MySQL replication is eventual under partition — you cannot bolt linearizability on top with an external orchestrator. If you need it, the storage layer itself must implement consensus. This is the entire reason CockroachDB, Spanner, and YugabyteDB exist.
Post-mortem
War story MongoDB (as reported by Jepsen)· 2020Documented across multiple Jepsen reports
🔥 What broke
MongoDB marketed itself as "strongly consistent" for years. Jepsen's tests showed that with the default writeConcern and readConcern, writes acknowledged by the primary could be rolled back if the primary was network-isolated for even a few seconds — because the majority quorum then elected a new primary that didn't have those writes.
🧯 The fix
MongoDB introduced writeConcern: 'majority' + readConcern: 'linearizable', but you have to opt in per operation and it slows writes by 2-3×. The default is still faster + weaker, and most apps never change it.
🎓 Lesson to steal
"Strong consistency" printed on a marketing page is worth exactly as much as the JSON flags you set on every query. Always read the docs on defaults, not on capabilities.
Post-mortem
War story Amazon · shopping cartthe design decision that made Dynamo famous
🔥 What broke
In 2004 Amazon's shopping cart, backed by a linearizable Oracle cluster, would refuse writes during any small network hiccup between racks. During peak traffic (Black Friday) this was measured in millions of "add to cart" failures per hour. Every failed click is money lost.
🧯 The fix
Rewrite the cart on Dynamo (eventually consistent, always-writable). Concurrent writes are kept as a set of "versions" reconciled by the app: if you add item A on one replica and remove item A on another simultaneously, the reconciled cart shows item A. Amazon chose "occasionally an item comes back after you remove it" over "occasionally the site rejects your click".
🎓 Lesson to steal
Availability > consistency for anything that isn't money. The bar to justify linearizability should be "if two things happen concurrently, someone loses cash".
Post-mortem

Where this shows up in the rest of the plan

Consistency models are the vocabulary for the next dozen sessions
S073 · Consensus (Paxos & Raft)
The protocol that lets you achieve linearizability across a cluster. Consistency is the goal; consensus is the mechanism.
S074 · Sharding & partitioning
The moment you shard, cross-key linearizability requires 2PC or Paxos across shards — this is why 'transactions' get slow in NoSQL.
S076 · Multi-region
Consistency across continents is where the physics bites: linearizable + global = 200 ms writes.
S044 · SQL transactions & isolation
Isolation levels (READ COMMITTED, SERIALIZABLE) are the single-node cousins of these models. Same shape, one machine.
S099 · Caching strategies
Every cache is a weaker consistency tier on top of the DB. TTL vs write-through vs read-through are all consistency knobs.
S128 · Design Twitter (interview)
The classic 'timeline eventually consistent, DMs linearizable' interview answer — you'll ace it after this session.

(e) Recall + stretch · 10 min

Quick recall · click to reveal
★ = stretch question

Explain-out-loud test

Teach these three, no notes:

  1. The six-rung ladder from linearizable to eventual, with one real datastore per rung.
  2. Why 'strong consistency' is a meaningless phrase until you say per-what-scope.
  3. One anomaly (stale read / non-monotonic / lost update) and which model would have prevented it.

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.