S072 · Consistency Models — Linearizable, Sequential, Eventual
The hierarchy of promises a distributed system makes about what you'll read after you write. Pick the wrong rung and you either lose money or lose latency.
🎯 Rank the six consistency models from strongest to weakest, name a real system at each rung, and pick the correct one for a shopping cart, a leaderboard, and a bank ledger.
Why this session exists
In a single-node database every write is followed by a read that sees it. In a distributed database that is a promise the system has to actively pay for — with latency, with dollars, with availability during a partition. Consistency models are the menu of promises. If you don't know the menu you either overpay (linearizable everything, 300 ms writes) or under-promise (eventual everything, "why does my balance flicker?"). Every senior backend interview lives in this space, and every real outage in a distributed database traces back to a mismatch between the model the code assumed and the model the store actually provided.
- Order the six consistency models from strongest to weakest and give one real datastore that sits at each rung.
- Draw the timing diagram that distinguishes linearizable from sequential from causal consistency.
- Explain why 'strong consistency' is a marketing term and what to ask instead ('linearizable? per-key? cross-shard?').
- Pick the right model for a shopping cart, a like counter, a bank ledger, and a chat app — and defend the choice against a PM who wants 'always strong'.
- Recognise the three canonical anomalies (stale read, non-monotonic read, lost update) in a bug report.
Prerequisites
- S071 · CAP & PACELC — the trade space this session picks a point in.
- S068 · Replication basics — you must know what a follower is before you can reason about what it lags by.
- S044 · Transactions & isolation levels — same shape of problem, one node scope. Consistency models are the multi-node cousin.
(a) Intuition · 5 min
Imagine you and three friends are in a group chat, all typing at once. Two things could be true. Version A: everyone sees every message in the exact same order at the exact same moment — as if there is one shared television screen we all watch. Version B: each phone shows messages in the order they arrived on that phone, and if your friend's message took 2 s longer over LTE you saw hers second while everyone else saw hers first.
Version A is expensive. Someone has to coordinate all four phones on every message. Version B is cheap but weird — the "conversation" you replay from your phone is not the one your friend replays from hers.
Version A is linearizability. Version B is eventual consistency. Every other model on the menu is a compromise between "expensive but obvious" and "cheap but weird". The choice is not a bug — it is a knob you turn per feature.
A stock exchange must be Version A: two "buy" orders at the same instant cannot both win. A YouTube view counter is happily Version B: your view might not show in the count for 30 s and no one cares.
The six-rung ladder — memorise these names
- Linearizable (aka strong / atomic) — every operation appears to take effect instantaneously at some point between its call and its return. There is a single, real-time-respecting order.
- Sequential — all clients see operations in the same order, but that order does not have to match real time. Lamport's original definition.
- Causal — if A happened-before B (you replied to a message), everyone sees A before B. Concurrent operations can be seen in any order.
- Read-your-writes (aka session) — you always see your own writes. Someone else might not, yet.
- Monotonic reads — successive reads never go backwards in time. You'll never see a value 'un-update' itself.
- Eventual — if writes stop, all replicas eventually converge. No guarantee about when or in what order in the meantime.
A quick history — where the models came from
- 1979Lamport · sequential consistencyDefined for multiprocessors. Every processor sees the same interleaving; real time doesn't matter.
- 1990Herlihy & Wing · linearizabilityAdded the real-time constraint. This is what you get from a single-node SQL database.
- 2007Dynamo paper · AmazonChose availability + eventual consistency over linearizability. Shopping cart never says 'sorry, retry'.
- 2011CAP re-explained · Gilbert & LynchFormal proof that during a partition you cannot have both linearizability and availability.
- 2012Spanner · GoogleExternal consistency (= linearizable) across continents, using atomic clocks (TrueTime) to bound uncertainty.
- 2017Cosmos DB · AzureShips five consistency levels as a per-request knob. First mainstream cloud DB to expose the ladder directly.
(b) Visual walkthrough · 15 min
The ladder as a diagram
The timing anomaly that separates the top three rungs
The four decisions inside "which consistency model?"
Linearizable per single key is cheap. Linearizable across two keys (a transaction) needs consensus — 10-100× more expensive.
Some systems (DynamoDB) offer 'strongly consistent read' as an opt-in flag on individual reads. Writes are always leader-serialised.
Linearizable inside one AZ: single Raft group, ~1 ms. Linearizable across continents: needs Paxos rounds crossing oceans, ~200 ms. Physics is not negotiable.
CAP: during a partition you pick consistency OR availability. Every system takes a side. Know which side yours takes before the incident, not during.
Layers where each model shows up
Where consistency models live in your stack
Comparing three real systems side by side
Linearizable, globally
- TrueTime API: atomic-clock-backed timestamps with bounded uncertainty (typically <7 ms)
- Every commit waits out the uncertainty interval before returning
- Cost: ~50-200 ms cross-region writes — you pay for the guarantee
- Use when: money is on the line and eventual would mean double-spend
Eventual by default, linearizable per-item on request
- Default reads: eventual, ~5 ms, cheap
- ConsistentRead=true: linearizable, 2× the read cost, still ~5 ms in-region
- No cross-item linearizability without Transactions API (which is slower again)
- Use when: item-scoped invariants (a single user's balance) suffice
Tunable per-request via QUORUM
- You pick W (write quorum) and R (read quorum); if W+R > N you get 'strong' reads
- No real-time guarantee — sequential-ish at best
- During partition: chooses AP — writes succeed to any reachable replica
- Use when: massive write throughput matters more than exact ordering
The mental model to hold
"Eventual consistency means the data will be correct after a short delay. It's just strong consistency with lag."
Eventual consistency guarantees only that replicas converge if writes stop. It says nothing about how long, and — crucially — nothing about what intermediate states clients observe. Without additional guarantees, a client can read a new value then an old one, see two clients' writes in different orders, or observe a state that never existed on any single replica. The delay is not the problem; the absence of ordering guarantees is.
The myth is sticky because in a healthy system convergence usually takes milliseconds, so the observed behaviour genuinely looks like "strong consistency with lag". The distinction surfaces only in the cases that matter: during a partition, under load, or when an application makes a decision from a read. A counter incremented by two clients concurrently does not converge to a delayed-but-correct value — it converges to a wrong one, permanently, unless the data type is designed for it.
Distinguish the two by asking what a sequence of reads can look like:
write(x = 1) -- acknowledged
read(x) -> 1 -- replica A, caught up
read(x) -> 0 -- replica B, behind
read(x) -> 1 -- replica A again
Under eventual consistency this is LEGAL.
Monotonic reads forbids the second line.
Linearizability forbids it and much more.If your application would misbehave given that sequence, "eventually consistent" is not a sufficient contract for it.
Why is linearizability so expensive, when it sounds like it should just require replicas to agree?
- 1Linearizability requires that every operation appears to take effect instantaneously at some point between its invocation and its response, in one global order consistent with real time.forced by · that is precisely what "behaves like a single copy" means to a client
- 2For a read to return the latest value, the responding node must know that no more recent write exists anywhere in the system.forced by · returning a stale value would place the read before a write that had already completed in real time, violating the order
- 3A node cannot know this from local state, because a write could have been accepted by another node microseconds ago.forced by · knowledge of remote events requires communication, and there is no shared clock precise enough to substitute for it
- 4Therefore every linearizable operation requires communication with a quorum of nodes, and that communication cannot be skipped even for reads.forced by · a majority intersects every other majority, which is the only way to guarantee overlap with the most recent write
- 5So each operation costs at least one round trip to a quorum, and its latency is bounded below by the distance to the furthest node in that quorum.forced by · the slowest member of the required majority determines when the operation can complete
Therefore linearizability's cost is not implementation inefficiency; it is the mandatory cost of establishing shared knowledge across a network. Latency is bounded by geography.
And note what this predicts: a linearizable system spanning continents has a hard latency floor of tens of milliseconds per operation, and no engineering removes it. It also predicts why "fast strongly consistent reads" appear in some systems only with extra machinery — a lease held by the leader, or tightly bounded clocks — because both are ways of establishing in advance that no other node could have accepted a write, thus removing the round trip from the read path. Once you see the derivation, those mechanisms stop being magic and become the only two available answers.
Consistency models form a ladder. At the top, linearizability: one copy, real-time order, quorum round trip per operation. Below it, sequential and causal consistency: a shared order, but not tied to real time. Near the bottom, eventual consistency: convergence only, no ordering. Every rung down removes a coordination requirement and buys latency and availability.
You do not pick one for your system. You pick one per operation, and the correct rung is the weakest one your application logic can tolerate.
- Causal consistency is the sweet spot for most applications: it preserves the orderings users can actually perceive — a reply after its message, a read after your own write — without requiring global agreement. It is achievable without a quorum round trip and it eliminates the anomalies users complain about.
- The session guarantees are what users experience: read-your-writes, monotonic reads, monotonic writes, writes-follow-reads. Most "eventual consistency is confusing" complaints are actually one of these four being violated, and they can be provided far more cheaply than linearizability.
- Convergence requires a deterministic conflict resolution rule. Last-writer-wins is simple and silently discards data; CRDTs converge without loss but constrain what operations you can express. Choosing not to decide means last-writer-wins by default, chosen by your database rather than by you.
- Read-modify-write is the operation that eventual consistency cannot support safely. Any logic that reads a value, computes from it and writes it back needs either a compare-and-swap, a lock, or a commutative data type — otherwise concurrent updates silently overwrite each other.
Fire this model when you see: a counter that drifts from its true value · a user seeing their update vanish then reappear · two clients disagreeing about the order of events · check-then-act logic against a replicated store · a design document that says "eventually consistent" with no further detail.
An inventory system must not oversell. Do you enforce that with strong consistency, or with an eventually consistent store and compensation?
Ask what a violation actually costs the business, in money and in customer trust, before choosing. Engineers overwhelmingly default to strong consistency because it is easier to reason about, then spend the following year optimising around contention that a weaker model would never have created.
The reframing that resolves most of these arguments: overselling is not a bug to be eliminated at any cost, it is a business risk to be priced. Physical retail has run on eventually consistent inventory forever and handles the exceptions with a process. If the compensation flow is cheaper than the coordination, build the compensation flow — and if nobody can say what an oversell costs, that is the question to answer before writing any code.
(c) Hands-on · 25 min
Let's see the models with a tiny simulator. We'll model three replicas with variable network delay, apply a workload, and check which histories are legal under each model. No external dependencies — pure Python.
#!/usr/bin/env python3
"""consistency_lab.py — a toy replicated store to feel the six models.
Run: python consistency_lab.py
Deps: standard library only. Tested on 3.10+.
We simulate 3 replicas with async replication. A "client" library
lets you write to any replica; replication happens after a random delay.
We then check whether the observed read history is legal under
each consistency model.
"""
from __future__ import annotations
import random
import threading
import time
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Callable
random.seed(42)
@dataclass
class Op:
"""A single client operation: write(k,v) or read(k) -> observed value."""
kind: str # "W" or "R"
key: str
value: str | None # written value, OR observed value for reads
ts_call: float # wall time the client called
ts_ret: float # wall time the client saw a return
client: str
@dataclass
class Replica:
"""One replica of a key-value store."""
name: str
store: dict[str, str] = field(default_factory=dict)
# Peers to forward writes to (async).
peers: list["Replica"] = field(default_factory=list)
# Simulated network delay range in seconds.
delay: tuple[float, float] = (0.01, 0.05)
def write(self, key: str, value: str) -> None:
self.store[key] = value
# Fire-and-forget replication with a delay per peer.
for peer in self.peers:
d = random.uniform(*self.delay)
threading.Timer(d, peer._apply, args=(key, value)).start()
def _apply(self, key: str, value: str) -> None:
# Last-writer-wins — the simplest (and weakest) merge policy.
self.store[key] = value
def read(self, key: str) -> str | None:
return self.store.get(key)
def run_workload() -> list[Op]:
"""Two clients, three replicas, some writes, some reads."""
r1, r2, r3 = Replica("r1"), Replica("r2"), Replica("r3")
r1.peers, r2.peers, r3.peers = [r2, r3], [r1, r3], [r1, r2]
history: list[Op] = []
def op(client: str, kind: str, key: str,
fn: Callable[[], str | None], value: str | None = None) -> None:
t0 = time.time()
result = fn()
t1 = time.time()
history.append(Op(kind, key, value if kind == "W" else result,
t0, t1, client))
# t=0 C1 writes x=1 to r1
op("C1", "W", "x", lambda: r1.write("x", "1"), value="1")
time.sleep(0.005)
# t=5ms C2 reads x from r3 — probably still None (write hasn't replicated)
op("C2", "R", "x", lambda: r3.read("x"))
time.sleep(0.100)
# t=105ms C2 reads x again from r3 — now sees "1"
op("C2", "R", "x", lambda: r3.read("x"))
# C1 writes x=2 then reads its own write from r1 — must be "2"
op("C1", "W", "x", lambda: r1.write("x", "2"), value="2")
op("C1", "R", "x", lambda: r1.read("x"))
time.sleep(0.150) # let everyone converge
op("C2", "R", "x", lambda: r3.read("x"))
return history
def check_linearizable(h: list[Op]) -> bool:
"""Real-time order must be respected: if op A returns before op B calls,
then A appears before B in any valid serialisation."""
# Simplified: check that every read sees the most recent write
# whose ts_ret < read's ts_call.
for op in h:
if op.kind != "R":
continue
latest = None
for w in h:
if w.kind == "W" and w.key == op.key and w.ts_ret < op.ts_call:
if latest is None or w.ts_ret > latest.ts_ret:
latest = w
expected = latest.value if latest else None
if op.value != expected:
return False
return True
def check_read_your_writes(h: list[Op]) -> bool:
"""After a client writes v to k, its own subsequent reads of k must return
v or a later value from the same client."""
latest_own: dict[tuple[str, str], str] = {}
for op in h:
if op.kind == "W":
latest_own[(op.client, op.key)] = op.value # type: ignore[assignment]
else:
expected = latest_own.get((op.client, op.key))
if expected is not None and op.value != expected:
return False
return True
def check_monotonic_reads(h: list[Op]) -> bool:
"""A client's successive reads of the same key never see an older write."""
by_client: dict[str, list[Op]] = defaultdict(list)
for op in h:
if op.kind == "R":
by_client[op.client].append(op)
# For simplicity we treat any transition value -> None as non-monotonic.
for _c, reads in by_client.items():
seen: set[str] = set()
for r in reads:
if r.value is None and seen:
return False
if r.value is not None:
seen.add(r.value)
return True
def check_eventual(h: list[Op], convergence_window: float = 0.5) -> bool:
"""After the last write, given enough time, all reads should agree."""
last_write = max((o for o in h if o.kind == "W"), key=lambda o: o.ts_ret,
default=None)
if last_write is None:
return True
for op in h:
if op.kind == "R" and op.ts_call > last_write.ts_ret + convergence_window:
if op.value != last_write.value:
return False
return True
def main() -> None:
h = run_workload()
print(f"\n{'time (ms)':>10} {'client':>6} op result")
t0 = h[0].ts_call
for op in h:
rel = int((op.ts_call - t0) * 1000)
detail = (f"W({op.key}={op.value})" if op.kind == "W"
else f"R({op.key})->{op.value!r}")
print(f"{rel:>10} {op.client:>6} {detail:<20}")
print()
print("Legal under linearizable? ", check_linearizable(h))
print("Legal under read-your-writes? ", check_read_your_writes(h))
print("Legal under monotonic reads? ", check_monotonic_reads(h))
print("Legal under eventual? ", check_eventual(h))
if __name__ == "__main__":
main()What each block does
Anatomy of the script
Modify Replica.__init__ to accept delay_matrix: dict[str, tuple[float,float]], then make the delay from r1→r2 fast (1 ms) and r1→r3 slow (100 ms). Repeat for the reverse writes. Add a third client that reads from r2 and r3 during the window and print both timelines. When they disagree on the order, you have a sequential-consistency violation. This is what the Cassandra "read repair" mechanism exists to paper over.
(d) Production reality · 15 min
A 43-second network partition between GitHub's East and West coast data centres caused the MySQL orchestration layer to promote a stale replica to primary in one region while the original primary was still accepting writes in the other. When the partition healed, the two primaries had diverged.
The system tried to reconcile — for hours. Reads returned inconsistent data (a repo would show 47 stars, refresh, show 42) because different app servers hit different sides of the split.
writeConcern and readConcern, writes acknowledged by the primary could be rolled back if the primary was network-isolated for even a few seconds — because the majority quorum then elected a new primary that didn't have those writes.writeConcern: 'majority' + readConcern: 'linearizable', but you have to opt in per operation and it slows writes by 2-3×. The default is still faster + weaker, and most apps never change it.Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
Teach these three, no notes:
- The six-rung ladder from linearizable to eventual, with one real datastore per rung.
- Why 'strong consistency' is a meaningless phrase until you say per-what-scope.
- One anomaly (stale read / non-monotonic / lost update) and which model would have prevented it.
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.