S076 · Multi-Region — Active-Passive, Active-Active, Failover

Module M08: Distributed Systems · Session 76 of 130 · Track: SYS 🌏

What you'll be able to do after this session

• Explain Multi-Region to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S074 — Sharding & Partitioning Strategies

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Multi-Region Architecture Explained (AWS re:Invent, Werner Vogels excerpt) — why and when to go multi-region.
• 🎥 Deep dive (30–60 min) · Global Tables and Multi-Region Databases — AWS re:Invent 2023 — a full session on active-active data.
• 🎥 Hands-on demo (10–20 min) · Route 53 Failover Routing (AWS) — the DNS-level failover mechanism most teams start with.
• 📖 Canonical article · AWS Well-Architected — Reliability Pillar, DR Strategies — the canonical RPO/RTO framing.
• 📖 Book chapter / tutorial · Google SRE Book — Chapter 10: Load Balancing at the Datacenter — foundations for global load balancing.
• 📖 Engineering blog · Cloudflare — How we built rate-limiting capable of scaling to millions of domains — one of the best writeups on truly global systems.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Surviving whole-region outages.

Imagine your company runs its only office in one city, and one day a hurricane knocks out power to the whole region for a week. If everything you own — servers, staff, backup tapes — is in that city, you're out of business for the week. Multi-region architecture is the same idea for software: run your system in multiple geographic regions so that a whole-region outage (AWS us-east-1 went down for six hours in December 2021 — half the internet noticed) doesn't take you down.

Multi-region exists for three related reasons: disaster recovery (survive a region failure), latency (serve users from close by — a user in Sydney hitting a US-east server pays 200 ms per round trip), and compliance (GDPR requires EU user data to stay in the EU). Each reason pushes toward a different architecture.

The three canonical patterns:

1. Active-Passive (warm standby): region A serves all traffic; region B is a read-only replica ready to promote. Cheap, simple, but failover takes minutes.
2. Active-Active: both regions serve traffic; data replicated bi-directionally. Fastest failover (seconds via DNS or Anycast), but conflict resolution is real work.
3. Active-Active with Sharding: EU users go to EU region and own their data there; US users to US. No cross-region writes needed. This is what Netflix, Slack, and Zoom mostly do.

Gotcha to carry forever: the hardest part of multi-region is not the servers — it's the database. Compute is stateless and trivial to replicate; a Postgres primary in region A that must survive region A's death, without split brain, is a genuinely hard problem that costs latency, money, or consistency (usually all three).

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — RPO vs RTO tradeoff.

• RPO (Recovery Point Objective) = how much data can I lose? Depends on replication lag.
• RTO (Recovery Time Objective) = how long can I be down? Depends on failover automation.

Failover mechanics (active-passive):

1. Route 53 health check probes https://api.example.com/health in region A every 10 s.
2. Three consecutive failures over 30 s → mark unhealthy.
3. Route 53 flips the DNS answer for api.example.com from region A's IP to region B's IP.
4. Client resolvers cache the DNS answer for the TTL (60 s is a reasonable choice).
5. Concurrently: promote region B's read replica to primary (usually via pg_ctl promote or an automation like Patroni).
6. Total RTO ≈ 30 s (detect) + 60 s (DNS TTL) + 30 s (DB promote) = ~2 min.

Active-active with a global DB (Spanner / DynamoDB Global Tables / CockroachDB): the database itself replicates writes across regions using consensus. You pay ~50–100 ms per commit (cross-region round trips) but get zero-effort failover and no split brain.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Simulate multi-region failover locally with two Postgres instances + HAProxy for DNS-style failover.

#!/usr/bin/env bash
# mr-demo.sh
set -euo pipefail
 
# 1. Start "us-east" primary + "eu-west" replica (reuse S071 setup)
docker network create mr-net 2>/dev/null || true
 
docker run -d --name pg-us --network mr-net \
  -e POSTGRES_PASSWORD=secret -e POSTGRES_DB=app -p 5432:5432 \
  postgres:16 -c wal_level=replica -c max_wal_senders=5
 
sleep 5
docker exec -u postgres pg-us psql -d app -c \
  "CREATE ROLE repl REPLICATION LOGIN PASSWORD 'repl';"
docker exec -u postgres pg-us bash -c \
  "echo 'host replication repl 0.0.0.0/0 md5' >> /var/lib/postgresql/data/pg_hba.conf"
docker exec -u postgres pg-us psql -c 'SELECT pg_reload_conf();'
 
docker run -d --name pg-eu --network mr-net \
  -e PGPASSWORD=repl -p 5433:5432 postgres:16 bash -c '
    rm -rf /var/lib/postgresql/data/*
    pg_basebackup -h pg-us -U repl -D /var/lib/postgresql/data -Fp -Xs -P -R
    exec postgres -c hot_standby=on'
sleep 8
 
# 2. HAProxy config for "DNS-style" failover
cat > /tmp/haproxy.cfg <<'CFG'
global
  daemon
defaults
  mode tcp
  timeout connect 5s
  timeout client 30s
  timeout server 30s
frontend db_front
  bind *:6432
  default_backend db_back
backend db_back
  option tcp-check
  tcp-check connect
  tcp-check send-binary 00000017
  tcp-check send-binary 00030000
  tcp-check send-binary 7573657200706f7374677265730000
  # Primary preferred; fall back to eu if us dies
  server us pg-us:5432 check inter 2s fall 2 rise 2
  server eu pg-eu:5432 check inter 2s fall 2 rise 2 backup
CFG
 
docker run -d --name lb --network mr-net -p 6432:6432 \
  -v /tmp/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg \
  haproxy:2.9
 
sleep 3
# 3. Write + read via the LB
docker exec -u postgres pg-us psql -d app -c \
  "CREATE TABLE evt(id serial, region text, ts timestamptz default now());"
PGPASSWORD=secret psql -h localhost -p 6432 -U postgres -d app -c \
  "INSERT INTO evt(region) VALUES ('us') RETURNING *;"
 
# 4. Kill us-east, force failover
docker stop pg-us
sleep 5
# Promote eu to primary
docker exec -u postgres pg-eu pg_ctl -D /var/lib/postgresql/data promote
sleep 3
PGPASSWORD=secret psql -h localhost -p 6432 -U postgres -d app -c \
  "INSERT INTO evt(region) VALUES ('eu-after-failover') RETURNING *;"
PGPASSWORD=secret psql -h localhost -p 6432 -U postgres -d app -c \
  "SELECT * FROM evt;"

Observe (checklist):

1. Writes through the LB go to pg-us while it's healthy.
2. When you stop pg-us, HAProxy detects failure in ~4 s (fall 2 * inter 2s) and routes to pg-eu.
3. The row inserted before the failure is preserved on pg-eu (that's RPO ≈ 0 in this demo because we didn't push writes fast enough to build lag).
4. Post-promotion, the "backup" is now primary; if pg-us returns, you have a split-brain risk — you must reprovision it as a follower.
5. Measure RTO: the wall-clock seconds between docker stop and a successful INSERT on pg-eu.

Try this modification: run pgbench -c 20 -T 60 against the primary right before killing it. Now the failover loses data (unreplicated WAL). Compute your actual RPO by comparing SELECT max(id) FROM evt; before and after — this is exactly what teams see in real DR drills.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Multi-region is where AWS bills stop being "reasonable" and where architectural decisions become permanent.

AWS us-east-1 outage, December 2021. DynamoDB API in us-east-1 was degraded for hours. Half the internet was affected because so many teams "run in us-east-1 only" — including large parts of AWS's own control plane. Post-incident, many teams re-audited what depended on us-east-1 and were horrified.

Common patterns from real teams:

• Netflix runs three regions active-active-active behind Zuul + Atlas + Chaos Monkey. They regularly simulate region failures ("Chaos Kong") in production to make sure failover actually works.
• Slack shards by workspace: your Slack team lives in one region. This is trivially "active-active" from Slack's perspective but not from yours (if your region dies, your workspace is down).
• Stripe runs everything active-active with careful conflict resolution — their public "Idempotency" and "Rate limiter" writeups are worth reading.

The big traps:

• Cross-region latency budget. us-east-1 ↔ eu-west-1 is ~80 ms one way. Every synchronous cross-region call adds this to your P99. A "chatty" service that does 5 cross-region hops per request is dead.
• Data residency. GDPR, India's DPDPA, China's data-localization laws — you cannot just replicate all data to all regions. Design for regional data ownership from day one.
• Split-brain during partition. Two "primaries" both accept writes; reconciling later is either impossible (last-write-wins loses data) or excruciating (manual per-record merges).
• Route 53 health checks aren't magic. They probe every 10 or 30 s from a handful of AWS locations. If your users hit region A from Asia while the health check hits from the US and passes, users still fail. Use client-side probes and multi-CDN too.
• Failover paths that were never tested. Rule from Google SRE: "if you haven't tested it in the last 6 months, it's broken." Do game-days.
• The "cost bomb": running full stack in three regions triples your infra bill and your team's ops load. Only businesses that provably need it (public infra, payments, healthcare, media at scale) should pay this cost.

Golden rule: start single-region. Move to multi-region only when a business case pays for it. Everything else is architecture astronaut cosplay.

(e) Quiz + exercise · 10 min

1. What is the difference between RPO and RTO?
2. Give one concrete failure mode of active-active replication that active-passive avoids.
3. Why is "DNS failover" not instant — what forces the RTO floor to be tens of seconds?
4. What is a "regional shard" and why does it side-step multi-region write conflicts?
5. Name three business or legal reasons a system has to be multi-region.

Stretch (connects to S074 · Sharding): Combine sharding + multi-region. Sketch how you would deploy a Postgres cluster serving 100M users globally with (a) EU data stays in EU, (b) failover within a region takes seconds, (c) failover between regions takes minutes. What shard key would you choose?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Multi-Region? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S077): The 3 Pillars — Metrics, Logs, Traces
• Previous (S075): Message Queues — SQS, RabbitMQ, Kafka as Queue
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.