Multi-Region — Active-Passive, Active-Active, Failover

Learning goal: Choose an active-passive / pilot-light / active-active topology given RTO/RPO targets, and design a failover plan you'd actually trust at 3 AM.

Why this session exists

A second region can preserve service after a regional failure, or fail in exactly the same way because both depend on the same credentials, configuration rollout or control plane. The difference lies in failure-domain independence, usable data, spare capacity and an executable recovery procedure. Regional diversity is not a guarantee against every outage.

You will be able to

• Define recovery time and recovery point objectives, then distinguish them from observed recovery duration and loss.
• Compare backup/restore, pilot light, warm standby and active-active against a workload's cost and recovery constraints.
• Choose acknowledgment and write-ownership policies for regional failures without assuming cross-region synchronous replication is always good or always bad.
• Design DNS-based failover with readiness checks, cache/connection behavior, fencing and failback.
• Explain data divergence, stale routing and dependency asymmetry through concrete failure traces.

Prerequisites

• S071 · Replication — the primitives you'll wire together across regions.
• S072 · Consistency models — cross-region consistency is where the physics of light bites.
• S073 · Consensus — know the quorum and leader-authority assumptions before placing replicas across regions.

Optional preparation and pacing

The original session suggested a 90-minute study block: preparation, intuition, visual reasoning, a local exercise, production failure analysis and recall. Treat all durations as planning cues, not measured completion times; pause and revisit the worked answers as needed. Optional resources are not prerequisites for running the offline examples. Videos have not been watched or timed here; their availability, creator attribution, contents and quality remain unverified. Search links are searches, not selected talks.

• AWS re:Invent 2024 - Best practices for creating multi-Region architectures on AWS (ARC323) — Optional regional-architecture talk pointer; title/creator are historical finding labels. Speaker identity and the proposed 15–30-minute segment were not verified.
• What is LOAD BALANCING? ⚖️ — Optional load-balancing orientation. DNS, Anycast and latency routing are suggested study topics, not verified contents of this recording; none supplies database authority.
• Postmortems of AWS us-east-1 outages — Search for regional-outage postmortems, not a verified recording. Incident year, affected dependency and observed customer impact need their own source.
• AWS · Disaster Recovery whitepaper — Primary disaster-recovery topology reference; the saved governing passage distinguishes pilot light from warm standby without a universal recovery-time guarantee.
• Active-Active for Multi-Regional Resiliency — Optional historical active-active case study, not evidence here for three-region cost, weekly region failures or zero customer impact.
• Cloud DNS + Health Checks · Route 53 documentation — Original URL was mislabeled as Route 53 documentation: its path names a LinkedIn GraphQL architecture article. Retained only for transparent destination accounting, not as DNS evidence; use the actual Route 53 source below.

(a) Intuition · 5 min

Think of a shop keeping an emergency inventory and ordering terminal at another location. A boxed spare needs installation; a running spare needs enough staff and stock; two live shops need an agreement about which orders and inventory each owns. The analogy highlights preparation and ownership, not a promise of zero interruption.

Backup/restore retains recoverable data plus infrastructure definitions. Pilot light keeps critical data/core services running but activates other capacity on recovery. Warm standby keeps a smaller functional environment running. Active-active serves production traffic in multiple regions; write ownership can still be partitioned by key or coordinated through consensus. “Active-active” does not necessarily mean independent conflicting writes to every row.

Greater preparation can reduce recovery work, but a topology label does not supply an RTO, RPO, cost multiplier or guaranteed availability. An active-active app with all authoritative state in one failed region still has a regional data dependency.

RTO and RPO — the two numbers everything else derives from

RPO is the tolerated recovery-point gap; RTO is the targeted recovery time. They are objectives, not measured achievements. Zero acknowledged-write loss across a region loss usually requires durable copies outside that failure domain before acknowledging, but this does not imply one universal cross-region algorithm. Detection, promotion, fencing, routing, warm capacity and dependent services all contribute to recovery time.

Active-active can mean both regions serve traffic, disjoint write ownership, or concurrent writes to the same objects—state which. Synchronous consensus trades communication latency and quorum availability for an agreed order; asynchronous replication trades a possible loss/divergence window for local progress. Neither is always the right answer. DNS failover is limited by old cache entries, client connection reuse and resolver behavior; lowering TTL now does not erase prior answers. A failover drill must verify data correctness, identity/key access, queues, storage, feature flags and rollback/failback, not just a green HTTP health check.

A quick history — how we got here

DNS's TTL caching model is specified in RFC 1035 (1987); it predates cloud failover and does not provide instantaneous global invalidation. Replicated-log protocols such as Raft formalize safe authority changes, while cloud disaster-recovery guidance distinguishes backup/restore, pilot light, warm standby and multi-site active-active. Their combination explains modern regional recovery: routing, authority and data recovery are separate mechanisms. Historical incident names alone cannot prove that a particular team's design would survive the same failure.

The four topologies

Compare regional designs by what is already running when the primary fails. The progression from backups to active-active buys less recovery work at the cost of more continuously operated infrastructure.

The diagram deliberately attaches no universal RTO/RPO values to topology names. A real failover still has to detect failure, redirect clients, and make usable data available; the sequence below exposes the routing part of that delay.

The failover sequence — what actually happens at 3 AM

Follow a client that has already resolved the primary's address. Health-check failure can change the DNS policy without instantly changing the address that client still has cached.

Routing recovery includes detection and cache expiry, not just flipping a policy. Even when the final request reaches region B, the diagram has not proved that B has current data or enough capacity; those are separate parts of the design below.

The five moving parts of a multi-region design

1. Entry and discovery: DNS policies, load balancers and client retries decide where requests go. Existing connections may outlive a DNS TTL; measure them.
2. Application and capacity: reproduce code, configuration, secrets access, images, quotas and dependencies. A stateless process is not self-sufficient merely because its container starts.
3. Data and ownership: define what acknowledgment survives which loss, how a new owner is selected, and how the old owner is fenced.
4. Identity and sessions: prove the standby can authenticate users, decrypt data and reach its identity/key services. A replicated database without keys is unusable.
5. Recovery and failback: validate user-visible behavior and data invariants, not just /health; plan how the repaired former primary catches up before it can serve again.

Data layer choices — the entire complexity

Asynchronous replication lets a source acknowledge before the remote destination has the change. If the source's only durable copy is destroyed, some acknowledged writes may be absent remotely. Observed lag helps estimate exposure, but it is not a contractual RPO: measurements may be stale and bytes/time do not necessarily identify exactly which transactions are recoverable. Retained log archives and surviving disks can change the outcome.

Synchronous replication/consensus can protect acknowledged writes against a specified regional loss if the acknowledgment set and placement ensure an appropriate durable survivor. The same design may block without its required quorum. If one sequential step requires an 80 ms round trip and local work is 2 ms, that path costs at least about 82 ms under those assumptions; no universal “RTT times two” or “100+ ms every write” follows. Geography and protocol are design inputs, not reasons to dismiss synchronous replication categorically.

Independent regional writers require defined conflict handling. LWW converges by choosing a winner and may discard an acknowledged update; a CRDT preserves its type's semantics, not arbitrary invariants. DynamoDB global tables explicitly distinguish MREC asynchronous replication/conflict resolution from MRSC strong multi-region operation. Do not generalize one mode to all global databases.

Home-region ownership avoids concurrent cross-region writers for a key while that ownership is enforced. Reassignment during failure must be fenced/versioned. Requests for another region's users, global constraints and shared inventory still need coordination. Backups protect against more than machine loss—especially corruption or operator mistakes if versions are isolated and retained. Daily backups without intervening logs can leave nearly a day's gap, plus backup completion/transfer timing; “exactly one-day RPO” is not automatic.

Side by side — three worked design choices

These are hypothetical requirements, not measured company architectures:

Pilot light reduces always-on application capacity but adds activation work. Active-active exercises both environments continuously but still needs spare capacity and recovery of affected ownership. Compare compute, replication egress, storage, standby reservations, operations and drill costs; there is no fixed 1.2× or 3–5× multiplier.

The mental model to hold

Failover moves authority as well as traffic. A primary can still accept writes while the health checker cannot reach it. Routing new users to B does not stop old users with cached addresses from writing A. A safe promotion either prevents A's writes or ensures the data authority rejects A's old epoch. CRDT merging is an alternative data contract, not fencing for a single-writer ledger.

Safety versus liveness: keeping both writers disabled while authority is uncertain can preserve safety but miss RTO. Promoting instantly can restore apparent availability while violating no-conflicting-writers safety. A runbook must make that trade explicit; “automatic” versus “manual” alone does not determine correctness. Automate validated steps and define operator escalation for ambiguous state.

Capacity arithmetic: two regions each at 60% of capacity cannot necessarily survive one loss: the survivor would need 120%. Three equal regions each at 60% need 90% in each of two survivors after redistribution. Those calculations assume equal capacity and divisible work; a hot home partition or global dependency can still defeat them. Include retry/replay traffic and degraded downstream capacity, not just normal requests.

Common-mode failure: replicating a corrupt write or deploying the same bad configuration everywhere can defeat regional redundancy. Use staged configuration/code rollouts, independently restorable backups and blast-radius controls. Multi-region can reduce some risks while adding others.

State the failure quantifier and legal placement boundary

“Survives one region” should mean every single-region loss in the declared deployment, not only the easiest region to lose. Three voting replicas placed A/A/B have a majority in A: losing B can leave progress, but losing A removes that majority. Putting one voter in A/B/C changes this single-region-loss argument, at the cost of remote communication. Still specify which writes are committed, which durable copies survive and which quorum can promote; replica count alone is not a proof. A two-region network partition is not the same fault as destruction: both sides can still run and receive clients. A single-writer protocol must deny stale authority even if that misses the availability target. Neither DNS nor an app-local epoch variable fences a remote database by itself.

Data residency is a placement constraint, not a routing hint. For a hypothetical policy that permits customer records only in jurisdictions X and Y, enumerate replicas, backups, outbox payloads, dead letters, telemetry, key access and support/export paths before selecting a failover destination. A nearby healthy region Z is ineligible under that policy. This is a requirements exercise, not a statement of any country's law or a compliance certification. Ask the responsible legal/security owner what storage, processing and access restrictions actually apply; enforce and test those restrictions in normal operation, recovery and failback. If only one permitted failure domain exists, do not promise multi-region survival by silently copying data elsewhere.

Worked placement check: with A/A/B voters, one-region-loss availability fails for loss of A even though two replicas are normally local and fast. With A/B/C voters, each single-region removal leaves two voters, under the toy assumption that the survivors can communicate and contain the required committed state. This arithmetic is not a Raft implementation or a regional drill.

DNS cutover, dependency validation and failback

For Route 53 active-passive failover, health-associated records select a healthy primary or the eligible secondary according to the documented policy. Health-check configurations and all-unhealthy cases matter; DNS routing is not a database promotion mechanism.

A hypothetical sequence budgets 30 seconds for detection, 20 for fencing/promotion, up to 60 more for cooperative DNS-cache refresh, and 10 for end-to-end validation: 120 seconds if these phases occur sequentially. Some steps overlap; some clients retain connections or ignore expected cache behavior. RFC TTL constrains cache lifetime, not an upper bound on every user's recovery. Lowering a TTL during the incident cannot shorten an answer already cached with a larger TTL. Preconfigure routing where possible; control-plane updates can themselves be unavailable.

Use a staged readiness signal: liveness says the process runs; readiness checks critical dependencies and current authority; a bounded synthetic transaction checks the user-visible path without mutating real accounts. Avoid one fragile dependency causing every region's probe to fail identically. Monitor from multiple vantage points, include hysteresis, and define when to fail closed rather than route to an unready secondary.

Before declaring recovery, test identity/key access, database reads and guarded writes, object storage, queue production/consumption, secrets/configuration, capacity and external providers. A standby that starts in 30 seconds but cannot decrypt database credentials has not met a 60-second service RTO.

For failback, treat the old primary as stale. Keep it fenced, preserve forensic data, determine divergence, reseed or replay from the new authority, validate a common recovery position, then deliberately transfer ownership and routing. Never merge divergent histories by simply re-enabling bidirectional replication and hoping timestamps choose correctly. Backups, replay and dedup retention must support this plan.

(c) Hands-on · 25 min

A deterministic logical-time model makes the assumptions visible. It does not simulate TCP, DNS resolvers, cloud APIs, disk durability or a real failover. Writes occur every 10 ms through t=1,000 ms; each reaches B 200 ms later. At t=1,000, process scheduled deliveries and the final local acknowledgment, then destroy A and discard all remaining in-flight deliveries. This deliberately selected failure model determines the result.

writes = {t: f'order-{t}' for t in range(10, 1001, 10)}
failure_ms, lag_ms = 1000, 200
survivor = {t: value for t, value in writes.items() if t + lag_ms <= failure_ms}
missing = set(writes) - set(survivor)
recovered_point_ms = max(survivor)
observed_gap_ms = failure_ms - recovered_point_ms
assert len(writes) == 100 and len(survivor) == 80
assert len(missing) == 20 and observed_gap_ms == 200
 
# A sequential runbook budget, not measured production recovery.
phases_seconds = dict(detect=30, fence_promote=20, route=60, validate=10)
recovery_seconds = sum(phases_seconds.values())
assert recovery_seconds == 120
assert recovery_seconds <= 300  # hypothetical five-minute objective
assert observed_gap_ms <= 60_000  # hypothetical one-minute point objective
 
# Enforced write authority: publishing an epoch and applying writes is atomic
# in this single-threaded toy. Real sinks need the corresponding guarantee.
class Sink:
    def __init__(self):
        self.epoch = 7
        self.value = 0
    def write(self, epoch, value):
        if epoch != self.epoch:
            return False
        self.value = value
        return True
 
sink = Sink()
assert sink.write(7, 10)
sink.epoch = 8  # controlled authority transfer
assert not sink.write(7, 99)
assert sink.write(8, 11) and sink.value == 11
 
# Same-key conflicting updates, unlike unrelated missing keys.
a, b = {'stock': 1}, {'stock': 1}
a['stock'], b['stock'] = 0, 0  # each independently sells the last item
sold_units = 2
assert a == b and sold_units > 1  # convergence alone does not prove no oversell
print('missing writes:', len(missing), 'recovery-point gap ms:', observed_gap_ms)
print('modeled recovery seconds:', recovery_seconds, 'old epoch rejected')

What each block does

The first block computes observed recovery-point gap and missing acknowledged IDs for one failure schedule. Twenty missing writes is a count; 200 ms is a time gap; neither is itself the business RPO, which is a target. If buffered deliveries survive A's failure or WAL is recovered, the outcome changes. The previous random threaded approach obscured these choices; fixed logical time makes the answer reproducible.

The second block adds runbook phases rather than measuring a real RTO. If key access takes 200 extra seconds after these steps, observed recovery becomes 320 seconds and misses the five-minute target even though app promotion succeeded.

The epoch fixture rejects a resumed former owner. The final stock fixture is intentionally stronger than comparing dictionary equality: both regions agree on zero yet sold two units. Divergent copies and business-invariant violations are different checks. Simply sending unique new keys randomly to regions demonstrates missing replication, not necessarily conflicting updates to the same object.

Exercise and answer: what happens if lag is 400 ms in this schedule? Only writes through 600 ms reach B by t=1,000; 60 survive, 40 are missing, and the recovery-point gap is 400 ms. If B's key service is down indefinitely, no finite service recovery duration follows from the model's 120-second routing budget.

(d) Production reality · 15 min

Use progressively scoped drills: tabletop review, isolated dependency tests, staging failover, then a separately approved production exercise with abort thresholds and an incident owner. Do not casually kill a production region to complete a lesson. Record the injected fault, traffic level, acknowledged IDs, recovered data position, recovery duration, user errors and failback outcome. A real regional-destruction test, a network partition and an application crash establish different evidence.

Three failures to rehearse are data divergence (two unfenced writers), routing lag (cached old endpoints and connections), and dependency asymmetry (missing keys, certificates, quotas, queues or authorization in the standby). Add a global bad rollout case because redundant regions can fail together. A successful drill under one fault model is evidence, not a universal future guarantee.

Where this shows up in the rest of the plan

Replication defines acknowledgment and replay positions; consistency defines forbidden observations; queues contributes durable intent and safe replay; sharding contributes ownership epochs. Observability and incident response turn these mechanisms into measured recovery.

(e) Recall + stretch · 10 min

1. RTO versus RPO? Target recovery duration versus tolerated recovery-point gap. Report measured duration, missing IDs and recovered position separately from targets.

2. Four topologies in order? Backup/restore, pilot light, warm standby and active-active generally preprovision increasing amounts of recovery capability. Actual time, loss and cost depend on the implementation; no automatic tripling rule applies.

3. Is synchronous cross-region always a bad idea? No. It can be appropriate when its covered loss guarantee is worth the measured latency/availability cost. Regional sync plus cross-region async protects a different failure scope and does not meet zero-loss regional destruction requirements.

4. Does a 300-second TTL guarantee all users switch by 300 seconds? No. Existing cache ages, health-check timing, client connection reuse and resolver behavior matter. TTL is not a global failover deadline.

5. What does active-active mean? More than one region serves production traffic. Ask separately who may write a particular key, how concurrent writes are ordered/merged and how ownership changes. Partitioned active-active is a legitimate design.

6. Do CRDTs prevent split-brain? They can make particular concurrent updates converge; they do not establish a single writer or preserve every invariant. The two-sales/one-unit example converges and still violates stock safety.

7. Zero loss locally but asynchronous DR? Regional durable consensus can cover specified local failures while asynchronous DR still risks missing acknowledged writes after regional destruction. This is a changed loss objective, not a cheaper way to meet the same zero-loss regional requirement.

8. Which engineering practice makes regional resilience credible? A scoped, authorized drill records surviving acknowledged IDs, user-visible duration, spare capacity and failback. A named company's chaos tooling is an optional case-study topic, not evidence that every team must destroy production monthly.

9. Which runbook failures should be checked? Stale endpoint/service names, missing promotion or key permissions, and a shared dependency in the failed region can each block recovery. Validate these before an incident and repeat after relevant changes; a green process probe does not cover them.

Global commerce exercise and answer

For RTO=5 minutes and RPO=1 minute, a candidate is warm standby for an order partition with asynchronous replication only if the business accepts potential order-state loss and has payment reconciliation. Budget the example's 120-second steps, but validate them at full failover load with 180 seconds of nominal remaining margin. Alert before lag exhausts the one-minute loss budget and define whether writes pause when the safe exposure limit is exceeded. Use an appropriate stronger durable protocol for records that cannot tolerate acknowledged loss. Routing a user to the nearest app must not bypass the data's home-owner policy. Track all dependencies and failback; no drill was performed here.

Explain-out-loud test

Explain why two 60%-loaded regions lack single-region-loss capacity, why DNS is not fencing, and why equal final stock dictionaries can hide overselling. Then state exactly which failures the selected data acknowledgment survives.

Continue with metrics, logs and traces to decide what recovery evidence to collect. Learning plan.

Optional incident pointers and onward study

The original AWS incident pointer, Cloudflare July 2019 report and Netflix active-active pointer above are retained for optional investigation. This review does not verify their detailed incident impact, exact duration, customer experience or drill cadence. Do not reuse the old claims that all multi-region customers were fine, all manual runbooks took hours, or a global rollout is immune to regional design choices.

Use them to ask concrete questions: which control-plane dependency failed, could a bad rule/configuration reach every region, and what did staged rollout or isolation actually protect? Chaos Monkey/Chaos Kong are historical tooling topics to investigate, not interchangeable proofs of a regional outage. Anycast/global load-balancer steering can avoid some DNS-cache behavior but adds its own health, connection and control dependencies; no universal seconds-to-recovery promise follows. Incident response supplies runbook ownership; rate limiting protects constrained surviving capacity; global-service design combines these contracts with replication and consistency.

Sources and review boundary

Full current-body review on 2026-10-10 reverified saved primary passages captured on 2026-10-07 through 2026-10-09 (not a fresh network retrieval); mutable documentation is bound to saved source hashes in the accompanying claim evidence. Numerical examples are explicit teaching assumptions. The bounded Python examples were run locally; no cloud/distributed deployment, PostgreSQL cluster, broker, object store, load benchmark or regional drill was executed. Compilation and diagram parsing do not establish deployed-system correctness.

• Dr — Pilot light and warm standby differ in ready-to-serve capability.; Operational recovery needs exercised procedures.
• RFC 1035 DNS, November 1987 — DNS TTL describes resource-record cache lifetime.
• Route53 — Route 53 active-passive selects eligible healthy resources according to failover policy.
• PostgreSQL 16 warm standby — Asynchronous WAL shipping exposes acknowledged-state loss if the primary copy is destroyed before shipping.
• Pg Failover — PostgreSQL failover needs old-primary exclusion.
• Pg Sync — remote_apply waits for replay/visibility on configured synchronous standbys.
• Dynamo Global — Current DynamoDB global tables distinguish MREC and MRSC.; MREC replication is asynchronous.
• Raft extended paper, 2014 — Raft election safety is term-scoped.
• Spanner Paper — Spanner replication uses Paxos groups in the 2012 design.