R16 · Week 16 Recall & Drill

Weekly revision · Week 16 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S076 — Multi-Region — Active-Passive, Active-Active, Failover
• S077 — The 3 Pillars — Metrics, Logs, Traces
• S078 — Prometheus, Grafana, OpenTelemetry — Hands-on
• S079 — SLIs, SLOs & Error Budgets — the SRE Math
• S080 — Incident Response — Runbooks, Postmortems, On-Call

90-min structure

Blank-page reconstruction · 30 min

S076 · Multi-Region

1. Define recovery time objective and recovery point objective in one sentence each.
2. Rank the four topologies by recovery time, slowest to fastest, and say what each costs.
3. Explain why synchronous replication across long distances is usually a bad idea.

Gotcha you probably forgot: name-resolution caching means a failover is not instantaneous even when everything else works. Clients hold the old address for the record's lifetime, and some resolvers and client libraries ignore it and cache longer. Lowering the record lifetime after an outage has begun helps nobody, because the clients that matter already cached the old value — it must be low beforehand, and that costs lookup traffic every day for an event that may never come.

S077 · Three Pillars

1. Define metrics, logs, and traces in one sentence each and give the question class each answers best.
2. Name the four golden signals.
3. Your latency at the high percentile spikes. Which pillar do you consult next, and why?

Gotcha you probably forgot: alerts should be based on aggregate signals, not on individual log lines or traces. Logs are unsampled and detailed, which makes them irresistible to alert on and guarantees noise; traces are sampled, so their absence proves nothing. Alert on the aggregate signal, then use the other two to explain what the alert is telling you.

S078 · Metrics Stack

1. Explain the pull-based scrape model and one operability advantage it gives you.
2. Write the query for an error ratio over a recent window, and one for a high percentile from a histogram.
3. Say what the collector component is for.

Gotcha you probably forgot: averaging latency hides exactly the users who are suffering. An average is dominated by the many fast requests, so a small fraction of very slow ones barely moves it — and that fraction is your incident. Percentiles are what you alert on, and they must be computed from a histogram rather than by averaging pre-computed percentiles, which is not a valid operation.

S079 · SLIs, SLOs & Error Budgets

1. Define the four terms and say how they relate.
2. State the allowed monthly downtime for the common availability targets from memory.
3. Explain burn rate and why it beats a plain threshold on the indicator.

Gotcha you probably forgot: measuring the indicator server-side often misses the failures that matter most. If the request never reached your service — resolution failure, load balancer error, connection refused — your server-side numbers look perfect while users see an outage. The indicator should be measured as close to the user as you can get.

S080 · Incident Response

1. Name the three roles and what each owns.
2. Explain why "mitigate first, diagnose later" is the rule.
3. Say what makes a postmortem blameless, and why that matters practically rather than morally.

Gotcha you probably forgot: a runbook step that says "check the dashboard" is not a runbook step. At three in the morning the responder does not know which dashboard, which panel, or what value is bad. Every step needs the exact link, the exact command, the threshold that distinguishes normal from wrong, and the next action for each outcome.

Hands-on drill · 30 min

Task: compute the numbers this week is actually about — percentiles, cardinality, budget burn — then write one real runbook.

mkdir -p ~/projects/w16-drill && cd ~/projects/w16-drill

Step 1 — why averages lie (8 min)

# percentiles.py
import random
import statistics
 
rng = random.Random(1)
# Realistic shape: most requests fast, a small tail very slow.
latencies = [rng.gauss(50, 10) for _ in range(9_700)] + [rng.gauss(3_000, 400) for _ in range(300)]
latencies = [max(1.0, x) for x in latencies]
 
def pct(data, p):
    s = sorted(data)
    return s[min(len(s) - 1, int(len(s) * p / 100))]
 
print(f"requests={len(latencies)}")
print(f"mean   = {statistics.mean(latencies):8.1f} ms")
print(f"median = {pct(latencies,50):8.1f} ms")
print(f"p95    = {pct(latencies,95):8.1f} ms")
print(f"p99    = {pct(latencies,99):8.1f} ms")
print(f"affected users at p99 threshold: {sum(1 for x in latencies if x > pct(latencies,99)):d}")
 
# The invalid operation people do every day: averaging percentiles.
chunks = [latencies[i:i+1000] for i in range(0, len(latencies), 1000)]
avg_of_p99 = statistics.mean(pct(c, 99) for c in chunks)
print(f"\ntrue overall p99      = {pct(latencies,99):8.1f} ms")
print(f"average of chunk p99s = {avg_of_p99:8.1f} ms   <-- not the same number, and not meaningful")

Expected outcome: the mean sits close to the fast bulk and gives no hint of the tail, while the high percentile lands in the slow group — three percent of requests are hundreds of times slower and the mean barely registers it. Then note that averaging per-chunk percentiles produces a different number from the true overall percentile. That is why percentiles must be computed from histogram buckets at query time rather than pre-computed per instance and averaged; the averaging step is not a valid operation on percentiles, and dashboards that do it are quietly wrong.

Step 2 — cardinality (7 min)

# cardinality.py
labels = {
    "service":    12,
    "endpoint":   40,
    "method":      5,
    "status":      6,
    "region":      3,
}
risky = {"user_id": 500_000, "request_id": 10_000_000, "full_path": 250_000}
 
def series(d):
    n = 1
    for v in d.values():
        n *= v
    return n
 
base = series(labels)
print(f"bounded labels only:        {base:>18,} series")
for name, card in risky.items():
    print(f"+ {name:<12} ({card:>10,}):  {base*card:>18,} series")
 
BYTES_PER_SERIES = 3_000        # a common rough planning figure for active series overhead
print(f"\nrough memory at {BYTES_PER_SERIES} bytes/series:")
print(f"  bounded only : {base*BYTES_PER_SERIES/1e6:>10.1f} MB")
print(f"  with user_id : {base*risky['user_id']*BYTES_PER_SERIES/1e12:>10.1f} TB")

Expected outcome: the bounded label set produces a manageable number of series, and adding a single unbounded label multiplies it by that label's cardinality — turning megabytes into an impossible figure. The number to internalise is that cardinality is multiplicative across labels, so the risk is not "one big label" but any label whose values are unbounded. Identifying data belongs in traces and logs, which are keyed per event rather than per series.

Step 3 — error budget and burn rate (8 min)

# budget.py
MINUTES_PER_30_DAYS = 30 * 24 * 60
 
print("SLO      allowed downtime per 30 days")
for slo in (99.0, 99.9, 99.95, 99.99):
    allowed = MINUTES_PER_30_DAYS * (1 - slo / 100)
    print(f"{slo:>6}%   {allowed:8.1f} min  ({allowed/60:5.2f} h)")
 
SLO = 99.9
budget = MINUTES_PER_30_DAYS * (1 - SLO / 100)
print(f"\nSLO {SLO}% -> budget {budget:.1f} min per 30 days")
 
def burn(observed_error_rate, slo=SLO):
    """Burn rate = how many times faster than sustainable you are consuming budget."""
    allowed_rate = 1 - slo / 100
    return observed_error_rate / allowed_rate
 
for rate, label in [(0.001, "0.1% errors"), (0.01, "1% errors"), (0.14, "14% errors")]:
    b = burn(rate)
    exhausted_hours = (30 * 24) / b if b > 0 else float("inf")
    print(f"{label:<12} burn={b:6.1f}x  budget exhausted in {exhausted_hours:8.1f} h")

Expected outcome: the downtime table is the thing to memorise — each extra nine cuts the allowance by a factor of ten, and the highest target leaves only minutes per month, which is less than a single careless deploy. Then the burn table shows why burn rate is the right alerting signal: a small error rate consumes the budget slowly and does not warrant a page, while a large one exhausts a month's allowance in hours and does. A fixed threshold on the raw error rate cannot distinguish those two cases; burn rate is exactly the normalisation that can.

Step 4 — write one real runbook (7 min)

Create runbook-error-rate.md and fill in every bracket with something specific to a system you actually run. If you cannot fill a bracket, that gap is the finding.

# Runbook: API error rate above SLO burn threshold
 
## 1. Confirm (target: 60 seconds)
- Open: [exact dashboard URL, exact panel name]
- Confirm: error ratio panel above [exact number] for more than [exact duration]
- If below: close the alert, note the flap, tune the alert. Stop here.
 
## 2. Declare (target: 2 minutes)
- Severity: [criteria that distinguish the levels — user impact, not gut feel]
- Roles: incident commander [who], comms [who], operations [who]
- Channel: [exact channel], status page: [exact link]
 
## 3. Mitigate BEFORE diagnosing
Try in order. Stop as soon as the error ratio drops.
1. Recent deploy in the last [duration]? Roll back: `[exact command]`
   Expected: ratio falls within [duration]. If not, continue.
2. One region or instance dominating errors? Drain it: `[exact command]`
3. New feature involved? Disable the flag: `[exact command]`
4. Dependency failing? Enable degraded mode: `[exact command]`
 
## 4. Communicate
- Every [interval] to [audience], even with no news: what is known, what is being tried, next update time.
 
## 5. Only now, diagnose
- Traces for failing requests: [exact link]
- Logs filtered by trace id: [exact query]
 
## 6. Close
- Budget consumed: [number] minutes of [budget] this period
- Postmortem owner and due date: [name], [date]

Expected outcome: a document a stranger could execute half-asleep. The test is that every step has a command or a link, a stated expectation, and a next action for both outcomes. Count how many brackets you could not fill — each one is a real gap in your operational readiness, and finding them now costs nothing.

Gap analysis + next week preview · 10 min

• How many brackets in the runbook could you not fill? That count is a more honest readiness measure than any dashboard.
• Did you know the allowed downtime figures before computing them? Those numbers come up in planning conversations constantly.
• Have you ever actually executed a failover, or only designed one? Write down which, honestly.

Next week (S081–S085) moves into applied system design: designing a URL shortener and a rate limiter; a news feed; a chat system; a notification service; and the design-interview framework for structuring any of them. The reliability, observability, and failover thinking from this week is what turns those designs from diagrams into systems you could actually operate.

Part of the 6-month evergreen learning plan.