R16 · Week 16 Recall & Drill

Assessment scope

Attempt the recall prompts unaided, then work through the implementations and answer keys. The local programs use explicit fixtures and independent checks; product behavior is tied to named primary sources, not inferred from the toy models. Service deployment and production readiness are separate from this conceptual review. Timing budgets are adjustable practice suggestions.

Sessions covered

• S076 — Multi-Region — Active-Passive, Active-Active, Failover
• S077 — The 3 Pillars — Metrics, Logs, Traces
• S078 — Prometheus, Grafana, OpenTelemetry — Hands-on
• S079 — SLIs, SLOs & Error Budgets — the SRE Math
• S080 — Incident Response — Runbooks, Postmortems, On-Call

By the end of this revision you can

• Define recovery time and recovery point objectives and derive a topology choice from numeric targets.
• Say which observability pillar answers which class of question, and which one to reach for first in a given incident.
• Write the common query shapes for rate, error ratio, and a latency percentile, and explain why averaging latency misleads.
• Explain what unbounded label values do to a metrics store and how series count, churn, sample rate and retention contribute to cost.
• Compute allowed downtime for common availability targets from memory, and design a burn-rate alert.
• Name the three incident roles and explain why mitigation precedes root cause.

90-min structure

The 90-minute budget is pacing guidance, not a claim that full execution and every extension fit every learner's session.

Blank-page reconstruction · 30 min

S076 · Multi-Region

1. Define recovery time objective and recovery point objective in one sentence each.
2. Compare backup/restore, pilot light, warm standby and active-active by their recovery work and running cost.
3. Explain the latency and failure tradeoffs of synchronous cross-region replication.

Answer key — start with numeric objectives. RTO is the target maximum time to restore the defined service after disruption; RPO bounds the acceptable age/loss window of recoverable data. Neither is a measured result merely because it appears in a diagram. Backup/restore must provision and restore, pilot light retains critical data/core infrastructure but must start/scale serving components, warm standby runs a smaller complete environment, and active-active serves in multiple locations already. This is a usual progression toward less recovery work and higher standing/coordination cost, not a guaranteed ordering of achieved RTO.

Suppose RTO is 15 minutes and RPO is 60 seconds. A tested 2-hour restore cannot meet the RTO. A warm standby is a candidate only if detection+decision+promotion+routing+warmup completes within 15 minutes, available replication/checkpoint lag is at most 60 seconds, and surviving capacity handles the declared critical workload. If 2 minutes detection + 3 promotion + 2 routing + 6 warmup = 13 minutes, there are only 2 minutes of margin. A 90-second replica lag fails the RPO even though compute is ready. A requirement for no acknowledged data loss may justify synchronous acknowledgement across failure domains, with extra latency and reduced write availability under a partition; it is not universally a bad choice.

DNS TTL is one part of routing delay. Lowering it cannot revoke existing cached answers; future refreshes can use the new value, while serve-stale behavior, client caches and persistent connections may delay transition further. The alternative region must have reachable dependencies, identity/authorization and tested capacity (or an explicit reduced-service contract), and the failover path must not depend solely on the broken region/control plane. Promotion needs fencing to prevent two active writers. A second region is capability; executed exercises provide evidence of whether that capability meets the targets.

S077 · Three Pillars

1. Define metrics, logs, and traces in one sentence each and give the question class each answers best.
2. Name the four golden signals.
3. Your latency at the high percentile spikes. Which pillar do you consult next, and why?

Answer key — combine signals. Metrics summarize measurements over time (is error rate rising?); logs record discrete contextual events (which validation failed?); traces link spans through a request path (which dependency consumed the time?). Latency, traffic, errors and saturation are the four golden signals. For rising p99, inspect traffic/error/saturation scope, then representative traces by affected route/version/region and correlated logs. A missing trace is not proof a request never failed: sampling, exporter loss or missing instrumentation can hide it.

Logs are not necessarily unsampled or complete; they can be rate-limited, dropped or sampled too. Aggregate user-impact signals are often good paging inputs, but a critical security/audit event can legitimately alert from a log, and trace-derived metrics may be useful with a documented sampling contract. Choose noise, latency and coverage consciously rather than banning all non-metric alerts. Redact or avoid sensitive request/user data; moving high-cardinality identifiers into logs/traces does not remove their privacy, storage or indexing cost. Server-only SLIs miss DNS/edge/connectivity failures before the request arrives, so use suitable edge/client/synthetic observations while accounting for their sampling and attribution limits.

S078 · Metrics Stack

1. Explain the pull-based scrape model and one operability advantage it gives you.
2. Write the query for an error ratio over a recent window, and one for a high percentile from a histogram.
3. Say what the collector component is for.

Answer key — collection and aggregation matter. Prometheus normally discovers targets and scrapes their exposed metrics; the central scraper controls schedule and can record reachability with up. A successful scrape does not prove user-facing requests succeeded, and a failed scrape is missing application data, not automatically zero errors. The OpenTelemetry Collector receives, processes and exports telemetry, allowing batching/filtering/routing and decoupling applications from backend-specific exporters; buffering and delivery are still bounded by its configuration and resources.

A counter accumulates events and can reset on restart, so take rate over each counter series before summing across instances. Divide total error rate by total request rate for a traffic-weighted ratio; averaging per-instance ratios wrongly gives a one-request instance the same weight as a million-request instance. Classic histogram buckets accumulate counts up to each le boundary. Merge compatible bucket rates retaining le before applying histogram_quantile; the estimated quantile depends on bucket layout and interpolation. A native histogram uses a different expression shape and must be treated separately.

Each distinct metric name/label combination is a time series. Cartesian products are upper bounds when not every combination occurs. Active series, churn, scrape frequency, retention, histogram bucket count and backend implementation all affect cost; no universal bytes-per-series constant applies. Use templated route labels such as /orders/{id} rather than full paths, bounded status/method/region labels, and privacy-reviewed exemplars/logs for selected request context. High cardinality is not only one large dimension: several moderate dimensions can multiply badly.

S079 · SLIs, SLOs & Error Budgets

1. Define the four terms and say how they relate.
2. State the allowed monthly downtime for the common availability targets from memory.
3. Explain burn rate and why it beats a plain threshold on the indicator.

Answer key — name numerator, denominator and window. An SLI is a quantitative service indicator such as good eligible requests divided by all eligible requests; an SLO is its target over a specified window; an SLA is an external agreement with stated consequences; the error budget is the allowed bad fraction/count/time associated with the SLO. A 99.9% target allows a 0.1% bad fraction, but that fraction is not automatically downtime unless the SLI is time-based.

For a 30-day time-based availability indicator the window is 43,200 minutes: 99% permits 432 minutes, 99.9% 43.2, 99.95% 21.6, and 99.99% 4.32. For one million eligible requests at 99.9%, the budget is instead 1,000 bad requests. Exclusions and the definition of “good” must be explicit; calling a failed checkout ineligible can make an attractive but useless SLI.

Burn rate is observed bad fraction divided by the allowed bad fraction. At fixed SLO, a burn threshold is algebraically a rescaled raw error threshold—not a new detector with magic discrimination. Its advantage is expressing consumption relative to the objective, enabling budget-based thresholds and multi-window sustained/transient checks. A constant burn of 10 consumes a fresh 30-day budget in 72 hours under comparable traffic/time weighting; variable request volume and a partially spent budget require counts/remaining budget rather than that simple forecast.

The budget supports an agreed release/reliability policy; it is not an instruction to cause failures. Zero consumed budget does not prove overprovisioning or under-shipping. When exhausted, a team's written policy may pause risky releases while permitting urgent fixes; contractual SLAs and safety duties are not waived by remaining budget. Set targets from user needs and cost, and review whether the indicator covers real failures.

S080 · Incident Response

1. Name the three roles and what each owns.
2. Explain why "mitigate first, diagnose later" is the rule.
3. Say what makes a postmortem blameless, and why that matters practically rather than morally.

Answer key — reduce harm under explicit command. The Incident Commander coordinates priorities, decisions and handoffs; Operations executes and records technical mitigations; Communications maintains the shared status and update cadence. These are responsibilities, not a law that every incident has exactly three people: larger incidents can split roles and smaller ones can combine them while keeping ownership explicit. One command channel and an action log prevent two well-intentioned responders from racing incompatible changes.

Mitigation reduces current user impact; root-cause analysis explains the mechanism sufficiently to prevent recurrence. A backward-compatible rollback can restore a broken checkout before anyone knows which code path regressed. Check its preconditions, preserve cheap diagnostic evidence, make one controlled change, and verify user-visible recovery. Some diagnosis is necessary to select a safe action; “mitigate first” never means blindly fail over, destroy forensic evidence, or postpone an urgent security containment decision. The completed runbook below gives permitted and forbidden branches rather than an unconditional list of commands.

A blameless postmortem reconstructs the timeline, information available to responders, contributing system conditions, detection gaps and mitigation results without assigning personal fault. It still records actions and accountable improvement owners. “An operator chose the wrong region” is not the end of analysis: ask why both selectors looked alike, why no precondition blocked the action and why the drill missed it. An actionable result could be: the platform owner adds an environment/region confirmation and a failing-precondition rehearsal by Friday; a peer verifies that the wrong-region fixture is rejected. Avoiding punishment for candid incident reporting makes missing evidence and near misses more likely to surface; an owner, deadline and verification test make the learning operational rather than a promise to be more careful.

Hands-on drill · 30 min

The executable fences calculate exact finite examples and simulate decisions only. PromQL is a source-reviewed query contract, not a live Prometheus evaluation. No rollback, failover or traffic changes are performed.

Step 1 — why averages lie (8 min)

The old title is a warning about using one summary, not a claim the mean is mathematically false. State the percentile convention: this exercise uses nearest rank ceil(p*n) for 0 < p ≤ 1, with p=0 defined as the minimum. Other libraries can interpolate differently on small samples.

import math
import statistics
 
def percentile(values, p):
    if not values or not 0 <= p <= 1 or any(not math.isfinite(x) for x in values):
        raise ValueError('finite nonempty sample and p in [0,1] required')
    ordered = sorted(values)
    return ordered[max(0, math.ceil(p*len(ordered))-1)]
 
a = [1,1,1,1,100]
b = [10]*15
pooled = a+b
assert statistics.mean(a) == 20.8
assert statistics.mean(b) == 10
assert statistics.mean(pooled) == 12.7
assert (statistics.mean(a)+statistics.mean(b))/2 == 15.4
assert (len(a)*statistics.mean(a)+len(b)*statistics.mean(b))/len(pooled) == 12.7
assert percentile(pooled,.95) == 10 and percentile(pooled,.99) == 100
assert (percentile(a,.95)+percentile(b,.95))/2 == 55
print('pooled mean/p95/p99',statistics.mean(pooled),percentile(pooled,.95),percentile(pooled,.99))

Answer: the mean of means fails because sample sizes differ; weighting by sizes repairs means. Neither unweighted nor size-weighted averages of group percentiles generally recover the pooled quantile. Here even weighted p95 would be (5×100+15×10)/20=32.5, not 10. Merge raw samples or compatible histogram/sketch state instead. p95 also misses the one slowest request in this sample, so use a percentile/threshold tied to user needs and examine request counts. One request is not necessarily one distinct user. The original 97% at 50 ms / 3% at 3,000 ms mixture has expected mean 138.5 ms, not “barely any change”; it still cannot describe the distribution alone.

Step 2 — cardinality (7 min)

from itertools import product
from math import prod
label_sizes = {'service':12,'endpoint':40,'method':5,'status':6,'region':3}
upper_bound = prod(label_sizes.values())
assert upper_bound == 43200
assert upper_bound*500000 == 21600000000
observed = {('checkout','/orders','POST','200','east'),
            ('checkout','/orders','POST','500','east'),
            ('checkout','/orders','POST','200','east')}
assert len(observed) == 2
small = list(product(range(2),range(3),range(4)))
assert len(set(small)) == 24
# Hypothetical 11 classic buckets including +Inf, plus sum and count.
assert upper_bound*(11+2) == 561600
print('label upper bound',upper_bound,'observed fixture',len(observed),'classic histogram bound',upper_bound*13)

Answer: the product describes all possible combinations, not guaranteed observed active series. Adding user IDs has a 21.6-billion upper bound here, already enough to reject that schema without pretending every combination occurs. A classic histogram additionally has bucket, sum and count series (optional created series are excluded from this exercise). The former 3,000-byte planning constant was not a universal measured memory cost; measure your implementation and include churn, samples and retention.

Concrete PromQL solution for a fixture contract: http_requests_total is a counter with job, instance and bounded code labels; http_request_duration_seconds_bucket is a classic histogram with compatible le boundaries on every selected instance. These snippets select the same job on numerator and denominator and aggregate away instance.

sum(rate(http_requests_total{job="checkout"}[5m]))

sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
)

A zero denominator or missing series requires an explicit no-traffic/missing-telemetry policy, not a fabricated zero-error ratio. If grouping by service/region, use identical labels on numerator and denominator and keep le for the histogram. The ratio is not a mean of per-instance percentages; the histogram quantile is an estimate, not an exact percentile of retained raw observations. Native histograms use histogram_quantile(0.99, sum(rate(http_request_duration_seconds{job="checkout"}[5m]))) when that metric actually contains native histogram samples.

Step 3 — error budget and burn rate (8 min)

from fractions import Fraction
WINDOW_MINUTES = 30*24*60
objectives = [Fraction(99,100),Fraction(999,1000),Fraction(1999,2000),Fraction(9999,10000)]
allowed = [WINDOW_MINUTES*(1-slo) for slo in objectives]
assert allowed == [432,Fraction(216,5),Fraction(108,5),Fraction(108,25)]
 
def burn_rate(bad, total, slo=Fraction(999,1000)):
    if total <= 0 or not 0 <= bad <= total or not 0 < slo < 1:
        raise ValueError('positive denominator, valid counts and SLO required')
    return Fraction(bad,total)/(1-slo)
assert burn_rate(10,1000) == 10
assert Fraction(30*24,10) == 72
# 14.4x for one hour consumes 2% of a fresh 30-day budget,
# assuming this interval has proportional time/request weight.
assert Fraction(72,5)/720 == Fraction(1,50)
# AND short/long windows: ratios are bad fractions, not percentages.
def page(long_bad,long_total,short_bad,short_total):
    return burn_rate(long_bad,long_total) > Fraction(72,5) and burn_rate(short_bad,short_total) > Fraction(72,5)
assert page(20,1000,20,1000)
assert not page(20,1000,1,1000)
assert not page(1,1000,20,1000)
print('30-day time budgets',list(map(float,allowed)),'minutes; 1% errors = 10x burn')

Alert design: page when both 1-hour and 5-minute burn exceed 14.4× for a 99.9% request SLO, with an explicit eligible-traffic/no-data policy. Both bad-ratio thresholds are 0.0144 (1.44%); the short window reduces stale alerts after recovery, while the long window supplies sustained-impact evidence. This is the Google SRE workbook's budget-based design pattern, not an assertion that these exact thresholds suit all products. A slow-burn ticket can use longer windows and a lower threshold. With very low traffic, use enough evidence, synthetic coverage or a separately defined policy rather than dividing by a made-up denominator.

Step 4 — write one real runbook (7 min)

Completed tabletop example — Checkout Exercise, not a deployed service. Roles for this exercise are Commander, Operator and Communications; the facilitator assigns participants before the drill. Input is a frozen incident packet containing slo-1h, slo-5m, deploy, replication-lag, standby-capacity and request-traces. The names below refer to those packet sections, not invented real URLs. The required production-binding work follows the solution.

1. Confirm within 60 seconds: read both bad/eligible request ratios in slo-1h and slo-5m. If both exceed 1.44%, request denominators are nonzero and telemetry is current, declare Checkout unavailable to a material fraction of users. If only short-window burn is high, investigate a developing event without labelling the long-window criterion met; if data is missing, invoke the telemetry/edge-observation branch instead of closing the incident as healthy.
2. Declare and coordinate: Commander records impact/start time and authorizes one mitigation at a time; Operator records proposed action and preconditions; Communications posts a tabletop status card and sets the next update to 10 minutes, even if there is no progress.
3. Mitigate: if deploy says a release occurred in the last 15 minutes, the previous artifact is available and database/API changes are backward compatible, choose rollback to that recorded artifact. Otherwise inspect request-traces for a reversible feature-flag or dependency degradation path. Do not invent an artifact ID or blindly undo a destructive schema change.
4. Failover branch only if necessary: require a verified healthy standby, at least the defined critical-workload capacity, data lag ≤60 seconds, reachable dependencies/identity, and fencing of the old write authority. If any precondition fails, do not promote; use the previously defined degraded/read-only mode and escalate. Do not repeatedly flip regions while measuring stale health data.
5. Verify or abort: after a simulated action, the next two five-minute observation windows must each show bad ratio ≤0.1%, acceptable user-visible latency and no data-integrity regression. If errors rise, capacity saturates or fencing is uncertain, stop further rollout, preserve the timeline and return to Commander for the next authorized branch; “command returned success” is not recovery.
6. Close only after stable recovery: calculate actual failed-request count against this request-based budget, not invented downtime minutes. Assign the postmortem to Operator with a two-business-day deadline; include detection gaps, mitigation risk and a tracked owner/deadline/test for each improvement.

Worked packet and branch oracle: the first observation has 20/1,000 bad requests in each window, a compatible recent deploy, and standby lag 90 seconds. The page criterion is met; rollback is permitted; failover is forbidden by RPO. Later five-minute observations of 1/1,000 and 0/1,000 satisfy the numerical recovery check if latency and integrity checks also pass. This does not assert an actual service recovered.

packet = {'recent_deploy':True,'compatible':True,'standby_capacity':True,'lag_seconds':90,'fenced':True}
def choose_mitigation(packet):
    if packet['recent_deploy'] and packet['compatible']:
        return 'rollback'
    if packet['standby_capacity'] and packet['lag_seconds'] <= 60 and packet['fenced']:
        return 'failover'
    return 'degrade-and-escalate'
assert choose_mitigation(packet) == 'rollback'
assert choose_mitigation({**packet,'compatible':False}) == 'degrade-and-escalate'
assert choose_mitigation({**packet,'compatible':False,'lag_seconds':30}) == 'failover'
assert all(Fraction(bad,total) <= Fraction(1,1000) for bad,total in [(1,1000),(0,1000)])

Turn it into a real runbook: replace packet section names with verified dashboards/queries, bind named on-call owners, and obtain exact supported rollback/fencing/degraded-mode commands with environment selectors, approval authority and abort steps. A peer should walk it in a nonproduction exercise. Those operational bindings and a real failover are intentionally not asserted here; the completed scenario teaches the decision process while keeping unknown production facts visible. Never run a command merely to make a worksheet look complete.

Recall quiz — distinguish arithmetic from policy

1. Mean or percentile? Use both where useful; means summarize total-per-count, quantiles describe positions, and neither alone covers all users. Pool samples/distributions, not percentile scalars.
2. Unbounded labels? They create many unique series; actual combinations and churn matter. Use bounded labels and privacy-controlled event context instead.
3. Three nines and burn? 43.2 minutes for a 30-day time SLI, or 0.1% of eligible requests for a request SLI. Burn normalizes the bad fraction; windows and budget policy provide the useful alert decisions.
4. Mitigation first? Choose a preconditioned reversible action to reduce user harm, collect cheap evidence, verify, then pursue root cause—not unreviewed failover.
5. Two-region readiness? Demonstrate capacity, lag/RPO, tested RTO/routing, fenced authority, dependencies/identity and a survivable failover control path. Topology alone is not a measurement.

Answer constraints and corrections

The time-based budget is 0.001×30×24×60=43.2 minutes; request-based SLOs use a different denominator. Means remain useful but cannot describe the tail alone. Quantiles cannot generally be averaged; merge raw data or suitable sketches/histograms, with approximation errors stated. Failover capacity and dependency availability must be tested rather than assumed.

Gap analysis + next week preview · 10 min

• Can you distinguish the pooled mean, weighted mean and nonmergeable scalar percentiles, including the chosen nearest-rank convention?
• Can you compute both time and request budgets, and explain why burn is only a rescaling at a fixed SLO?
• Which production runbook bindings or actual failover tests are still absent? The completed tabletop does not fill those operational facts for you.

Mark each answer unaided, hinted or not yet; select a failed assumption and use a new counterexample for the retest rather than memorizing one fixture. Keep the input contract, boundary case and independent expected answer next to any execution result.

Next week: S081–S085 cover authentication/authorization, TLS/PKI, OWASP/security, the ML feature/label/train-validation-test model and linear regression. Reliability and evidence boundaries remain relevant, but the original preview incorrectly listed unrelated design exercises.

Corrective source notes

• Google SRE: Postmortem Culture — blameless analysis investigates contributing system conditions; fear of punishment suppresses reporting. Exact official passage checked 2026-10-09.
• Architecture strategies for disaster recovery - Microsoft Azure Well-Architected Framework | Microsoft Learn — RPO defines acceptable data-loss age and RTO defines time to restore; they are targets, not proof that a topology achieves them. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• RFC 1035 DNS, November 1987 — Previously cached DNS answers cannot be recalled by changing a later authoritative TTL; routing recovery needs a cache-aware design. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Google SRE monitoring ditributed system - sre golden signals — Latency, traffic, errors and saturation are the golden signals; successful and failed request latency should be interpreted separately. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Collector | OpenTelemetry — The Collector receives, processes and exports telemetry; it does not itself make every observation complete or durable. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Query functions | Prometheus — Prometheus recommends rate before aggregation so counter resets are detected per series. Classic histogram aggregation retains le; native histogram aggregation differs, and quantiles involve bucket-dependent interpolation. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Instrumentation | Prometheus — Every unique labelset costs resources; a Cartesian product is a capacity upper bound, not a guaranteed measured active count. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Google SRE - Prometheus Alerting: Turn SLOs into Alerts — Burn rate normalizes error-budget consumption relative to the target; the lesson derives the exact fraction and time/count limits. The SRE workbook combines 14.4x burn over both one hour and five minutes to improve reset behavior. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.
• Google SRE Managing Incidents, 2016 — Incident command, operations and communications are separable responsibilities; controlled mitigation and updates need explicit owners. Retrieved 2026-10-07; exact passage reviewed 2026-10-07.

These passages support the named propositions, not blanket product certification. The lesson-specific derivations and fixtures are separate evidence; runtime boundaries are stated beside each worked exercise.