Incident Response — Runbooks, Postmortems, On-Call

An outage creates two simultaneous jobs: reduce harm and maintain a shared understanding of what is happening. Technical expertise without coordination can produce conflicting changes; coordination without technical evidence can produce confident but wrong updates. This lesson develops a response that keeps both jobs visible, from the first report to verified follow-up work.

Optional preparation

Before the tabletop, distinguish reliability response from security-specific containment and evidence handling. Choose resources to fit the original optional 30-minute preparation budget; the listed video labels do not establish runtime, contents, creator identity, availability or quality.

• Cybersecurity incident detection and response — security-adjacent video pointer, not evidence that it teaches this lesson's IC/Ops/Comms structure.
• What is Google Cloud? — retained broad cloud-orientation pointer. The old annotation called this a blameless-postmortem lesson, which the label does not support; use the primary postmortem chapter below instead.
• Search: incident postmortem conference talks — a discovery search, not a verified Cloudflare, Slack, GitHub or Stripe recording. Compare a selected report's timeline, uncertainty and actions with the template.
• SRE Book: managing incidents — incident-command and coordination preparation.
• SRE Book: postmortem culture — contributing-factor and blameless-review preparation.
• PagerDuty incident-response documentation — optional templates/checklists pointer; adapt to your organization's authority and verified runbook rather than assuming operational readiness.

Roles, authority and incident declaration

The useful part of an emergency-room analogy is role separation, not a claim about medical outcomes. One person coordinates decisions, technical responders carry out authorized work, and another communicates to affected people. In software incidents, Google's incident-management chapter distinguishes incident command, operations and communications; it also discusses planning and a live incident document. IMOC/IMAG terminology varies across organizations. These are responsibilities, not a universal job-title taxonomy.

• Incident commander (IC): owns coordination, priorities, delegation, escalation and handoffs. Maintains the authoritative state of the response and resolves conflicting proposed actions. Avoid deep debugging when it prevents coordination.
• Operations/technical lead: organizes diagnosis and mitigation, assigns owners and confirms observed effects. Familiarity and access matter more than being the most senior person present.
• Communications lead: publishes accurate internal/customer updates, records impact and the next update time, and routes questions away from technical responders. Does not invent a recovery ETA.
• Scribe/planning support: records evidence and decisions, arranges staffing, tracks unresolved work and prepares the next shift.

One person may temporarily hold multiple roles in a small incident; split them explicitly as complexity grows. “IC never touches a keyboard” is a mnemonic for protecting attention, not a literal ban on writing the incident document or a rule that makes solo response impossible. A handoff is an acknowledged transfer of authority, not silently leaving a call. Four-hour rotations and 15-minute updates are examples to negotiate, not universal requirements.

Declare early when impact, uncertainty or coordination needs meet the team's criteria. An SLO burn can trigger declaration, but data corruption, security compromise and credible customer reports can require response before a budget alert fires. A SEV-1/SEV-2 scale belongs to a particular organization; record affected users, regions, duration, safety/security exposure and correctness, not just a fixed 5xx threshold.

If assessment finds no incident, record the reason, observation window and re-escalation condition. “Ignore and move on” loses useful evidence and can conceal an intermittent problem.

Mitigation is not blind rollback

The objective is to reduce user harm under uncertainty. Rollback, feature disablement, capacity relief, rate limiting, failover or queue isolation can help before the full causal story is known. Some diagnosis is usually needed to choose safely. Do not impose a hard wall that forbids learning during mitigation or demands a complete root cause before taking a reversible action.

Compare three options for a suspected deployment regression:

A rollback is a hypothesis-driven intervention, not a proven remedy because a deployment happened nearby. Capture safe, minimal evidence when feasible—version/config identifiers, sanitized errors, affected population and timestamps. Heap dumps can pause processes and contain credentials or personal data; they are not a universally cheap “60-second snapshot.” Security incidents may require specialized evidence retention and containment authority. Never delay urgent safety action solely to collect a perfect trace.

One change, one owner, one expected effect

Before a risky change, record: who authorizes it, who executes, exact environment/resource scope, hypothesis, expected metric change, timeout, abort conditions and rollback/roll-forward plan. Coordinate concurrent actions: two well-intended responders changing routing and pool size can make both hypotheses uninterpretable. Automation needs the same bounds, not an exemption.

Decision at 14:08 UTC (synthetic tabletop)
Observation: checkout errors rose after version v2; DB saturation unchanged.
Hypothesis: payment-client regression in v2; not yet a proven cause.
Authority: IC approves reverting app image after schema compatibility check.
Operator: Ops-A; scope: one canary, then staged fleet rollback.
Expected: canary valid-order success returns to baseline within observation window.
Abort: error/duplicate-charge rate increases, or readiness fails.
Recovery: retain previous routing; stop rollout and escalate to payments owner.
Evidence: deployment IDs, sanitized request IDs, query snapshot and decision log.

This decision record is more useful than “try restarting it.” The operator should refuse a step whose prerequisites or permission scope cannot be established.

The incident timeline and communication cadence

Define timestamps before deriving metrics: impact start, detection, declaration, mitigation, full recovery and resolution. Time-to-detect may start at impact onset; time-to-mitigate might start at detection in one team's dashboard and onset in another. State the definition. Root-cause-understanding time is a separate milestone. Equal mitigation and diagnosis timestamps do not prove avoidable harm: the safe mitigation may have depended on diagnosis, or the timestamps may be coarse.

A useful update names affected functions/populations, observed start time, current status, workaround if verified, and next update time. Separate observed from suspected causes. Do not say “identified” simply because a plausible hypothesis exists. Include UTC or an explicit timezone, avoid sensitive details in public updates, and correct earlier estimates rather than silently rewriting history.

For a synthetic timeline, take impact onset at 14:00 UTC, detection at 14:02, declaration at 14:03, mitigation at 14:10 and complete recovery at 14:30. Detection delay is two minutes; detection-to-mitigation is eight; impact-to-recovery is thirty. Record resolution separately after the agreed stability checks. These are arithmetic labels, not measured team performance or a benchmark. A role handoff at 14:15 must transfer current impact, pending changes, access scope and the next update, with incoming acknowledgement; it does not reset the incident clock.

Mitigated is not necessarily resolved. Suppose rollback returns p99 latency to normal but two hours of payment messages remain. Measure oldest-message age, successful replay, duplicate/idempotency outcomes and customer reconciliation. Resolution requires the declared user-recovery criteria, not just a green process check. Stale caches and delayed notifications can continue harm after the main API recovers.

A runbook a new responder can use safely

A runbook links the alert to a bounded decision tree. It includes symptoms, impact, environment/access prerequisites, diagnostic steps, mitigation, verification, abort/rollback and escalation. Commands should use exact service-owned identifiers, but an educational template must not masquerade as a validated production command. The following is a complete tabletop runbook, with read-only reference commands and explicit gates for writes. No cluster or database command was executed here.

# Runbook: CheckoutErrorBudgetBurn
Owner: checkout on-call; dependency owners listed in service directory
Last verified: record a real drill date and revision before operational use
Scope: designated checkout environment; never infer context from shell prompt
Severity: use incident policy; data integrity/security can override ratio thresholds
 
## Entry and prerequisites
- Read alert population, windows, rule revision and linked dashboard.
- Confirm environment, Kubernetes context, namespace and read-only access.
- Confirm whether the failing operation is read-only, payment creation or retry.
- If scope/access cannot be confirmed: stop commands and escalate.
 
## Symptoms and impact
- Compare valid-order failures, timeouts, latency, traffic and missing telemetry.
- Record affected tenants/regions and whether orders or charges are duplicated.
- No dollar estimate without known counts and an explicit value model.
 
## Read-only diagnosis (reference; replace with tested service-owned values)
1. Confirm context and recent application state:
   ```bash
   kubectl config current-context
   kubectl rollout history deployment/checkout-api -n payments
   kubectl get pods -n payments -l app=checkout-api
   ```
   Compare deployment time with impact onset; correlation alone is not causation.
2. Inspect database concurrency using an already-authorized readonly connection:
   ```sql
   SELECT state, count(*) FROM pg_stat_activity GROUP BY state;
   ```
   Compare with normal limits and transaction ages. A count of 100 is not a
   universal incident threshold. Do not terminate arbitrary transactions.
3. Inspect the approved error dashboard with a bounded time range and tenant-safe
   sampling. Record query/timezone and sanitized evidence; never paste credentials.
 
## Candidate mitigations and gates
A. App rollback: only after backward schema/data compatibility and artifact checks.
   IC names operator, canary size, observation window, abort rule and fallback.
   Use this service's tested deployment procedure; a generic CLI flag is not proof.
B. Dependency failure: contact owner, reduce retries, queue only if durable and
   bounded, expose degradation. Never repair another team's production blindly.
C. Pool saturation: confirm cause and in-flight transaction semantics. Consider
   pausing the offending job or reducing concurrency under owner approval.
   Do not invent an unauthenticated /admin/reset-pool endpoint.
 
## Verification
- User success/latency improved across affected populations, not only one pod.
- No new duplicate/lost charge evidence; backlog age/count converge safely.
- Check monitoring health independently; retain an observation period.
- If worsening, stop the change and use the recorded recovery plan.
 
## Escalation and handoff
- Escalate immediately for integrity/security/scope uncertainty.
- Otherwise use the incident-specific time box; do not wait for a ritual 15 minutes.
- Handoff: named IC/ops/comms, impact, facts, hypotheses, changes, access scope,
  remaining risks, next update and acknowledgements from incoming owners.

The outer four-backtick fence intentionally contains inner code fences. The original three-backtick wrapper could terminate early and turn runbook headings into article headings. Source/AST/rendered fence parity checks catch that kind of silent structural fault.

A SELECT result can be useful evidence but not proof of database health. Likewise, a healthy pod or a /health response does not establish downstream transaction correctness. Runbooks should describe the expected output and the branch it justifies, not “check dashboard; restart database.”

Executed tabletop state machine

The original interactive simulator graded free-text keywords: “do not rollback” could pass a rollback check, and comma-separated text counted as owned actions. It also revealed a root cause at eight minutes then reported a five-minute mitigation. Replace that fake performance scoring with explicit state and authorization checks.

This pure model uses synthetic minutes, no input, sleep, network, commands or customer data. It verifies lifecycle invariants only; choosing an intervention still requires a human's system-specific judgment.

from dataclasses import dataclass, field
 
@dataclass
class Incident:
    state: str = "detected"
    roles: dict = field(default_factory=dict)
    events: list = field(default_factory=list)
    last_time: int = -1
    pending: str | None = None
 
    def log(self, minute, event):
        if minute < self.last_time:
            raise ValueError("time cannot go backwards")
        self.last_time = minute
        self.events.append((minute, event))
 
    def declare(self, minute, roles):
        if self.state != "detected" or not all(roles.get(k) for k in ("IC", "Ops", "Comms")):
            raise ValueError("declare with explicit roles")
        self.log(minute, "declared")
        self.roles = dict(roles)
        self.state = "active"
 
    def authorize(self, minute, actor, action, reversible, scope_confirmed):
        if self.state != "active" or actor != self.roles["IC"] or self.pending:
            raise ValueError("IC authority and one pending change required")
        if not reversible or not scope_confirmed:
            raise ValueError("this exercise requires a reversible scoped change")
        self.log(minute, "authorized: " + action)
        self.pending = action
 
    def observe_result(self, minute, actor, user_recovered):
        if actor != self.roles.get("Ops") or not self.pending:
            raise ValueError("authorized operator/change required")
        self.log(minute, "observed recovery" if user_recovered else "no recovery")
        self.pending = None
        if user_recovered:
            self.state = "mitigated"
 
    def resolve(self, minute, user_stable, backlog_correct, evidence_recorded):
        if self.state != "mitigated" or not all((user_stable, backlog_correct, evidence_recorded)):
            raise ValueError("recovery gates incomplete")
        self.log(minute, "resolved")
        self.state = "resolved"
 
    def review(self, minute, actions):
        if self.state != "resolved" or not actions:
            raise ValueError("review after resolution with follow-up")
        for a in actions:
            if not all(a.get(k) for k in ("mechanism", "owner", "due", "verification")):
                raise ValueError("incomplete action")
        self.log(minute, "reviewed with owned actions")
        self.state = "reviewed"
 
if __name__ == "__main__":
    incident = Incident()
    incident.declare(1, {"IC": "A", "Ops": "B", "Comms": "C"})
    incident.authorize(3, "A", "canary rollback", True, True)
    incident.observe_result(5, "B", True)
    try:
        incident.resolve(6, True, False, True)
    except ValueError:
        pass
    else:
        raise AssertionError("backlog must block resolution")
    incident.resolve(20, True, True, True)
    incident.review(60, [{"mechanism": "test client error handling", "owner": "checkout",
                          "due": "next review", "verification": "regression fixture"}])
    assert incident.state == "reviewed"
    assert [t for t, event in incident.events] == [1, 3, 5, 20, 60]
    print("roles, authority, recovery and follow-up gates: PASS")

The model’s role checks cover authorize and observe_result, not a real identity/access-control system. Its recovery flags are caller-supplied assertions, not observations from a backend. resolve and review have no actor parameter; real incident authority, acknowledged handoff, relapse handling and evidence durability belong in the runbook/workflow. In the demonstrated trace, declaration is minute 1, authorization 3, mitigation 5, resolution 20 and review 60; minute zero detection is a scenario convention, not an event logged by this class. The unresolved backlog deliberately blocks resolution even though the user-facing mitigation improved.

Three tabletop scenarios and the debrief

1. Deployment-associated 500s: compare deployment and error populations; check compatibility; canary a rollback with explicit abort conditions. Expected evidence is recovery, not the word “rollback.” If nothing improves, abandon that hypothesis rather than endlessly repeating it.
2. Database saturation: inspect query/transaction/resource evidence. Pause or cancel a confirmed, authorized offending batch if safe; arbitrary kill/pg_terminate is not the right answer merely because CPU is high. Track impact on transaction correctness and reprocessing.
3. Mail provider outage: validate provider status against local evidence, limit retries, bound durable queues, communicate delays and verify recovery/replay. Waiting alone is not mitigation; a queue can fill, messages can expire and a status page can lag.

Rotate IC/Ops/Comms during drills. Record decision rationale, unknowns, handoff quality and what a new responder could not infer from the runbook. Do not award invented 5/10/15-minute “ideal” outcomes or claim keyword matching measures response competence. Repeat drills at a cadence appropriate to change rate and team risk.

Blameless postmortems with accountable follow-through

Google's postmortem guidance focuses on understanding contributing causes without blaming people for acting with the information available. Blameless does not mean anonymous, consequence-free, or ownerless. A useful review includes:

• Summary and impact: affected operations/populations, duration, severity and uncertainty.
• Evidence-linked timeline: observations, hypotheses, interventions and recovery milestones with timezones.
• Contributing factors: technical design, access/tooling, rollout, detection, communication and recovery conditions. “Human error” starts an investigation; it does not explain why the action was possible or reasonable.
• What helped and what hindered: retain useful signals/runbooks as well as identifying gaps.
• Action items: prevention, detection and recovery changes, each with owner, due date and a verification criterion. Prioritize by risk rather than generating a long ceremonial list.

Five whys is a questioning technique, not proof that a complex failure has a single root cause. A review within five business days is a possible team target; the important part is timely evidence and completed actions. Example action: “Checkout team adds malformed-provider-response fixture by the next release; CI rejects the previously failing build,” rather than “Alice must be more careful.” Naming Alice as the owner is compatible with blamelessness.

First-party incident evidence

Cloudflare, 2 July 2019: its detailed report attributes CPU exhaustion to a globally deployed WAF regular expression with excessive backtracking and describes the response and remediation. Read the report's actual times/impact rather than claiming every edge/server/site behaved identically or that every promised fix shipped within weeks. The mechanism worth carrying forward is that configuration is code with a blast radius: staged rollout, performance testing and a reliable disable path matter.

GitLab, 31 January 2017: its database outage postmortem documents accidental production data removal during replication troubleshooting and problems with backup/recovery paths. The report is internally inconsistent about the lost-data endpoint: its opening says 17:20 to 00:00 UTC (6 hours 40 minutes), while “Data loss impact” says 17:20 to 23:30 (6 hours 10 minutes). Preserve that discrepancy rather than silently selecting an exact loss interval. It separately says copying staging data took around 18 hours, labels the downtime “about 18 hours” in its root-cause analysis, and dates database restoration without webhooks at 17:00 UTC on February 1 and final restoration procedures around 18:00. These are different milestones; neither a six-hour outage nor an exact total derived from copying alone is justified. Restoring a recent staging snapshot was a recovery path, not evidence that backups were healthy. Derive actions from this chain: distinguish environment/role, restrict destructive authority, monitor backup creation and—separately—test restoration. Do not invent a years-old unmonitored spreadsheet or a shipped weekly-restore policy absent from the report.

The original dated GitLab postmortem URL is retained as a historical navigation pointer alongside the canonical source above; redirect availability was not tested.

These reports support mechanisms, not a promise that publishing a postmortem prevents all recurrence. A sanitized public report can improve understanding while respecting privacy and investigation needs.

Sustainable on-call and a 90-day improvement plan

A primary/secondary rotation gives escalation cover; a follow-the-sun arrangement can reduce overnight load but needs timezone coverage, expertise and explicit handoffs. Two-tier L1/L2 response can reduce expert interruptions only if L1 has training, access, tested runbooks and a fast escalation path. None requires exactly 15 engineers or guarantees no night pages. Google's on-call chapter gives its staffing/workload reasoning; its numbers are context-specific.

Track pages requiring action, repeated incidents, false positives, time interrupted, overnight disruption, recovery time, acknowledgement gaps and follow-up toil. Five pages per week is not a universal health threshold; one six-hour critical incident may be worse than several short actionable pages. Adding staff may be necessary, but it does not remove recurring causes or noisy routing.

For a synthetic team with 15 pages per week:

• Weeks 1–2: classify recent pages by actionable impact and recurrence. Group correlated alerts; route nonurgent information to tickets without deleting coverage for a serious failure. Baseline interruptions and affected user outcomes.
• Weeks 3–4: fix the highest-impact recurring causes and improve SLO/missing-data rules. Do not promise a 30–50% reduction without measurement.
• Weeks 5–8: test runbooks with unfamiliar responders; rehearse handoffs. Add bounded auto-remediation only with permissions, idempotency, stop conditions and rollback—not an indiscriminate restart/failover loop.
• Weeks 9–12: compare page actionability, overnight burden and detection/mitigation evidence. Track owned fixes to completion and revise staffing, escalation and compensation/recovery time with the team.

The pager itself can have team-agreed service objectives, but deleting a frequently firing signal solely because it exceeds a quarterly quota can hide an unresolved serious problem. Change the cause or routing with evidence.

Recall and worked answers

1. Three response roles? IC coordinates decisions, Ops performs authorized technical work, Comms updates stakeholders. Add scribe/planning when useful; explicitly acknowledge role changes.
2. Why mitigate before full root cause? User harm accumulates while investigation proceeds. Choose a safe reversible/containing action from enough evidence; diagnosis may still be essential.
3. What is blameless? Examine mechanisms and context rather than personal fault, while retaining named owners and verifiable actions.
4. First five minutes at 3 AM? Acknowledge, read impact/runbook, confirm environment and permissions, declare/escalate as needed, name roles, record facts and execute only authorized bounded steps. Do not wait for a failed fix to declare severe impact.
5. Why is “check dashboard” inadequate? It lacks a link/query, population/window, expected result and decision branch. Even an explicit red panel does not automatically authorize rollback.
6. Three postmortem elements? Evidence-linked timeline, contributing factors and owned/time-bound/testable follow-up; also quantify impact and preserve what worked.
7. Healthy page count? No universal number. Measure actionability, interruption duration, overnight burden, recurrence and staffing; use a local target and protect critical detection.
8. SEV-1 versus SEV-2? Follow pre-agreed impact criteria; the IC applies and revises them as evidence changes. Security/integrity may outweigh error percentage.
9. Follow the sun? Distributed coverage across working hours with expertise and acknowledged handoffs. It reduces, not magically eliminates, overnight escalation.
10. Fix 15 pages/week? Baseline/classify, repair recurring causes, improve routing/runbooks, safely automate bounded actions, and measure both fewer interruptions and preserved detection over the 90-day plan.

Finish by writing one runbook, walking it with a colleague, recording every unsupported assumption and replacing “be careful” with one testable prevention or recovery mechanism.

Prerequisites and study route

Bring observability and SLO concepts, deployment/recovery basics and your organization's access policy. Response consumes telemetry and may use failover or feature flags; drills test those mechanisms, while security incidents can require specialized authority and evidence handling.

Preparation links: S077, three pillars, S078, Prometheus/Grafana/OTel and S079, SLOs supply observations, queries and contracts. S076, multi-region supplies failover design questions, not permission to trigger one. The original chaos-engineering study topic supplies controlled-drill questions with stop/recovery conditions; no matching standalone destination was verified, and the old “S092” now names evaluation metrics. The original “S128 System design reviews” label likewise differs from the current chat-design title; take the monitoring/response plan to system-design framework, S126. These links restore the study route, not new technical reviews of their destinations.

Next: S081 · AuthN vs AuthZ, Sessions & Password Storage — the security track adds authorization and credential-handling concerns to operational response.

Hub: The 6-Month Learning Plan. This retains the original series hub, not a new endorsement of every linked lesson.

Study pacing: this is S080 in the original 130-session series. Use intuition → visual model → bounded practice → production decisions → recall as the learning route. The original 90-minute session and 5/15/25/15/10-minute phase labels are flexible planning cues, not a verified completion time (the phase labels total 70 minutes). Leave time for preparation and discussion; service deployment remains a separate authorized exercise.

Sources and verification boundary

The earlier primary snapshots were captured on 2026-10-08. This body and its governing passages were re-reviewed on 2026-10-10 with exact path/hash matching of inherited snapshots and fresh pinned-source additions where needed. Raw/canonical extraction, quotation offsets, original coverage and bounded fixtures are retained in the batch-42 packet. Historical pins describe their own versions, not current deployment recommendations.

The exact state machine and negative-case oracle were executed. Runbook commands are reference/tabletop material, not production interventions; no cluster or database service was contacted.

• Google SRE - Incident Management: Key to Restore Operations — Google SRE Book, 2016.
• Google SRE - Blameless Postmortem for System Resilience — Google SRE Book, 2016.
• Google SRE - On Call Engineer Best Practices for IT Services — Google SRE Book, 2016.
• Details of the Cloudflare outage on July 2, 2019 | Cloudflare Blog — Cloudflare report on 2019-07-02 outage.
• Postmortem of database outage of January 31 — GitLab postmortem published 2017-02-10.