S080 · Incident Response — Runbooks, Postmortems, On-Call

Module M09: Observability & SRE · Session 80 of 130 · Track: SYS 🚨

What you'll be able to do after this session

• Explain Incident Response to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S079 — SLIs, SLOs & Error Budgets — the SRE Math

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Blameless Postmortems (Google Cloud Tech / SRE) — 10-minute intro to the culture that makes incident response actually work.
• 🎥 Deep dive (30–60 min) · Incident Response at Scale (Netflix — SREcon talk) — 45 min on how Netflix runs incident command for a global service.
• 🎥 Hands-on demo (10–20 min) · PagerDuty Incident Response — 20 min Walkthrough (PagerDuty) — see the tooling side end-to-end.
• 📖 Canonical article · Google SRE Book — Chapter 14: Managing Incidents — the canonical structure (Incident Commander, Ops Lead, Comms Lead).
• 📖 Book chapter / tutorial · PagerDuty Incident Response Documentation — open-source, comprehensive, tactical.
• 📖 Engineering blog · Cloudflare — Full Postmortem on Cloudflare 2019 Outage — the gold-standard public postmortem.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Turning outages into learning.

3:14 AM. Your phone screams. Checkout is down. Every second is money and reputation burning. What now? If the answer is "panic and Slack everyone" you don't have an incident response process — and you'll waste the first 20 minutes just figuring out who's in charge.

Incident response is the muscle memory that turns a chaotic 3 AM outage into a choreographed one. It has three parts: during the incident (roles, comms, mitigation), after (postmortem to learn), and before (runbooks and drills to make the next one easier). Companies that do this well recover from big outages in minutes; companies that don't take days and repeat the same mistakes.

The core insight: incidents are systems problems, not people problems. An engineer typing the wrong command that took down the site did so because the system allowed it. The response and postmortem must be blameless — otherwise people hide mistakes, and you learn nothing. This one cultural shift is what enables the rest.

Roles during an incident (Google's ICS-inspired model):

• Incident Commander (IC): coordinates, delegates, does not fix things themselves. Their job is decisions, not debugging.
• Ops Lead: leads the technical investigation. Runs commands, forms hypotheses.
• Comms Lead: talks to customers, execs, status page, Twitter. Frees the IC and Ops Lead to work.
• Scribe (bigger incidents): keeps the timeline in real time — invaluable for the postmortem.

Gotcha to carry forever: the first person to notice is the IC by default, until they hand off. Explicit handoff is required: "I'm handing over IC to Priya at 03:22 UTC — confirm?" Without this, two people quietly assume responsibility and neither actually leads.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Severity levels — what "SEV1" actually means.

A worked timeline — the anatomy of a well-run 45-minute incident.

The blameless postmortem template (5 sections):

1. Summary — 3 sentences: what happened, impact, root cause.
2. Timeline — with UTC timestamps, actor, evidence.
3. Root cause & contributing factors — plural, always. There's rarely one cause.
4. What went well / what didn't — celebrate the good, learn from the bad.
5. Action items — SMART (specific, measurable, assigned, realistic, time-bound). Owner, due date, ticket.

Anti-pattern to avoid: "Fred pushed a bad deploy" — this teaches nothing. Correct framing: "Our deploy process allowed a schema migration to reach production without dry-run validation." Now you have an actionable system fix.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Build a lightweight, real, working incident-response toolkit: alerts wired to a "channel", a runbook, and a postmortem template.

#!/usr/bin/env bash
# runbook-checkout.sh — copy-pasteable ops steps
# save at runbooks/checkout.sh in your repo
set -euo pipefail
 
echo "=== Checkout runbook ==="
echo "1. Confirm the alert (Grafana dashboard)"
echo "   → https://grafana.internal/d/checkout"
echo
echo "2. Check recent deploys"
kubectl -n prod rollout history deployment/checkout | tail -5
echo
echo "3. Check dependent services"
for svc in payment-service inventory user-db-primary; do
  echo -n "  $svc: "
  kubectl -n prod get pods -l app=$svc -o jsonpath='{.items[*].status.phase}' \
    || echo unreachable
  echo
done
echo
echo "4. Common mitigations (pick one, get IC approval):"
echo "   a) kubectl -n prod rollout undo deployment/checkout"
echo "   b) kubectl -n prod scale deployment/checkout --replicas=20"
echo "   c) Toggle feature flag: LaunchDarkly checkout.v2 → off"
echo
echo "5. Verify:"
echo "   curl -sf https://api.example.com/checkout/health && echo OK"
echo
echo "6. After mitigation: comment in #inc-XXX, update status page, schedule PM."

<!-- postmortem-template.md — commit at postmortems/YYYY-MM-DD-title.md -->
# Postmortem: <Short Title>
 
- **Date of incident:** 2026-07-21
- **Duration:** 03:14 - 03:35 UTC (21 min)
- **Severity:** SEV1
- **Impact:** ~4 200 failed checkouts, ~$18 000 lost revenue, 0 data loss
- **Author:** Ava (IC)
- **Reviewed by:** SRE lead, Backend lead
 
## 1. Summary
At 03:14 UTC the checkout endpoint began returning 500 errors for all users. The
root cause was a schema migration in deploy abc123 that dropped a column still
in use by an older running pod. Rollback restored service within 21 minutes.
 
## 2. Timeline
| Time (UTC) | Event |
| --- | --- |
| 03:11 | Deploy abc123 completed |
| 03:14 | Alertmanager fires checkout-p99>500ms |
| 03:16 | IC declared, incident channel created |
| 03:24 | Root cause identified: dropped column `merchants.legacy_flag` |
| 03:25 | Rollback initiated |
| 03:35 | Metrics back to baseline; incident closed |
 
## 3. Root cause and contributing factors
- **Primary:** Migration dropped a column still referenced by rolling-update pods.
- **Contributing:** No staging replay of migrations against production-shape data.
- **Contributing:** Feature flag for new schema path was default-on.
 
## 4. What went well / what didn't
Went well:
- Alert paged within 3 min of impact.
- Runbook was up to date; rollback command copy-pasted successfully.
- Clear IC/Ops/Comms handoff.
 
Didn't:
- Comms Lead joined 6 min later than target.
- No status page update between "investigating" and "resolved."
 
## 5. Action items
| # | Action | Owner | Due |
| --- | --- | --- | --- |
| 1 | Add pre-deploy migration replay against prod-schema staging | @ben | 2026-08-04 |
| 2 | Alert on comms-lead-not-joined within 5 min | @cara | 2026-07-28 |
| 3 | Backfill: audit last 90 days of migrations for drop-column risk | @sre-team | 2026-08-11 |

Observe (checklist):

1. Store runbooks/*.sh in the same repo as the code — they get reviewed and versioned.
2. When you write an alert, include the runbook URL in its annotations — every page has a link.
3. Postmortem docs live in the repo (or Notion) with a template like the above; blameless framing is mandatory.
4. Every action item has an owner and a due date; assign them a Jira/Linear ticket immediately.
5. Review action items 30 days later — are they done? If not, why?

Try this modification: run a game day. Pick a Friday, ask an SRE to break something non-critical (kill a pod, saturate a DB, block egress to a dependency). Time how long detection, IC assignment, mitigation, and status-page update take. Publish the game-day report like a real postmortem — the improvements you find are typically higher-value than any code change.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

"Response quality is a leading indicator of company maturity." — SRE folk wisdom.

What top teams do (Google, Netflix, Stripe, Cloudflare):

• On-call is compensated. Being paged is unpleasant work; pay for it (bonus, comp time, priority for time off). Otherwise you get burnout, gaming of alerts, and eventually resignations.
• Follow-the-sun rotations. For global services, hand off between geographies so nobody is on-call at their local 3 AM every week.
• Runbooks are code. Version-controlled, code-reviewed, updated in every postmortem. A stale runbook at 3 AM is worse than none.
• Alerts have runbooks. No page without a link. If you can't write the runbook, the alert isn't ready.
• Public postmortems. Cloudflare, GitLab, GitHub publish detailed postmortems on major outages. It's marketing, sure — but it also enforces internal discipline.
• Game days & Chaos Engineering. Netflix's Chaos Monkey randomly kills pods in production. Amazon has "GameDay Fridays." You cannot expect to be good at incident response without practice.

Common pitfalls:

• Blame culture. One outage where an engineer got yelled at → everyone hides small mistakes → next big outage nobody saw coming. Blameless postmortems are non-negotiable.
• Endless action items, nothing shipped. A postmortem with 30 action items is a wishlist. Prioritise 3 that ship in 2 weeks.
• Same incident repeatedly. If you're having a similar SEV1 quarterly, your action items aren't fixing the class of problem, only the instance.
• IC role skipped. In a small team, everyone jumps into debugging. Nobody coordinates. Explicitly appoint an IC even if it's a 2-person incident.
• Comms silence. Customers hate uncertainty more than they hate outages. Update status page at least every 30 min during a SEV1 even if there's "nothing new" — say so explicitly.
• PagerDuty fatigue. Too many non-actionable alerts train the on-call to ignore alerts. Ruthlessly delete/tune alerts. Every alert should be actionable right now.

The 2-in-the-morning test. Would a tired, groggy engineer with 6 minutes of context be able to (a) understand this alert, (b) find the runbook, (c) take the right first action? If not, fix the alert, not the engineer.

Metrics for incident response itself: MTTA (mean time to acknowledge), MTTR (mean time to recovery), MTBF (mean time between failures), % of alerts that are actionable, % of action items completed on time. Track them. If they don't improve over 6 months, your process isn't working.

(e) Quiz + exercise · 10 min

1. Why is "blameless" the essential adjective for a postmortem culture?
2. What are the four canonical roles in a well-run incident response, and what does each do?
3. Give one reason a status page update without new information is still valuable during an incident.
4. What is a "game day" and why do mature teams schedule them?
5. Your alert has fired 40 times in the last week and nobody has ever taken action. What should you do?

Stretch (connects to S079 · SLOs & Error Budgets): After an incident that burned 60% of your monthly error budget in 4 hours, how should the team's plans for the next 3 weeks change? Reference the error-budget policy vocabulary from S079.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Incident Response? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S081): AuthN vs AuthZ, Sessions & Password Storage
• Previous (S079): SLIs, SLOs & Error Budgets — the SRE Math
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.