Observability & SRE — Metrics, Traces, SLOs
A deep-dive on Observability & SRE — part of a 36-topic evergreen learning series.
Why this session matters
Part of a 36-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.
Session 19 · SYS track 📈
Metrics, logs, traces — the three pillars. This session covers what to instrument, how to instrument it (OpenTelemetry, Prometheus, Grafana), how SLOs work (SLI → SLO → error budget → alerting), and what an on-call incident actually looks like. Without observability, you're flying blind in production.
Pre-read (30 min before session)
Watch 1–2 of these before the deep-dive:
- SRE Fundamentals — Google SRE Book Playlist
- OpenTelemetry Explained — Honeycomb
- SLO/SLI/SLA — Liz Fong-Jones
Then skim chapter 4 of the Google SRE Book.
Deep-dive (90 min)
1. The three pillars (20 min)
Metrics (aggregated, cheap, time-series) — Prometheus/OpenTelemetry. Logs (high-cardinality, expensive, text) — Loki/ELK. Traces (per-request causal graph, sampled) — Jaeger/Tempo. Correlation IDs to tie them together. When to reach for each.
2. Instrumentation (25 min)
OpenTelemetry as the vendor-neutral standard (metrics + traces + logs API). Auto-instrumentation vs manual spans. Metric types: counter, gauge, histogram (percentiles vs averages). Cardinality budgets (why user_id in labels kills Prometheus). Structured logging (JSON, key-value).
3. SLOs & error budgets (25 min)
SLI (measured), SLO (target), SLA (contractual). Availability SLOs (success/total requests over rolling window). Latency SLOs (p99 < X). Error budget = 1 − SLO. Burn-rate alerts (fast burn = page, slow burn = ticket). Multi-window multi-burn-rate alerting (Google SRE workbook).
4. Incident response (20 min)
On-call ergonomics: runbooks, dashboards, escalation. First 5 min of an incident (declare, page, comm channel, scribe). Blameless postmortems. Five-why root cause. Action items with owners + dates.
Reading list
- Site Reliability Engineering (Google SRE Book, free online)
- The Site Reliability Workbook (SRE Workbook, free)
- Charity Majors — Observability Engineering
Research links
Hands-on drill
Add OpenTelemetry to a small Flask or FastAPI app. Emit a counter for requests, a histogram for latency, and a trace across 2-3 spans (handler → db → external call). Pipe to a local Grafana + Prometheus + Tempo stack via docker-compose. Set an SLO of p99 < 200ms and write the Prometheus rule that alerts on burn rate.
Post-session checklist
- Can you explain the difference between SLI, SLO, and SLA in 60 seconds?
- Can you explain burn-rate alerting in 60 seconds?
- Can you explain why high-cardinality labels break Prometheus in 60 seconds?
- Did you complete the hands-on drill above?
- Did you write 3 flashcards for tomorrow's recall?
- What's the one thing you'd want to revisit in the next revision session?
What's next
Session 20 continues the series. See the hub page for the full sequence and revision pattern.