S079 · SLIs, SLOs & Error Budgets — the SRE Math

Module M09: Observability & SRE · Session 79 of 130 · Track: SYS 🎯

What you'll be able to do after this session

• Explain SLIs, SLOs & Error Budgets to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S077 — The 3 Pillars — Metrics, Logs, Traces

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · SRE Fundamentals: SLIs, SLOs, SLAs (Google Cloud Tech) — 8-minute animated explainer from the SRE team.
• 🎥 Deep dive (30–60 min) · Implementing SLOs (Google SRE — Alex Hidalgo, SREcon) — a full talk on how to design them without lying.
• 🎥 Hands-on demo (10–20 min) · Building SLOs with Prometheus & Grafana (Grafana Labs) — 15-minute practical walkthrough with real PromQL.
• 📖 Canonical article · Google SRE Workbook — Implementing SLOs (Chapter 2) — the authoritative reference.
• 📖 Book chapter / tutorial · Google SRE Book — Service Level Objectives — the original chapter that started it all.
• 📖 Engineering blog · The Google Cloud Blog — SRE Error Budget Policy — how policy backs the math.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How to measure reliability without lying.

Imagine you promise your friend "I'll always answer your call within 2 rings." Sounds nice — until she calls at 3 AM during your vacation and you miss it, and now you're a liar. SLIs, SLOs, and error budgets are the engineer's honest answer: "I'll answer 99% of your calls within 2 rings, measured over a month. If I miss more than 1% — that's my 'error budget' — we'll talk about why."

Reliability is not a boolean. Nothing is 100% available — not Google, not AWS, not the phone system. The relevant question is: what fraction of failure is acceptable to your users, and how do you measure it? SLOs give you the vocabulary, the math, and — crucially — the decision framework for when to slow down new features and shore up reliability.

Three terms in order of increasing seriousness:

• SLI (Indicator): the measurement — e.g., "fraction of requests to /checkout that returned in < 500 ms with a non-5xx status."
• SLO (Objective): the target — e.g., "99.9% of /checkout requests over any 30-day window."
• SLA (Agreement): the contract — e.g., "if we breach 99.5%, we refund 10% of your monthly bill." SLAs live in commercial contracts; SLOs are internal engineering targets set below the SLA to give you slack.

Error budget = 100% − SLO. If your SLO is 99.9% over 30 days, your error budget is 0.1% × 30 days = 43 minutes of downtime per month. Every minute of user-visible failure eats the budget; when it runs out, feature velocity slows and reliability work takes priority. This mechanism aligns engineering incentives — devs want to ship, SREs want stability, error budgets make the tradeoff explicit.

Gotcha to carry forever: an SLO is only useful if breaching it changes behaviour. A 99.99% SLO that everyone ignores when it fails is worse than a 99% SLO that triggers a real reliability sprint. The math is easy; the org discipline is the hard part.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — the 99.9% availability month.

• SLO: 99.9% of requests succeed over 30 days.
• Total requests in 30 days: 100 M.
• Allowed failures: 100 M × 0.1% = 100 000 failed requests.
• That's your error budget. Every failure spends a coin.
• If the first 3 days ate 60 000 of them (perhaps due to a bad deploy), you've burned 60% of your monthly budget in 10% of the time — this is a burn-rate alert.

Burn-rate alerting (Google's multi-window pattern):

A 14.4× burn rate means you're spending your monthly budget 14.4× faster than expected. Ignore it and it's gone in ~2 hours.

How to pick an SLO number (the honest way):

1. Ask users, not engineers. What downtime actually loses customers?
2. Look at your current 90-day baseline (awk-together your logs). Set the SLO at or slightly below current performance — an SLO you're already violating is not credible.
3. Higher isn't better. 99.99% costs ~10× more than 99.9% — every extra nine buys less reliability at more cost. Only add a nine if the business case pays for it.
4. Different SLOs per endpoint. /checkout might be 99.99%; /marketing-blog might be 99%. Don't force a single number across everything.

Availability translated to real time:

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Compute an SLI + SLO from real Prometheus data, then simulate an error-budget burn alert.

# Add these Prometheus recording + alerting rules to prometheus.yml
groups:
  - name: slo.rules
    interval: 30s
    rules:
      # SLI: fraction of good checkout requests (2xx or 3xx AND < 500ms)
      - record: sli:checkout_good:ratio_rate5m
        expr: |
          (
            sum(rate(http_request_duration_seconds_bucket{route="/api/users",le="0.5",status!~"5.."}[5m]))
          )
          /
          sum(rate(http_requests_total{route="/api/users"}[5m]))
 
      # 30d rolling error budget consumed
      - record: slo:checkout_error_budget:remaining_30d
        expr: |
          (
            (1 - 0.999)               # 0.1% allowed error
            -
            (1 - avg_over_time(sli:checkout_good:ratio_rate5m[30d]))
          )
          / (1 - 0.999)
 
  - name: slo.alerts
    rules:
      - alert: CheckoutFastBurn
        expr: |
          (
            1 - sli:checkout_good:ratio_rate5m
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Checkout SLO burning 14.4x faster than budget (2h to exhaust)"
          runbook: "https://runbooks.internal/checkout-slo-burn"
 
      - alert: CheckoutSlowBurn
        expr: |
          (
            1 - avg_over_time(sli:checkout_good:ratio_rate5m[1h])
          ) > (6 * (1 - 0.999))
        for: 15m
        labels: { severity: ticket }

Then a small script to simulate a burn and inspect the results:

# burn_sim.py — needs the app from S078 running behind Prometheus
# pip install requests
import requests, threading, time, random
 
BASE = "http://localhost:8000"
 
def normal_traffic():
    while True:
        try: requests.get(f"{BASE}/api/users", timeout=1)
        except: pass
        time.sleep(0.01)
 
def broken_deploy(duration_s=180):
    # Simulate a bad deploy: inject bursts of 500s and slow calls.
    end = time.time() + duration_s
    while time.time() < end:
        # In the demo app, the failure rate is baked in — imagine a real
        # deploy hitting `/api/users` with a poisoned payload
        try: requests.get(f"{BASE}/api/users?slow=1", timeout=5)
        except: pass
 
threading.Thread(target=normal_traffic, daemon=True).start()
print("Normal traffic running. Ctrl-C to stop.")
print("Simulating a 3-minute broken deploy in 10s...")
time.sleep(10)
broken_deploy(180)
print("Broken window done.")
time.sleep(3600)

Observe (checklist):

1. In Prometheus, sli:checkout_good:ratio_rate5m sits near 0.9 (matches the 10% error injection in the S078 app).
2. Compute your monthly error budget: with 99.9% SLO you're allowed 0.1% errors — you're at 100× the budget! Your SLO is unrealistic for this app.
3. Adjust SLO to 90% (1 - 0.9) and re-inspect — now the ratio makes sense.
4. Trigger the fast-burn alert by ramping up failure rate temporarily (edit app.py to make failures 30% for a minute).
5. Alertmanager shows the alert firing; a 2-minute window with 14.4× burn = pageworthy.

Try this modification: add a latency SLI (99% of requests under 250 ms) alongside the availability SLI. Now your service has two SLOs. Should you page on either? On both? This is the multi-SLO decision every real team wrestles with.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

SLOs are the tool that separates "we have monitoring" from "we have SRE." They also create some of the most political conversations in engineering.

How Google actually uses them (from the SRE book):

• Every service has an SLO owner (usually a tech lead + SRE).
• Error budget policy documents exactly what happens when the budget is burned: freeze non-critical launches, prioritise reliability work, escalate to leadership if 2 consecutive quarters miss.
• Reviewed quarterly with product management — SLOs are as much a product decision as an engineering one.

Real war stories:

• The vanity SLO. A team sets 99.99% on everything. First real incident, they miss the SLO for the month, everyone shrugs, no policy triggers. The SLO is theatre. Fix: policy first, number second.
• The measurement gap. Your SLI measures from inside your service, but users experience failures at the CDN edge (TLS handshake failures, DNS errors). Real SLIs should measure as close to the user as possible — synthetic probes or RUM.
• Overly narrow SLI. "Availability" = "at least one request succeeded per minute." A single-request-per-minute service passes even during a 90%-error outage. Aggregate over enough traffic.
• Ignoring the burn during firefighting. During an incident, teams focus on the immediate problem. Burn-rate alerts help make sure the reliability posture is also managed — sometimes you need to freeze deploys for a week even after the fire is out.
• SLA vs SLO confusion. Someone sets an internal SLO at 99.99% because the customer SLA says 99.99%. Now you have zero slack — the first minor breach violates the customer contract. Internal SLOs should be tighter than external SLAs.

Cross-team dynamics.

• Dev teams like to ship; SRE likes reliability. Error budgets convert this into a shared, quantitative conversation instead of a value judgement.
• When budget is unspent, encourage risk: chaos experiments, dependency upgrades, canary launches. Unused error budget is wasted engineering flexibility.
• When budget is overspent, both dev and SRE work on reliability. This is the whole point.

Practical starter kit: pick one critical endpoint. Compute its actual current success rate over 90 days. Set your first SLO 0.5% below that number. Write a one-page error-budget policy. Review in 90 days. Iterate. That's it.

(e) Quiz + exercise · 10 min

1. Define SLI, SLO, and SLA in one sentence each, in your own words.
2. What is an "error budget" and how do you compute it from an SLO?
3. Why is 99.99% ~10× more expensive than 99.9%?
4. What is "burn rate" and why do teams alert on multiple burn-rate windows?
5. Give one concrete example of an SLI that is narrower or broader than user experience — and why it would mislead you.

Stretch (connects to S077 · Three Pillars): Design an SLI for a distributed system using trace data instead of request logs. Concretely: how would you define "good request" for a /checkout that fans out to 5 microservices, using OpenTelemetry span attributes?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is SLIs, SLOs & Error Budgets? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S080): Incident Response — Runbooks, Postmortems, On-Call
• Previous (S078): Prometheus, Grafana, OpenTelemetry — Hands-on
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.