SLIs, SLOs & Error Budgets — the SRE Math

“The dashboard is green” is not a reliability contract. A useful SLO says whose experience counts, what qualifies as success, how observations are collected, the target and the evaluation window. This lesson derives the budget, implements a deterministic rolling-window model and turns it into an explicit operating policy.

Optional preparation

Choose a vocabulary introduction before deriving budgets, then compare the workbook's examples with your own eligible-event contract. The original “30-minute pre-read” is a selectable preparation budget, not a verified total runtime. Video labels below are historical finding aids; content, attribution, duration and availability have not been verified by watching.

• SLI, SLO, SLA in 5 min — optional definitions pointer, not a timing guarantee.
• Class SRE Implements SLOs — optional framework walkthrough pointer; not evidence for the arithmetic.
• Search: Liz Fong-Jones practical SLOs — a search, not an identified recording or verified creator attribution.
• SRE Book: service-level objectives — preparation for measurement, target and agreement distinctions.
• SRE Workbook: implementing SLOs — worked-ratio reading; retain population and window when adapting examples.
• Honeycomb: alerting on SLOs — optional operational perspective on noise and responsiveness; the primary workbook, not this unreviewed pointer, governs the burn-rate recipe here.

Define the population before choosing the nines

An SLI is a quantitative indicator of a service property. This lesson uses good-event ratios, though not every conceivable SLI is a ratio. An SLO is a target for that indicator over a stated window. An SLA is an agreement with consequences and its own measurement rules; it is not necessarily a financial-credit contract or always numerically below the internal SLO. An error budget is the permitted failure implied by the objective. Google's 2016 SLO chapter distinguishes the terms; the 2018 workbook develops practical ratio-based SLIs.

A scorecard analogy is useful: choose observations and thresholds, then agree what a breach changes. But do not import a restaurant's invented legal inspection thresholds or treat safety-critical harm as a spendable software-error allowance. Product, security, legal and engineering constraints can override a release-budget policy.

Write an SLI specification before the query:

Counting all non-5xx responses as good is a possible proxy, not a universal business definition. A 200 response with the wrong amount is bad; a rate-limited eligible request may be bad for this contract; a valid “card declined” can be a correct business result. Client retries require an explicit choice between attempts and logical operations. Do not quietly change the denominator during an incident.

Availability, latency, freshness and correctness

• Availability: good operations / eligible operations, with the success definition above.
• Latency: eligible operations completed within a threshold / eligible operations. An individual request does not have a p95; the population does. Counting “requests whose p95 is below 300 ms” is a category error. Decide whether failures/timeouts count as slow bad events too.
• Freshness: reads whose returned data age is within a threshold / eligible reads, or scheduled jobs completed before a deadline / expected jobs. Declare the source timestamp and timezone.
• Correctness: verified correct results / evaluated results. Reference quality, sampling and delayed adjudication affect this SLI; it is not a claim that unevaluated outputs are correct.

A nightly batch can use “ready by 06:00 UTC on 99% of scheduled days”; over 365 scheduled days the allowance is 3.65 failures, so three is within target and four is not. A queue can use “99% of accepted messages processed within 60 seconds,” with duplicate processing, permanent failure and pending messages explicitly accounted for. Queue age and backlog size are useful diagnostic gauges but need an event/time contract to become an SLO.

Derive budgets with units intact

Let eligible events be $E$, good events $G$, bad events $B=E-G$, and objective $S$. For $E>0$:

$R$ is the fraction of budget remaining. It can be negative: clipping it to zero hides how far over budget the service is. With no eligible events, the ratio is unknown, not 100% success. A 100% objective has zero error budget; a burn-rate division by that budget is undefined, so use separate zero-tolerance rules rather than silently returning a reassuring number.

For one million requests at 99.9%, the budget is 1,000 bad requests. If 1,200 fail, remaining budget is −20%. By contrast, a time-based objective over a 30-day month has 43,200 minutes. Its permitted bad time is (1-S) × 43200.

These are exact arithmetic assumptions about 30/365-day windows, not claims that each target dictates a particular architecture. They are not request-based outage allowances unless the event/time weighting is compatible. If half a month's requests arrive in one busy hour and all fail, the request SLI loses 50 percentage points; the wall-clock fraction is only 1/720. Partial failures, changing traffic and lost telemetry make casual conversions particularly misleading.

Rolling windows are not monthly refills

In a calendar window, accounting starts again at the boundary under the contract. In a rolling window, old events age out continuously; new successes also change the denominator. A bad hour can keep affecting the ratio until it exits. “Budget remaining 15% means freeze” is not inherent in the math: compare remaining time, current burn, unresolved risks and the agreed policy. No one should generate failures merely to use up an allowance. A persistently unused budget is a reason to reassess target/cost/user value, not proof of over-provisioning or under-shipping.

Burn rate and multiwindow alerts

The burn rate is observed bad fraction divided by allowed bad fraction:

At a 99.9% SLO, 1% errors means $r=10$. With a constant compatible event rate, consuming a fraction $f$ of a window's budget over a duration $w$ in a total period $T$ gives:

For 30 days, $T=720$ hours: 14.4 × 1/720 = 2%; 6 × 6/720 = 5%, not 10%; 1 × 72/720 = 10%. A burn rate of 14.4 would consume a full nominal budget in 50 hours if sustained, not a prediction about an arbitrarily changing future population.

The SRE workbook alerting chapter recommends starting points, not universal thresholds:

Require both windows in a pair to cross the threshold. The long window demands material sustained impact; the short window checks whether the problem is continuing and clears sooner after recovery. An AND does not inherit only the short window's detection latency: both conditions must become true. Short threshold alerts can be valid for imminent destructive conditions; multiwindow alerts are not a theorem that every other alert is bad. A long for on a short ratio can miss repeated spikes that keep resetting it, while a full-month average can hide a fresh outage.

PromQL reference with named assumptions

Assume recording rules produce per-job bad-event ratios for each listed window, using the same eligible population and error taxonomy. The expression below is the workbook's structure for a 99.9% objective. Recording rules must preserve identical matching labels across windows; mixing service populations can produce wrong or absent matches.

(
  job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
)
or
(
  job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
  and
  job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
)

For example, a rate5m recording rule for a service that explicitly treats 5xx as its bad proxy is:

groups:
  - name: checkout-sli-reference
    rules:
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          (sum by (job) (rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
           or 0 * sum by (job) (rate(http_requests_total{job="checkout"}[5m])))
          / sum by (job) (rate(http_requests_total{job="checkout"}[5m]))

In Prometheus 2.53 operator semantics, these comparisons filter samples; and intersects label sets and or unions them. Do not add bool to the comparisons: false values would remain as zero-valued vector elements, and set intersection is not a test that their values are true. The strict > also means equality alone does not trigger. A missing member of a window pair prevents that pair from producing an alert; it does not demonstrate health. If both page pairs match the same job, or yields one job element, not two independent pages from this expression. Keep severity in the alert labels, not mismatched recording-rule dimensions.

Define the 1h, 6h, 30m and 3d variants by changing both record suffix and window; validate them with controlled failures before paging. Zero denominator remains undefined, and missing series must get a separate instrumentation/discovery signal. A long-window budget query can use counter increases and the same denominator, but reset/extrapolation, source gaps and retention affect the result; use a durable event accounting pipeline when exact contractual counts are required. These rules were source/syntax-reviewed, not run on Prometheus here.

Measurement location and sparse traffic

Server metrics are usually inexpensive and attributable, but only see requests that arrive and are instrumented. Synthetic journeys continue at a controlled rate during low traffic and cover an external path, yet do not represent every real workflow. Client telemetry measures render/load/retry experience but can be sampled, blocked or unable to report during an outage. Reconcile these views instead of declaring one universally complete. One failing request in a ten-request window has a high ratio; decide whether to use a longer window, minimum sample rule, synthetic corroboration or a different escalation. A minimum traffic condition must not suppress a genuine outage that prevents requests from reaching the server.

Deterministic rolling-window simulator

The original simulator mixed real-time incident scheduling with simulated event time, retained millions of individual events and advertised multiwindow alerts while checking only one window per alert. This replacement stores exact minute populations. It models a rolling period with complete synthetic history and explicit unknown-data handling; it is not a production monitoring engine.

Call the simulator with integer minute evaluation times, a reusable sequence of valid observations and a retention period at least 4,320 minutes when using its fixed alert windows. alerts() is an instantaneous policy result, not an implementation of Prometheus evaluation cadence, pending/for state, Alertmanager routing or deduplication. Custom shorter periods can support smaller ratio() queries but not the fixed alert suite.

Each tuple is (end_minute, eligible, bad) for the half-open minute interval ending at that timestamp. Windows select now-width < end <= now; timestamps increase. A missing or zero-traffic minute makes this deliberately conservative model's ratio unknown. The policy differs from a production extrapolating counter query, and that difference is part of the exercise.

from collections import deque
from fractions import Fraction
 
class SLOTracker:
    def __init__(self, target=Fraction(999, 1000), period=30*24*60):
        if not 0 < target < 1 or period < 1:
            raise ValueError("positive period and 0 < target < 1 required")
        self.target, self.period = target, period
        self.minutes = deque()
        self.last = None
 
    def observe(self, end, total, bad):
        if not isinstance(end, int) or self.last is not None and end <= self.last:
            raise ValueError("increasing integer timestamps required")
        if not isinstance(total, int) or not isinstance(bad, int) or not 0 <= bad <= total:
            raise ValueError("invalid counts")
        self.last = end
        self.minutes.append((end, total, bad))
        while self.minutes and self.minutes[0][0] <= end-self.period:
            self.minutes.popleft()
 
    def ratio(self, now, width):
        if not isinstance(width, int) or not 0 < width <= self.period:
            raise ValueError("window outside retained period")
        rows = [(t, n, b) for t, n, b in self.minutes if now-width < t <= now]
        if len(rows) != width or any(n == 0 for t, n, b in rows):
            return None
        return Fraction(sum(b for t, n, b in rows), sum(n for t, n, b in rows))
 
    def burn(self, now, width):
        ratio = self.ratio(now, width)
        return None if ratio is None else ratio/(1-self.target)
 
    def remaining(self, now):
        burn = self.burn(now, self.period)
        return None if burn is None else 1-burn
 
    def alerts(self, now):
        out = []
        for label, long, short, threshold in [
            ("PAGE-fast", 60, 5, Fraction(72, 5)),
            ("PAGE-sustained", 360, 30, Fraction(6)),
            ("TICKET", 4320, 360, Fraction(1)),
        ]:
            a, b = self.burn(now, long), self.burn(now, short)
            if a is not None and b is not None and a > threshold and b > threshold:
                out.append(label)
        return out
 
def simulate():
    tracker = SLOTracker()
    # Complete 30-day history: 6000 requests/minute, 3 bad = 0.05%.
    for minute in range(1, 43201):
        tracker.observe(minute, 6000, 3)
    assert tracker.remaining(43200) == Fraction(1, 2)
    for minute in range(43201, 43231):
        tracker.observe(minute, 6000, 300)  # exactly 5% for 30 minutes
    assert "PAGE-fast" in tracker.alerts(43230)
    # 30*(300-3) additional failures replace baseline minutes.
    assert tracker.remaining(43230) == Fraction(149, 320)
    for minute in range(43231, 43236):
        tracker.observe(minute, 6000, 3)
    assert "PAGE-fast" not in tracker.alerts(43235)
    assert tracker.burn(43235, 60) > Fraction(72, 5)  # long window still high
    assert tracker.ratio(43236, 5) is None            # absent newest minute
    print("baseline remaining=50%; after incident=46.5625%; short-window reset: PASS")
    return tracker
 
if __name__ == "__main__":
    simulate()

This 30-minute incident adds 8,910 failures above baseline. At 259,200,000 monthly requests, the 99.9% budget is 259,200 events. The additional consumption is 3.4375% of budget, leaving 46.5625% from the 50% baseline. At 99.99%, even the baseline 0.05% error rate burns at 5×; there was no healthy initial budget. Do not claim the same short incident alone necessarily consumes an entire 30-day budget. The exact model tests include the boundary timestamp zero, gaps, weighted traffic, negative remaining values, short/long conjunction and invalid inputs; no sleep or live traffic is needed.

A policy people can actually follow

An error budget is a decision input, not the only permitted argument. Google's example policy names owners, consequences and exceptions. A product-specific policy should distinguish security fixes, reliability mitigations, planned experiments and ordinary features, and say who can approve exceptions.

Service: checkout (illustrative; targets require stakeholder agreement)
Window: rolling 30 days
Availability: correct eligible attempts / eligible attempts >= 99.9%
Latency: eligible attempts completed within 300 ms / eligible attempts >= 99%
Measurement owner: checkout service team; data-gap owner: telemetry team
 
Normal budget: staged rollout with rollback and monitoring remains mandatory.
Rapid burn or unresolved recurrence: pause risky changes, investigate, assign owners.
Exhausted budget: prioritize reliability; only approved mitigation/security changes.
Exception: named incident authority records risk, expiry and follow-up.
Recovery: restore user SLIs and backlog correctness; review before lifting restrictions.
Review: assess targets, exclusions and policy quarterly and after major product changes.

One may choose 50%/10% bands, but they are policy assumptions rather than proof that “ship anything” is safe. A 99% service can be appropriate for a particular customer-facing feature; 99.99% does not mechanically require multi-region. Each extra nine has product and engineering costs, but no defensible universal 3–10× infrastructure multiplier. A 100% observed outcome can occur; promising universal fault-free service is different. Some safety/security invariants legitimately demand zero tolerated violations even when a general availability SLO does not.

Worked product exercise and failure analysis

For a food-delivery design, propose—not assert as company practice—order placement 99.9% correct success and 99% under 500 ms; driver-location reads 99% successful, with 95% returning a position no more than three seconds old by the declared device timestamp; search 99% successful, 99% completed under 200 ms, and 99% of eligible reads reflecting menu updates within five minutes; notification delivery 99.5% within 30 seconds of acceptance. For each, name the eligible event and where “delivered” is measured. Provider acceptance is not device delivery; clock error affects position freshness. Prioritize pages by actionable user impact and burn, not a blanket rule that only checkout matters.

A synthetic “reliability theatre” scenario excludes scheduled outages and upstream failures until the SLO is green while users cannot order. Correct it by documenting exclusions, adding ingress/client evidence and including dependencies the user cannot distinguish. A chaos experiment is justified by a hypothesis, stop conditions and recovery plan; budget availability does not make a destructive experiment safe. The original unsourced Netflix “stability week” story is not needed to teach this mechanism and is not repeated as history.

Optional original production pointers: Google’s SLO chapter provides the measurement/policy context; Netflix’s AWS-outage lessons is a separate resilience-reading pointer, not corroboration of the removed “stability week” or an assertion of Netflix’s error-budget policy. No chaos experiment was run.

If 90% of the budget disappears in five days, assess current burn and the agreed policy, pause risky launches where required, preserve measurement evidence and prioritize contributing factors. Do not silently lower the SLO to make the chart green or assume the calendar will refill a rolling budget tomorrow.

Recall and worked answers

1. SLI/SLO/SLA/budget? Measurement, target plus window, agreement with consequences, and permitted bad outcomes in the declared units. Their numeric ordering is not guaranteed.
2. Is 100% always wrong? No universal slogan applies. Distinguish availability tradeoffs from zero-tolerance safety constraints, observed results from guarantees, and costs from invented multipliers.
3. 99.9% monthly downtime? 43.2 minutes for a time-based 30-day objective; one million eligible requests instead allow 1,000 bad events.
4. Burn rate? Bad fraction divided by 1-S; 10× means 1% bad at a 99.9% target. Relate it to budget consumption only under compatible weighting assumptions.
5. Multiwindow recipe? 14.4 over 1h AND 5m, or 6 over 6h AND 30m for the workbook's page starting points; 1 over 3d AND 6h for a ticket. Six over six hours consumes 5%, not 10%.
6. Five-nines SLA request? Show 25.92 seconds per 30 days for time-based accounting, then clarify user harm, exclusions, measurement, consequences, failure domains and feasible operations before accepting a contract.
7. Nightly batch SLI? Scheduled runs meeting a deadline divided by expected runs; add correctness/freshness requirements. A 99% annual objective allows three failures out of 365, not four.
8. 90% budget spent early? Follow the named policy and escalation, contain rapid burn, investigate and gate risky releases; do not rewrite the target mid-incident.
9. Why server-side blind spots? DNS/TLS/network/client failures and requests never reaching the service may be absent. Pair complementary measurements and track missingness.
10. Food-delivery SLOs? Use the four separate event contracts above; verify event timestamps, delivery semantics and user-priority decisions rather than copying arbitrary percentages.

Explain the target in one sentence with units and window, derive a burn threshold, then change traffic weighting and show which conclusion no longer follows.

Prerequisites and study route

Bring ratios, time windows and basic metric queries. Instrumentation supplies observations; this lesson gives them a user-facing contract, and incident response applies the resulting decisions. Multi-region and rate limiting are design options to assess against that contract, not consequences of a particular number of nines.

Preparation links: S077, three pillars supplies measurement concepts; S078, Prometheus/Grafana/OTel supplies the query/reference-stack mechanics. Continue to S080, incident response: burn is one possible declaration input, not the entire definition of an incident. Compare S076, multi-region and rate limiting and token buckets against the user contract; the number of nines alone dictates neither. Carry population/window and recovery requirements into system-design framework, S126.

The original “S089 Rate limiting” and “S128 System design reviews” labels are legacy orientation, not the current titles at those numbers. The topic links above avoid sending readers to decision trees or chat design by mistake.

Next: S080 · Incident Response — Runbooks, Postmortems, On-Call.

Hub: The 6-Month Learning Plan. This retains the original series hub, not a new endorsement of every linked lesson.

Study pacing: this is S079 in the original 130-session series. Use intuition → visual model → bounded practice → production decisions → recall as the learning route. The original 90-minute session and 5/15/25/15/10-minute phase labels are flexible planning cues, not a verified completion time (the phase labels total 70 minutes). Leave time for preparation and discussion; service deployment remains a separate authorized exercise.

Sources and verification boundary

The earlier primary snapshots were captured on 2026-10-08. This body and its governing passages were re-reviewed on 2026-10-10 with exact path/hash matching of inherited snapshots and fresh pinned-source additions where needed. Raw/canonical extraction, quotation offsets, original coverage and bounded fixtures are retained in the batch-42 packet. Historical pins describe their own versions, not current deployment recommendations.

The exact rolling-window simulator and independent arithmetic/window oracles were executed. PromQL is a source-reviewed reference; no Prometheus backend or real service traffic was used.

• Google SRE - Continuous Improvement To Get Reliability — Google SRE Workbook, 2018.
• Google SRE - Defining slo: service level objective meaning — Google SRE Book, 2016.
• Google SRE - Prometheus Alerting: Turn SLOs into Alerts — Google SRE Workbook, 2018.
• Google SRE - Error Budget Policy for Service Reliability — Example Error Budget Policy, 2018-02-19.