S078 · Prometheus, Grafana, OpenTelemetry — Hands-on

Module M09: Observability & SRE · Session 78 of 130 · Track: SYS 📊

What you'll be able to do after this session

• Explain Prometheus, Grafana, OpenTelemetry to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S077 — The 3 Pillars — Metrics, Logs, Traces

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Prometheus Explained in 100 Seconds + Deep Cut (Fireship / TechWorld) — quick tour of what Prometheus is and why it dominates.
• 🎥 Deep dive (30–60 min) · The Prometheus + Grafana Stack (TechWorld with Nana) — 60-minute crash course; install, scrape targets, PromQL, dashboards, alerts.
• 🎥 Hands-on demo (10–20 min) · OpenTelemetry Collector — Getting Started (OpenTelemetry) — 15 min, ship traces + metrics from a demo app in one config file.
• 📖 Canonical article · Prometheus Docs — Overview — the canonical entry point.
• 📖 Book chapter / tutorial · Grafana Labs — PromQL Getting Started — the best beginner PromQL tutorial.
• 📖 Engineering blog · SoundCloud — Prometheus: Monitoring at SoundCloud — the origin story from the team that built it.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The default stack.

Prometheus + Grafana + OpenTelemetry is the "default open-source observability stack" of 2020s software. Practically every SRE job description mentions all three; every cloud platform (EKS, GKE, AKS) ships with them integrated. Learning this stack is like learning Git — it's not the only option, but it's the lingua franca.

Think of the three like this:

• Prometheus is the warehouse — it scrapes numeric metrics from your services on a schedule (say every 15 s), stores them as time series, and answers questions via PromQL. It's pull-based, self-hosted, and famously good at reliability.
• Grafana is the shop window — it queries Prometheus (and Loki for logs, Tempo for traces) and turns the data into dashboards, alerts, and drill-down views. It's the "single pane of glass" everyone talks about.
• OpenTelemetry (OTel) is the shipping standard — one vendor-neutral SDK + protocol for emitting metrics, logs, and traces. Before OTel, every backend had its own SDK; you were locked in. OTel means you can swap Datadog for Prometheus for Honeycomb without touching app code.

The typical modern pipeline: app → OTel SDK → OTel Collector → (Prometheus for metrics, Loki for logs, Tempo for traces) → Grafana.

Gotcha to carry forever: Prometheus is pull-based and not for high-cardinality data. If you find yourself putting user_id or request_id as a metric label, stop — you'll blow up Prometheus. Those go into logs or traces. Metrics are for low-cardinality, aggregate signals.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — the four metric types.

Anatomy of a Prometheus scrape:

GET /metrics HTTP/1.1

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/api/users",status="200"} 12345
http_requests_total{method="GET",route="/api/users",status="500"} 42

# HELP http_request_duration_seconds Request duration
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/api/users",le="0.1"} 12000
http_request_duration_seconds_bucket{route="/api/users",le="0.5"} 12340
http_request_duration_seconds_bucket{route="/api/users",le="1.0"} 12345
http_request_duration_seconds_bucket{route="/api/users",le="+Inf"} 12345
http_request_duration_seconds_sum{route="/api/users"} 987.6
http_request_duration_seconds_count{route="/api/users"} 12345

Every 15 seconds Prometheus fetches this endpoint from every target, parses it, and stores the deltas. That's it. No agents installed on the target, no complex push protocol.

PromQL cheat sheet — the queries you'll write 90% of the time:

# QPS by route
sum by (route) (rate(http_requests_total[5m]))
 
# Error rate as fraction
sum by (route) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by (route) (rate(http_requests_total[5m]))
 
# p99 latency
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
 
# Alert: p99 > 500ms for 10 min
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{route="/checkout"}[5m])))
  > 0.5

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Full local stack: Prometheus + Grafana + a Python app exposing metrics, plus an alert.

# docker-compose.yml
services:
  app:
    image: python:3.12-slim
    working_dir: /app
    volumes: ["./:/app"]
    command: bash -c "pip install flask prometheus-client && python app.py"
    ports: ["8000:8000"]
 
  prometheus:
    image: prom/prometheus:latest
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]
    ports: ["9090:9090"]
 
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment: [GF_AUTH_ANONYMOUS_ENABLED=true, GF_AUTH_ANONYMOUS_ORG_ROLE=Admin]

# prometheus.yml
global: { scrape_interval: 5s }
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['app:8000']

# app.py
from flask import Flask
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import random, time
 
app = Flask(__name__)
REQ  = Counter('http_requests_total', 'Total requests', ['route', 'status'])
LAT  = Histogram('http_request_duration_seconds', 'Latency', ['route'],
                 buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5])
 
@app.route('/api/users')
def users():
    with LAT.labels(route='/api/users').time():
        time.sleep(random.uniform(0.01, 0.4))
        status = '500' if random.random() < 0.1 else '200'
        REQ.labels(route='/api/users', status=status).inc()
        return ("oops", 500) if status == '500' else "ok"
 
@app.route('/metrics')
def metrics():
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}
 
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8000)

docker compose up -d
sleep 10
# Generate traffic for a minute
for i in $(seq 1 500); do curl -s http://localhost:8000/api/users >/dev/null; done

Observe (checklist):

1. Open http://localhost:9090/targets — the demo-app job is UP.
2. In Prometheus UI, run rate(http_requests_total[1m]) — you see a per-status QPS graph.
3. Run histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[1m]))) — you see live p99 latency.
4. In Grafana (http://localhost:3000), add Prometheus datasource (http://prometheus:9090), then create a dashboard with the two queries above.
5. Try sum by (status) (rate(http_requests_total[1m])) — you can see error rate climb because of the 10% failure injection.

Try this modification: add an Alertmanager rule for p99 > 300ms for 2 minutes. In prometheus.yml add:

rule_files: ["alerts.yml"]

and a alerts.yml with a PrometheusRule — then confirm the alert enters the firing state under load.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Prometheus + Grafana is battle-tested by hundreds of thousands of companies. Even so, some patterns bite everyone.

Kubernetes shipping. Almost every Kubernetes cluster runs the "kube-prometheus-stack" Helm chart, which bundles Prometheus + Alertmanager + Grafana + node exporter + kube-state-metrics. Learn this one chart and you can operate the monitoring of 90% of K8s clusters you'll ever see.

Real war stories:

• The cardinality bomb. A team added user_id as a label to http_requests_total. Prometheus memory grew from 4 GB to 40 GB overnight. Fix: drop the label at scrape time with metric_relabel_configs.
• Federation vs remote_write. Single Prometheus doesn't scale past ~1M active series. Solutions: Thanos, Cortex, Mimir (Grafana), or Prometheus's own remote_write to a scalable backend. Pick early; migrations are painful.
• Alertmanager silence-storms. After an incident, someone silences an alert for 24h to stop the pages — and forgets. The next real incident is invisible. Fix: enforce max silence duration in policy.
• Grafana dashboard sprawl. 500 dashboards, none of them maintained. Instill "dashboards as code" (Jsonnet, Grafonnet, or Terraform) and treat dashboards like any other artifact.
• PromQL learning cliff. Team writes queries that look right but silently wrong (rate vs irate, forgetting sum by, _sum / _count for averages instead of histogram_quantile). Invest in a shared PromQL cookbook.
• Managed vs self-hosted. Grafana Cloud, AWS Managed Prometheus, Google Managed Prometheus, and Chronosphere all remove ops burden. If you're small, use managed. If you're huge and your bill exceeds SRE salaries, run your own with Mimir.

On OpenTelemetry specifically:

• OTel Collector is a stateful piece — deploy it as a DaemonSet on Kubernetes with a gateway tier for aggregation and sampling.
• OTel logs are still less mature than metrics/traces; many teams still ship logs via Fluentbit/Vector directly to Loki.
• Auto-instrumentation covers ~80% of common libraries (Flask, Django, requests, SQLAlchemy). The last 20% is manual tracer.start_as_current_span(...).

Golden operational rules: alert on symptoms not causes (alert on error rate, not CPU); every alert must link to a runbook; rehearse alert playbooks at least quarterly.

(e) Quiz + exercise · 10 min

1. What is the difference between a Prometheus counter, gauge, and histogram? Give an example use of each.
2. Why is Prometheus pull-based rather than push-based?
3. What is "cardinality" and why does it matter for Prometheus memory usage?
4. Write a PromQL query for the error rate (as a fraction) of /checkout requests over the last 5 minutes.
5. What role does the OpenTelemetry Collector play, and why is it usually deployed separately from the app?

Stretch (connects to S077 · Three Pillars): Your team wants to correlate a Prometheus alert with a specific trace. Sketch the tag/label conventions you'd standardise (service, environment, trace_id linking) so that clicking a spike in Grafana lands you on a specific trace in Tempo.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Prometheus, Grafana, OpenTelemetry? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S079): SLIs, SLOs & Error Budgets — the SRE Math
• Previous (S077): The 3 Pillars — Metrics, Logs, Traces
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.