S077 · The 3 Pillars — Metrics, Logs, Traces

Module M09: Observability & SRE · Session 77 of 130 · Track: SYS 🔭

What you'll be able to do after this session

• Explain The 3 Pillars to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S060 — OS Basics — Processes, Threads, Memory, FDs

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Observability vs Monitoring (Honeycomb / Charity Majors excerpt) — 10-minute crisp distinction from the person who coined "observability" in this sense.
• 🎥 Deep dive (30–60 min) · The Three Pillars of Observability (Cindy Sridharan, DevOps Days) — the canonical talk that popularised the framing.
• 🎥 Hands-on demo (10–20 min) · OpenTelemetry in 15 Minutes (TechWorld with Nana) — one CLI, one demo app, all three signals emitted.
• 📖 Canonical article · OpenTelemetry — Observability Primer — the canonical vendor-neutral primer.
• 📖 Book chapter / tutorial · Google SRE Book — Chapter 6: Monitoring Distributed Systems — the four golden signals in the authors' own words.
• 📖 Engineering blog · Honeycomb — What is Observability? — the modern "observability != monitoring" argument in one page.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The senses of a running system.

Imagine you're a doctor. Your patient walks in feeling terrible. You could look at their vital signs (pulse 120, temp 39°C, BP 140/90 — like metrics: numeric, aggregated over time, cheap). You could read their medical history journal (dates, symptoms, medications — like logs: rich event records, one per line). Or you could order a full-body scan that shows exactly which organs are affected and how signals propagate (like distributed traces: end-to-end request paths across systems).

You need all three, and they answer different questions. Metrics answer "is it broken now, and how badly?" Logs answer "what exactly happened in this specific event?" Traces answer "which service in a chain of 15 microservices caused the 3-second latency?"

Observability exists because modern systems are too complex to debug by SSH-ing into a box and reading log files. When a request touches 20 microservices across 3 regions, a broken payment can only be diagnosed if you can reconstruct the specific request's path, specific failure, and system state at that instant. The "three pillars" — metrics, logs, traces — are the industry's convergent answer.

Gotcha to carry forever: the three pillars are not independent — they are connected. A useful setup lets you jump from a metric spike ("error rate up") to the logs from that time window, then to a specific slow trace, then to the exact log lines from that trace. Tools that let you pivot (Honeycomb, Datadog, Grafana + Tempo/Loki/Prometheus stack) beat tools that silo the three.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — the pivoted debugging flow.

You get paged at 3 AM: SLO breach: p99 latency > 500 ms for /checkout.

Comparison of the three pillars:

Sampling matters. You cannot afford to store a trace for every request in a system doing 100 k QPS. Two strategies: head-based sampling (decide at the edge, e.g., keep 1%) and tail-based sampling (collect all, keep the interesting ones — errors, slow requests). Modern OpenTelemetry pipelines support both.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Instrument a tiny Flask app with all three pillars using OpenTelemetry auto-instrumentation.

# app.py — pip install flask opentelemetry-distro opentelemetry-exporter-otlp
#          then: opentelemetry-bootstrap --action=install
from flask import Flask, jsonify
import logging, random, time, os
from opentelemetry import trace, metrics
 
app = Flask(__name__)
tracer = trace.get_tracer(__name__)
meter  = metrics.get_meter(__name__)
req_counter = meter.create_counter("checkout_requests_total")
err_counter = meter.create_counter("checkout_errors_total")
lat_hist    = meter.create_histogram("checkout_latency_seconds")
 
logging.basicConfig(level=logging.INFO,
    format='%(asctime)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s')
log = logging.getLogger()
 
@app.route('/checkout')
def checkout():
    start = time.perf_counter()
    req_counter.add(1)
    with tracer.start_as_current_span("checkout") as span:
        span.set_attribute("user.id", random.randint(1, 100))
        with tracer.start_as_current_span("fetch_cart"):
            time.sleep(random.uniform(0.01, 0.05))
        with tracer.start_as_current_span("charge_card"):
            time.sleep(random.uniform(0.05, 0.30))
            if random.random() < 0.05:      # 5% failure
                err_counter.add(1)
                span.record_exception(RuntimeError("card declined"))
                span.set_status(trace.Status(trace.StatusCode.ERROR))
                log.warning("card declined for checkout")
                return jsonify(error="declined"), 402
        with tracer.start_as_current_span("send_receipt"):
            time.sleep(random.uniform(0.005, 0.02))
    lat_hist.record(time.perf_counter() - start)
    log.info("checkout ok")
    return jsonify(status="ok")
 
if __name__ == "__main__":
    app.run(port=8000)

Run with the OTel auto-instrumentation wrapper, pointing at a local collector:

# 1. Local OpenTelemetry Collector (writes to stdout for demo)
cat > otel.yaml <<'EOF'
receivers:
  otlp: { protocols: { grpc: {}, http: {} } }
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces:  { receivers: [otlp], exporters: [debug] }
    metrics: { receivers: [otlp], exporters: [debug] }
    logs:    { receivers: [otlp], exporters: [debug] }
EOF
docker run -d --name otelcol -p 4317:4317 -p 4318:4318 \
  -v $PWD/otel.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector:latest --config /etc/otelcol/config.yaml
 
# 2. Run instrumented app
export OTEL_SERVICE_NAME=checkout-svc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
opentelemetry-instrument python app.py &
sleep 3
 
# 3. Generate traffic
for i in $(seq 1 50); do curl -s http://localhost:8000/checkout > /dev/null; done
 
# 4. Watch collector output
docker logs -f otelcol

Observe (checklist):

1. Traces show a root span checkout with children fetch_cart, charge_card, send_receipt — you can see which sub-call is slow.
2. Metrics show checkout_requests_total, checkout_errors_total, and a histogram for latency.
3. Log lines include trace_id=... and span_id=... — this is what lets you jump from a metric to logs to a trace.
4. Failed requests are marked with Status=ERROR in the trace; you can filter for them in a real tracing UI (Jaeger, Tempo).
5. Kill the collector and restart the app — the app keeps running, dropping telemetry rather than blocking. This "fail-open" is critical.

Try this modification: point the collector at a real Grafana Cloud (free tier) or Honeycomb (free tier) account and build a dashboard showing p99 latency, error rate, and a sampled slow-trace panel.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Observability is where "we thought we had monitoring" meets "we can't debug at 3 AM."

The pre-observability world (2015-ish): Nagios pinging /health, syslog files scp'd nightly, one Grafana dashboard with 30 graphs. It works until your first cascading multi-service outage — and then it does not, and RCA takes days.

Where teams end up:

• Metrics: Prometheus (self-hosted) or Datadog / Grafana Cloud (managed) with alerts wired to PagerDuty. Cardinality kept low ({route, status, region} — never {user_id, order_id}).
• Logs: structured JSON, single sink (Loki, ELK, Splunk, Datadog Logs). Always include trace_id and service fields. Never log full request bodies — you will accidentally log PII or credit-card numbers, and be very sad.
• Traces: OpenTelemetry SDK + a backend (Tempo, Jaeger, Honeycomb, Lightstep). Sample aggressively at scale (1% + all errors + all slow requests).

Common bugs and gotchas:

• Cardinality explosion. You add a user_id label to a Prometheus metric; now you have 10M time series and Prometheus OOMs. Rule: labels are for aggregations, not identifiers. Put identifiers in traces or logs.
• Metrics-only alerting. Alerts fire but you have no way to drill down. Every metric alert should link to a runbook that includes a trace/log query.
• Log-only debugging. Grepping GBs of logs at 3 AM. Add metrics for the top failure modes and traces for latency; save logs for the last-mile detail.
• Trace sampling that drops the interesting stuff. Head-based 1% sampling misses your one broken customer's request. Tail-based sampling (via OTel Collector) keeps every error trace + a fraction of successful ones.
• Cost. Datadog bills are notorious. New Relic, Splunk, Datadog can each cost as much as your compute. Observe your observability bill.

Charity Majors' insight: monitoring answers "is this thing I already knew about broken?"; observability answers "what is this new thing I've never seen before?" Modern production always has both known-unknowns (SLO alerts) and unknown-unknowns (novel bugs), so you need both.

(e) Quiz + exercise · 10 min

1. In one sentence each, what question does each of metrics, logs, and traces best answer?
2. Why do you need trace_ids in your log lines?
3. What is "cardinality explosion" in metrics, and give one concrete example.
4. Explain the difference between head-based and tail-based sampling.
5. If you had to pick only one of the three pillars to add first to a system with none, which and why?

Stretch (connects to S060 · SLIs/SLOs foundation): Design the four "golden signals" (latency, traffic, errors, saturation) for a hypothetical /login endpoint that fronts a Postgres DB and calls an external SSO. Which pillar do you use to measure each, and what specific metric name / trace attribute / log field would you use?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is The 3 Pillars? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S078): Prometheus, Grafana, OpenTelemetry — Hands-on
• Previous (S076): Multi-Region — Active-Passive, Active-Active, Failover
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.