The 3 Pillars — Metrics, Logs, Traces

Why this session exists

A latency alert, a trace and a log can describe the same request at different levels. Their value is the question they answer and the context that survives between them—not the presence of three vendor dashboards. This lesson defines the signals, derives cardinality and sampling costs, instruments a bounded checkout fixture, and provides a source-reviewed OpenTelemetry/Flask reference. No collector, exporter endpoint, application server or paid monitoring service is started.

Prerequisites

Bring HTTP routes/statuses, function-call nesting, counters and basic percentiles. Distributed context matters when work crosses processes, threads, queues and asynchronous tasks. Keep credentials and personal payloads out of teaching telemetry. The executed model uses synthetic IDs only.

Optional preparation and pacing

The original session suggested a 90-minute study block: preparation, intuition, visual reasoning, a local exercise, production failure analysis and recall. Treat all durations as planning cues, not measured completion times; pause and revisit the worked answers as needed. Optional resources are not prerequisites for running the offline examples. Videos have not been watched or timed here; their availability, creator attribution, contents and quality remain unverified. Search links are searches, not selected talks.

• Top 7 Most-Used Distributed System Patterns — Optional distributed-patterns pointer; the old claim that it is a three-pillars cartoon was not verified.
• OpenTelemetry Course - Understand Software Performance — Optional OpenTelemetry course pointer; API/SDK and vendor-neutral instrumentation are suggested topics, not a claim that one call emits every signal.
• Charity Majors on Observability 2.0 — Search for Charity Majors observability and wide-event discussions, not an identified recording or verified creator.
• Google SRE Book · Chapter 6 (Monitoring) — Primary monitoring reference for the four golden signals and alert design; technical claims are bound to saved passages.
• OpenTelemetry · What is OTel — Optional OTel overview entry point; APIs, SDKs and exporters have separate roles, not one automatic all-signals call.
• Observability: A Manifesto — Attributed wide-event viewpoint, not proof of an invented customer story or a measured debugging-speed improvement.

Intuition: vital signs, chart and procedure timeline

A hospital analogy separates aggregate measurements, event records and a procedure's timeline. Metrics can reveal a population trend; logs can record a specific event; traces can associate spans with a causal request path. The analogy does not mean logs always explain root cause or traces capture every operation. Missing instrumentation, sampling, drops and clock issues can leave gaps.

A service needs enough evidence to answer its reliability/security questions, not necessarily three separately purchased products. Structured events can produce aggregate metrics, and sufficiently complete context-bearing events can reconstruct request paths. Ordinary counters plus unrelated text logs cannot reconstruct causality that was never recorded.

Precise definitions

• Metrics: numeric measurements/aggregations over time with dimensions. Counters, gauges and distributions support trends, alerting and capacity questions. Pre-aggregation cannot later recover a discarded dimension.
• Logs/events: records of occurrences with timestamps and attributes. Structured fields make filtering/joining possible, but volume, indexing and retention still cost resources. A log is not automatically a legally adequate audit trail.
• Traces: spans connected by trace context, parent relationships and links. They expose the path and timing of recorded work across boundaries. Concurrent spans mean a trace is not just a list whose durations should all be summed.

OpenTelemetry collects, processes and exports telemetry. It is not the backend that stores/query-displays all signals, and one API call does not automatically emit a metric, log and trace. APIs, SDKs, instrumentation libraries, exporters and collector pipelines have separate responsibilities and maturity/version contracts.

Historical and wide-event context

Google's 2010 Dapper report describes low-overhead distributed tracing at scale; it is not a claim that Google invented every tracing idea. OpenTelemetry unifies vendor-neutral instrumentation concepts across signals. The Honeycomb manifesto supplies an attributed argument for richer structured events and exploratory questions; it is not evidence for the old invented customer story or a universal “ten seconds rather than days” improvement.

A wide event records many useful dimensions for an operation—route, deployment, region, outcome and approved correlation IDs—so new groupings can be queried later. It can reduce siloed joins and derive aggregates, at the cost of ingestion, indexes, scan/query work and privacy exposure. It does not provide infinite drill-down, remove all cardinality costs or recover unsampled fields. Specialized metric/trace representations and wide-event storage are compatible design choices.

The signals side by side and the investigation path

Suppose a hypothetical p99 spike starts at 14:03. A metric can identify affected route/region/deployment. An exemplar or known trace ID can lead to a slow request. That trace shows time waiting for a connection pool; its correlated log records the timeout category and queue observation. This suggests pool contention, but root cause may be a slow downstream, undersized pool or leaked connections. Correlation narrows hypotheses; it is not causal proof or a guarantee of a four-minute investigation.

If the trace is absent, inspect sampling/export health and alternative evidence. If clocks differ, align durations and causal parentage carefully. If work crosses a queue, inject context into the producer message and extract it at the consumer with the instrumentation's semantic conventions; batches/fan-in may need links rather than a fabricated single parent. Never treat incoming baggage or trace IDs as authorization claims.

Four golden signals and actionable alerts

Google SRE's four signals are latency, traffic, errors and saturation. Measure latency distributions with an explicit population and distinguish successful from failed/fast-rejected requests. Traffic describes demand; errors need a denominator and a definition that reflects user outcomes, not merely one HTTP status class. Saturation includes queue/pool pressure and resource limits. A mean is useful for some questions but cannot represent every tail; p50/p95/p99 answer different ones.

Alerts should trigger an action. User-impact/error-budget symptoms often provide good paging criteria; imminent capacity exhaustion or a security/audit condition can also justify an actionable alert. It is false that alerts must always be metrics or that CPU can never warrant paging. Reduce noisy per-log-line pages by defining windows, thresholds, minimum traffic, routing and a runbook. The SRE workbook's multiwindow burn-rate approach is a concrete alternative to a universal percentage threshold.

For a 99.9% success SLO, the allowed error fraction is 0.001. A 0.01 observed eligible-request error fraction consumes budget at 0.01 / 0.001 = 10 times the sustainable rate under that window. This does not by itself select a paging duration; use the specified SLO period, burn-window design and traffic assumptions. Exclude or include health checks explicitly rather than letting scrape traffic distort the user denominator.

Pipeline: instrumentation is not storage

1. Application instrumentation records well-defined measurements and spans; logging records allowlisted events while the relevant span is active.
2. SDK processors/exporters batch or emit them. A bounded queue can drop under pressure; synchronous exporting can add request latency.
3. A collector/agent can receive, transform, redact, batch and route. Redact sensitive data at the application boundary too; downstream filtering cannot un-send it.
4. Transport may be OTLP/gRPC or OTLP/HTTP, a Prometheus scrape or another configured path. TLS, authentication and network policy still matter; “pull means no auth headaches” is false.
5. Backends store/index data and provide query APIs; dashboards and alert routing consume those APIs. Swapping vendors may preserve instrumentation but still requires exporter/configuration and semantic compatibility work.

Monitor the pipeline itself: queue utilization, export failures, dropped spans/events, scrape gaps, collector memory and ingestion throttling. Three empty or delayed signals are not evidence that users are fine.

Derive cardinality and cost

One metric time series corresponds to a metric name plus a distinct label-value combination. If 4 methods, 20 route templates, 5 status classes and 3 regions all co-occur, the counter can have 4*20*5*3 = 1200 series. The product is an upper bound if many combinations never occur. Adding a user ID multiplies potential combinations and churn; a request ID can create a new series for nearly every event.

For a classic Prometheus histogram with 10 finite bucket boundaries, there are 11 cumulative buckets including +Inf, plus sum and count: 13 core series per label combination. Some clients also expose created timestamps; native histograms have a different representation. Thus the classic core upper bound here is 1200*13 = 15600, not “under 100 labels” or a universal one-million-series crash threshold.

Logs/traces generally pay for events/spans, attributes, indexes, retention and query work; indexed high-cardinality fields can still be expensive. Loki labels also have cardinality constraints. Moving raw IDs out of metric labels does not make storing every ID free or privacy-safe. Use route templates rather than /order/12345, bound label vocabularies, and choose approved identifiers in logs/spans when investigation value justifies them.

Sampled traces do not directly estimate population rates

Suppose 9900 successes and 100 failures occur. Keep 1% of successes (99) and every failure (100). The retained set has 100/199 ≈ 50.25% errors although the population error fraction is 1%. Unsampled eligible-event metrics give the direct rate. An estimator using retained events needs the valid inclusion probabilities and assumptions; tail policy can make that difficult.

Head sampling decides early, cheaply, before seeing the final outcome unless the decision uses information already available. Tail sampling waits for more spans/outcomes, retaining interesting traces under configured policies but requiring buffering, deadlines and routing of trace fragments. It cannot keep an error that upstream head sampling already discarded, and late/dropped spans or full buffers can defeat “keep every error.” Keep a representative baseline for successful requests too: apparently successful operations can contain performance or business defects.

Hands-on: exact correlation and aggregate oracles offline

This fixture uses contextvars to model active span context, a fixed logical clock and deterministic IDs. It emits one root and one child span per operation, one structured event inside the root, and unsampled route/status counters plus cumulative histogram buckets. It is not an OpenTelemetry implementation or exporter.

from collections import Counter
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from itertools import count
import json
 
current = ContextVar('current_span', default=None)
ids = count(1)
spans, events = [], []
requests = Counter()
buckets = [5, 10, 50, 100, float('inf')]
hist = Counter()
clock = [0]
 
@dataclass
class Span:
    trace_id: str
    span_id: str
    parent_id: str | None
    name: str
    start: int
    end: int | None = None
    error: bool = False
 
@contextmanager
def span(name):
    parent = current.get()
    sid = format(next(ids), '016x')
    tid = parent.trace_id if parent else format(next(ids), '032x')
    item = Span(tid, sid, parent.span_id if parent else None, name, clock[0])
    token = current.set(item)
    try:
        yield item
    except Exception:
        item.error = True
        raise
    finally:
        item.end = clock[0]
        spans.append(item)
        current.reset(token)
 
def emit(outcome):
    active = current.get()
    assert active is not None
    # Allowlist: no request body, token, email or raw URL.
    events.append({'event': 'checkout.complete', 'outcome': outcome,
                   'trace_id': active.trace_id, 'span_id': active.span_id})
 
def checkout(found=True, delay=5):
    start = clock[0]
    status = '200'
    with span('GET /order/{id}'):
        try:
            with span('db.lookup'):
                clock[0] += delay
                if not found:
                    raise LookupError('fixture absent')
            emit('ok')
        except LookupError:
            status = '404'
            emit('not_found')
        requests[('GET', '/order/{id}', status)] += 1
        elapsed = clock[0] - start
        for boundary in buckets:
            hist[boundary] += elapsed <= boundary
    return status
 
assert checkout(True, 5) == '200'
assert checkout(True, 50) == '200'
assert checkout(False, 10) == '404'
assert sum(requests.values()) == 3
assert [hist[x] for x in buckets] == [1, 2, 3, 3, 3]
assert len(spans) == 6 and len(events) == 3
by_id = {s.span_id: s for s in spans}
for event in events:
    owner = by_id[event['span_id']]
    assert owner.trace_id == event['trace_id'] and owner.parent_id is None
for item in spans:
    if item.parent_id:
        parent = by_id[item.parent_id]
        assert item.trace_id == parent.trace_id
        assert parent.start <= item.start <= item.end <= parent.end
assert sum(s.error for s in spans) == 1
assert current.get() is None
assert 4 * 20 * 5 * 3 == 1200
assert 1200 * (10 + 1 + 2) == 15600
assert 100 / 10000 == .01 and 100 / 199 > .5
assert abs(.01 / .001 - 10) < 1e-12
for event in events:
    assert set(event) == {'event', 'outcome', 'trace_id', 'span_id'}
    json.dumps(event)
print('correlation, contexts, histogram, cardinality and sampling bias: PASS')

Anatomy and interpretation

The log/event is emitted inside the root span, fixing the old initial log line that ran before a span existed. Each child shares the trace ID and names its actual parent. A handled not-found raises inside the child, marking it errored, while the root can represent a completed 404 according to its policy; HTTP 404 is not automatically a server fault. Context restoration in finally prevents request context leaking into the next operation.

The histogram buckets are cumulative: a 5-unit observation increments every boundary at least 5. The three durations 5, 50, 10 produce [1,2,3,3,3]. Counts are unsampled, whereas stored traces could be sampled later. Do not sum parent and child durations to estimate end-to-end latency: their intervals overlap. Independent oracles check alternate durations, exception cleanup and trace/context identities.

OpenTelemetry Python and Flask reference

This complete reference targets OpenTelemetry Python API/SDK 1.38.0 interfaces. The SDK is not installed or executed in this review; tagged source and documentation establish the API contract, while the offline fixture above supplies executed correlation/aggregation checks. No new dependency is needed merely to run the reviewed models. In a separately authorized environment, use the pinned compatible packages and review current security/support requirements before deploying an old teaching pin.

It uses OTel metrics/traces and a structured standard-library log with trace correlation. It does not call that log an OTel LogRecord export, nor claim a Prometheus scrape endpoint exists. Configure the appropriate exporters/collector separately; do not expose development servers or telemetry endpoints publicly.

# three_pillars_reference.py — API/source reviewed; no SDK/server execution here.
import json
import logging
import time
from flask import Flask, jsonify
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
 
traces = TracerProvider()
traces.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
tracer = traces.get_tracer('orders.reference', '1.0')
reader = InMemoryMetricReader()
meters = MeterProvider(metric_readers=[reader])
meter = meters.get_meter('orders.reference', '1.0')
requests = meter.create_counter('demo.http.requests', unit='{request}')
latency = meter.create_histogram('demo.http.duration', unit='s')
log = logging.getLogger('orders.reference')
log.addHandler(logging.StreamHandler())
log.setLevel(logging.INFO)
log.propagate = False
app = Flask(__name__)
 
def completion_event(status):
    context = trace.get_current_span().get_span_context()
    log.info(json.dumps({'event': 'order.complete', 'status': status,
                        'trace_id': format(context.trace_id, '032x'),
                        'span_id': format(context.span_id, '016x')}))
 
@app.get('/order/<int:order_id>')
def order(order_id):
    start = time.perf_counter()
    status = 500
    with tracer.start_as_current_span('GET /order/{id}') as root:
        root.set_attribute('http.route', '/order/{id}')
        try:
            with tracer.start_as_current_span('db.lookup'):
                if order_id == 500:
                    raise LookupError('fixture absent')
                result = {'order_id': order_id, 'total': 42}
            with tracer.start_as_current_span('payment.check'):
                root.set_attribute('demo.payment.checked', True)
            status = 200
            return jsonify(result), status
        except LookupError:
            status = 404
            return jsonify({'error': 'not_found'}), status
        except Exception as error:
            root.record_exception(error)
            root.set_status(Status(StatusCode.ERROR))
            return jsonify({'error': 'internal'}), 500
        finally:
            root.set_attribute('http.response.status_code', status)
            completion_event(status)
            labels = {'http.request.method': 'GET', 'http.route': '/order/{id}',
                      'http.response.status_code': status}
            requests.add(1, labels)
            latency.record(time.perf_counter() - start, labels)
 
if __name__ == '__main__':
    # Flask test client exercises dispatch without listening on a socket.
    try:
        with app.test_client() as client:
            assert client.get('/order/42').status_code == 200
            assert client.get('/order/500').status_code == 404
        assert reader.get_metrics_data() is not None
    finally:
        meters.shutdown()
        traces.shutdown()

The explicit provider objects avoid silently replacing an application's global provider. Console export is synchronous and intended for the fixture, not high-throughput production. perf_counter measures monotonic elapsed time. Route templates bound cardinality; raw user/order IDs are not metric labels. The first-party OTel documentation and tagged SDK source define instrument signatures, current-span behavior and metric-reader usage. This reference deliberately lacks automatic Flask incoming context extraction; adding that requires supported instrumentation or explicit propagator extraction and a trusted ingress policy.

Cross-service context and Prometheus integration

For an outgoing HTTP request, the OTel propagator injects trace context into a carrier while the client span is active; the receiving service extracts it before starting the server span. Queue messages need equivalent propagation and lifecycle handling; links can represent batch relationships. Trace flags and remote context are untrusted telemetry metadata, not permission to reveal a secret or perform an operation. Limit baggage fields and propagation destinations.

A Prometheus integration can expose counters and classic histograms through its supported client/exporter and a protected scrape endpoint. The original Prometheus pattern used http_requests_total with method/route/status labels and http_request_duration_seconds buckets. Preserve that semantic distinction: counters increase or reset; gauges can go up/down; histograms capture distributions. A /metrics endpoint does not itself establish a scrape, storage, retention or secure access configuration.

Useful query shapes for classic histograms are source-supported references, not queries executed against a real backend:

sum(rate(http_requests_total[5m])) by (route)
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

Choose actual label names matching the instrumentation. Aggregate bucket rates with the le label before computing a quantile; do not average independently calculated per-instance p99 values. Define zero-traffic handling and whether business failures with HTTP 200 count in the SLI.

Sampling, retention and privacy design

Keeping everything maximizes recorded evidence only for what was instrumented; it cannot answer every possible future question. Ingestion and query costs can force short retention. Aggressive sampling reduces cost but can miss rare events; security/audit requirements may need a different durable, access-controlled stream. Tiered retention is a policy choice with an owner, not a universal fourteen-day/seven-day recipe.

For a hypothetical 10,000-RPS checkout service, 86,400 seconds/day yields 864 million requests/day. At one 500-byte event each, that is 432 GB/day before compression, indexes, replication and metadata. At five 300-byte spans/request, full spans add 1.296 TB/day before those factors. A 1% uniform sample reduces that span payload to about 12.96 GB/day, but retaining all failures/slow traces changes the volume according to their rate. These are transparent arithmetic assumptions, not vendor prices.

Use unsampled bounded-label success/error/latency metrics for the SLO; retain a representative trace baseline plus outcome-sensitive samples within a tested collector budget; record structured allowlisted completion/error events with correlation IDs. Pick retention from incident discovery needs, audit/privacy constraints and cost. Do not put raw authorization headers, payment data, full query strings or unbounded user text in attributes. Access controls, deletion/retention and incident handling apply to telemetry too.

Failure scenarios and diagnostic practice

Three unconnected datasets: metric spike exists but logs lack trace IDs and spans stop at a queue. Repair context and propagation tests, not just dashboard links. Biased retained traces: errors look like half of traffic because the sample keeps all failures; use unsampled counters or a justified estimator. Label explosion: raw paths create growing series; replace with templates and bound attributes. Collector saturation: dropped spans hide the interesting requests; monitor pipeline loss and size buffers/limits. Log-only investigation: large volume can make scanning slow, but indexed structured events may still answer the question; choose representation from measured query needs rather than declaring logs useless.

A wide-event query grouping a slow subset by deployment/feature/region can reveal a correlation worth testing. The same outcome may be available from appropriately recorded trace attributes. No source here demonstrates the old invented Honeycomb customer timing or that tracing is the only possible debugging method for more than two services.

Recall and worked answers

1. Define the three signals

Metrics describe numeric populations over time; logs/events describe occurrences; traces connect recorded spans of work. They overlap and can be derived when enough information survives, but no combination reconstructs data that was never recorded.

2. What follows a p99 alert?

Inspect the affected population and a representative slow trace/exemplar if available, then correlated logs and resource evidence. Check sampling and export health before assuming an absent trace means an absent failure. Test the bottleneck hypothesis rather than treating correlation as causation.

3. Four golden signals

Latency, traffic, errors and saturation. Define populations/windows and user outcomes; include resource/pool pressure and separate successful/error latency where meaningful. An average and a tail percentile serve different questions.

4. Cardinality

Count distinct co-occurring label combinations, not just number of label names. The fixture's 4×20×5×3 upper bound is 1200; a ten-finite-bucket classic histogram has 13 core series per combination. Raw request IDs make churn unbounded. Logs and trace indexes can also pay cardinality costs.

5. Why sample and what is lost?

Reduce stored spans/ingestion/query load while retaining useful examples. Head sampling cannot use an unseen final outcome; tail sampling needs buffering and can still lose late/dropped/head-discarded data. Keep unsampled aggregate SLIs and a useful success baseline. “100% errors kept” is a tested pipeline objective, not a universal guarantee.

6. What does OTel provide?

Vendor-neutral APIs/SDK conventions, instrumentation and export paths. It does not store everything, emit every signal from one call, or eliminate exporter/configuration changes between backends. Standard-library correlated logs in the reference are not falsely advertised as OTel log export.

7. Correlate logs and traces

Emit trace/span IDs while the relevant span context is active; propagate across service/queue boundaries deliberately. Test context cleanup and missing-context behavior. UI linking needs backend/query configuration, and a remote trace ID must not become an authorization credential.

8. Reduce paging fatigue

Page on actionable impact or imminent risk using a defined population/window, often SLO burn rates and supporting metrics. Route logs/security alerts separately where justified. Add runbooks and remove unactionable duplicates; “metrics only” is not a law.

9. Wide-event argument

Preserve richer approved operation attributes so later queries can group/filter without having anticipated every dimension. It trades pre-aggregation for retained evidence and query cost, with schema, index, privacy and sampling constraints. It does not give infinite or free drill-down.

10. Design the 10K-RPS service

Budget 864 million daily requests under the stated constant-rate assumption. Instrument bounded route/status/region dimensions and unsampled eligible-event metrics, correlated spans with tested sampling, and allowlisted structured logs. Choose retention and collectors from payload estimates and requirements; monitor drops, protect secrets and use an independent audit stream where necessary. Do not copy arbitrary months/days retention as an established best practice.

Explain-out-loud test

Trace one completion event to its parent/child spans, compute the retained-sample bias, explain histogram series multiplication and specify what an alert operator should do. Reproduce the correlation without a dashboard before claiming the dashboard solved observability.

Optional historical pointer and onward study

The original Dapper publication pointer is retained as a historical discovery URL; the canonical titled publication below supplies the checked identity. No old days-to-minutes investigation claim or fixed sampling rate is accepted from this pointer. The original Honeycomb link in preparation preserves the attributed wide-event argument, not the invented customer anecdote.

Continue with Prometheus, Grafana and OpenTelemetry for tool-specific work; use SLOs/error budgets to choose eligible-event measurements, incident response to act on evidence, multi-region work to compare regional paths, rate limiting to interpret saturation and system-design reviews to state what would be monitored. Learning plan. These destinations are navigation, not a claim that their service integrations were tested here.

Connections and primary sources

Tool-specific OTel/Prometheus lessons supply deployment details; SLOs define what to measure; incident response uses the evidence; multi-region and rate-limit decisions require regional/queue context. A design review should name both telemetry and its collection/privacy failure modes.

Full current-body review on 2026-10-10 reverified saved primary snapshots captured on 2026-10-08 (not a fresh network retrieval): OpenTelemetry concepts/Python documentation, tagged Python 1.38.0 source, Prometheus metric/instrumentation guidance, Google SRE monitoring/workbook, Dapper and Honeycomb's manifesto. Offline exact programs and mathematical oracles were executed. OTel SDK, collector/exporters, Flask reference dispatch, Prometheus queries, browser dashboards and retention storage were not run.

• OpenTelemetry signals and context propagation
• Python instrumentation
• Prometheus metric types and instrumentation practices
• Google SRE monitoring and SLO alerting
• Dapper, 2010
• Honeycomb observability manifesto