Prometheus, Grafana, OpenTelemetry — Hands-on

A request can be slow while every process is up. The useful question is not “which monitoring product should we install?” but “which population did we measure, where did the observations go, and what decision does the alert justify?” This lesson builds that chain with Prometheus, Grafana and OpenTelemetry (OTel).

Optional preparation

Choose the introduction if these tools are new; use the deeper pointers to compare scrape/query vocabulary, dashboard construction and Collector pipeline design. The original “30-minute pre-read” was a planning cue, not the sum of the listed durations. These are optional finding labels, not claims that the videos were watched, their creators verified, or their availability/runtime checked.

• Prometheus in 100 Seconds — introductory video pointer for pull and time-series vocabulary.
• Prometheus Explained — Beginner's Guide — optional longer walkthrough pointer for scrape configuration, PromQL, alerts and Grafana; no quality ranking implied.
• Decoding OpenTelemetry Collector: a deep dive — architecture pointer; compare receivers/processors/exporters with the pinned configuration below.
• Prometheus getting started — optional tutorial; its latest-version instructions are not the 2.53.0 contract used here.
• Grafana first dashboard — optional UI exercise: reproduce a panel, then change a query. UI steps were not run here.
• Collector configuration — optional pipeline reference; enabled signal wiring and versioned components are taught below.

Scraping, storing, querying and routing

Think of a weather station: sensors expose measurements, a collector records them, and a display queries the history. Prometheus usually scrapes HTTP endpoints and stores labelled time series. Grafana queries data sources and presents panels; it is not the metric store. OTel defines instrumentation APIs, SDKs, semantic conventions and protocols; its Collector receives, processes and exports telemetry, rather than serving as the durable analytics backend.

Pull gives the observer a configured/discovered target inventory, scrape scheduling and the up metric. up=0 means the scrape failed, not necessarily that the application is down: authentication, routing, TLS, timeout or an invalid exposition can fail independently of user requests. A target removed from discovery can disappear instead of remaining up=0. Monitor discovery and absent series as well as failures. Push systems can also maintain expected-sender inventories and heartbeats; pull is not the only way to detect missing telemetry. Large responses and too many series can overload a pull collector despite its controlled schedule. Prometheus configuration defines both scrape and sample limits.

Short-lived service-level batch jobs may finish between scrapes. The Pushgateway guidance describes its narrower role and the need to manage retained series; it is not a universal replacement for scraping long-running services. Pull also requires network reachability in the observer-to-target direction; neither direction is universally easier to secure.

The telemetry path

This diagram is an architectural option, not a claim that every exporter exists in every distribution. An alternative sends OTel metrics to a Collector with a Prometheus exporter, which exposes a scrape endpoint. Prometheus pulls that endpoint; the exporter does not magically insert into Prometheus's TSDB. A remote-write exporter instead pushes to a separately configured compatible receiver. Never ingest the same logical measurements through both routes without accounting for duplicates.

Signal correlation: put a trace/span ID in structured logs when a span exists; use exemplars where the metric/backend supports them. Request IDs are unsuitable as unbounded metric labels. Clocks help compare timings but do not establish causal order alone; parent/child span context provides that relationship. Trace sampling is a budget choice, not a mathematical necessity for every deployment. Parent-based head sampling and tail sampling have different decision points and failure modes. Logs without spans can still be useful, and metrics do not have infinite retention.

History and product choice without invented thresholds

Prometheus's official overview dates its SoundCloud inception to 2012 and CNCF joining to 2016. OTel combined OpenTracing and OpenCensus; the 2022 metrics announcement is a better milestone than calling every language's metrics SDK stable in 2021. Specification, protocol, individual SDK and exporter stability are separate. The reference below deliberately uses the original 2024-era versions, not an unverified “latest” stack.

The Prometheus 3.0 announcement describes configurable OTLP metrics ingestion and the 2024 major release; that is a later option, not the 2.53.0 scrape route in this exercise. The OpenTracing/OpenCensus merger explains OTel’s common API/protocol goal, not universal backend equivalence. Grafana’s immutable v1.0 README describes a Graphite dashboard and says it is based on Kibana. That early dashboard/client-side architecture is history, not the Grafana 11.1 deployment model. Grafana’s dashboard role is distinct from both telemetry instrumentation and storage; origin stories do not establish present compatibility.

Self-hosting trades vendor management for responsibility for retention, backups, patching, tenant isolation and on-call operation. A hosted offering can reduce that work but introduces pricing dimensions, egress, retention rules and service limits. Compare actual query compatibility, histogram support, authentication, alert migration and data residency before moving between Grafana Cloud, Datadog or another backend. This lesson makes no current free-tier or cost-multiplier promise. Single-node Prometheus does not stop at 15 days or a universal one-to-two-million-series ceiling: retention is configurable, and feasible capacity depends on hardware, churn, scrape volume and queries. Mimir/Thanos and managed systems address additional scale, durability or availability requirements; they are not required merely to store a sixteenth day.

From counters to population-correct PromQL

A series is identified by its metric name and full label set. http_requests_total{method="GET",endpoint="/order/<id>",status="200"} is different from the same labels with status 500. Prometheus samples values; it does not store one event row per request. The exact three label values must match the instrumentation, including the templated endpoint spelling.

• A counter increases and may reset on restart. Use rate(counter[5m]) for a per-second estimate or increase(counter[5m]) for the estimated increase over the window. Neither is an exact event ledger.
• A gauge can rise and fall: queue depth or resident bytes. Do not treat a declining gauge as a counter reset.
• An instant vector has one selected sample per matching series at the evaluation time, subject to lookback/staleness. A range vector contains samples over a window.
• Compute rate on each original counter before summing, so resets remain visible. sum by (status) (...) drops other dimensions intentionally.

# Request rate per status for this job
sum by (status) (rate(http_requests_total{job="app"}[5m]))
 
# Weighted fleet error ratio; missing numerator becomes zero only for existing totals
(sum(rate(http_requests_total{job="app",status=~"5.."}[5m]))
 or 0 * sum(rate(http_requests_total{job="app"}[5m])))
/
sum(rate(http_requests_total{job="app"}[5m]))
 
# Fleet p99 estimate: preserve le for classic histograms
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{
    job="app",method="GET",endpoint="/order/<id>"
  }[5m])))

A zero denominator is not healthy traffic; it is an undefined ratio. A missing target is not zero traffic either. Handle low traffic and absent scrapes with distinct alerts or corroborating probes. In a multi-service query retain job or a service label through numerator, denominator and histogram aggregation; otherwise one healthy high-volume service can mask another.

If A has 10 requests and one error, while B has 10,000 and ten errors, averaging their error percentages gives 5.05%. The true combined fraction is 11/10010, about 0.10989%. Weight by event counts. Likewise, averaging two p99 values does not recover a fleet p99. A mean latency is valid for total-work/capacity analysis; it simply does not describe the tail. With 99% at 10 ms and 1% at 10 seconds, the mean is 109.9 ms, while that slow 1% remains important.

Histograms, summaries and cardinality

Classic histogram buckets are cumulative counters. Aggregate compatible boundaries across replicas, retain le, then estimate a quantile. The returned value is interpolated inside a bucket, not the exact observed p99. Put a boundary at a latency SLO threshold: the bucket count divided by total count directly measures the fraction at or below that threshold (subject to collection/rate estimation). With k finite boundaries, account for the +Inf bucket, _sum and _count, and possibly client-created series; “12 buckets means exactly 12 series” is incomplete.

Native/exponential histograms use sparse bucket representations and can cover wide ranges. Check compatibility through client, scrape protocol, Prometheus version, remote storage and dashboards. In Prometheus 2.53 they are a version-specific feature, not an assumed universal replacement. A summary can provide client-computed quantiles with configured error/window behavior, where the client supports them. Such quantiles cannot generally be merged across replicas; summary sums and counts can still be aggregated for a mean. The histogram guide distinguishes these operations.

Cardinality is the number of observed series, not necessarily the full Cartesian product of labels. Nevertheless, 20 routes × 4 methods × 5 status classes × 10 instances can create 4,000 series for one metric if all combinations occur. Adding 100,000 user values is a dangerous potential multiplier. IDs and arbitrary URLs belong in controlled logs/events/traces, not metric dimensions.

# Active series in Prometheus head
prometheus_tsdb_head_series
 
# Expensive diagnostic: count currently selected series by metric name
# Scope this to a known job; do not run unbounded on a struggling server.
topk(10, count by (__name__) ({job="app",__name__=~".+"}))
 
# Number of currently observed user_id values, if this bad label already exists
count(count by (user_id) (http_requests_total{job="app"}))

count by (__name__) (scrape_samples_scraped) counts scraper meta-series, not the application metric families causing a spike. Label cardinality is not an exact storage-bill multiplier, and series disappear according to staleness/head/retention behavior, not “indexed forever.” During an incident, fix instrumentation or drop the offending metric carefully. Blindly dropping an identity label can collapse formerly distinct series into duplicate label sets; relabelling does not aggregate their values.

A local reference stack with explicit wiring

The following is a source-reviewed, unexecuted service reference, not a production deployment or a request to install anything. Version contract: Prometheus 2.53.0, Grafana 11.1.0, Collector contrib 0.104.0, OTel Python 1.25.0. These historical pins make APIs inspectable; check maintained releases and advisories before any new deployment. Use the installed pure model below for the executed exercise.

To avoid the original Docker-host/loopback mismatch, this reference uses native processes on one development machine, all endpoints on 127.0.0.1. A container cannot reach a host-loopback Flask process just because its target is named host.docker.internal. If converting to Compose, put the app and collectors on the same private network, use service DNS, and publish only loopback-bound host ports. Do not enable anonymous Grafana Admin. Keep Grafana authentication enabled and configure credentials through its supported private setup, never in committed YAML.

Save these files together; launch tools only in a separately authorized local exercise. Prometheus needs --config.file=prometheus.yml --web.listen-address=127.0.0.1:9090. The Collector needs --config=otel-config.yaml. Configure Grafana's HTTP address as 127.0.0.1 and its Prometheus data-source URL as http://127.0.0.1:9090. Mount/copy both the scrape config and the rule file when adapting to a container.

# prometheus.yml -- rule path relative to this config file
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - alert_rules.yml
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["127.0.0.1:5000"]
  - job_name: otel-collector
    static_configs:
      - targets: ["127.0.0.1:8889"]

Instrument the app once, with bounded labels

The app exports Prometheus metrics directly and sends traces through OTLP. It does not send OTel metrics merely because the Collector has a metrics pipeline. The route label and method must be included on the histogram if queries filter them. A monotonic timer avoids wall-clock corrections corrupting durations. Controlled 500 responses mark spans as errors; exceptions must still record metrics through finally.

# Reference app.py; not run here (SDK/exporter and prometheus-client unavailable).
# Dependencies: Flask 3.0.2, prometheus-client 0.20.0; OTel API/SDK/OTLP-gRPC exporter 1.25.0.
import time
from flask import Flask
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
 
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://127.0.0.1:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("orders.reference")
app = Flask(__name__)
requests = Counter("http_requests_total", "Requests", ["method", "endpoint", "status"])
latency = Histogram("http_request_duration_seconds", "Duration", ["method", "endpoint"],
                    buckets=(.005, .01, .025, .05, .1, .3, .5, 1, 2.5))
 
@app.get("/order/<int:order_id>")
def order(order_id):
    started = time.perf_counter()
    status = 500
    with tracer.start_as_current_span("get_order") as span:
        try:
            with tracer.start_as_current_span("db_lookup"):
                result = {"order_id": order_id}  # synthetic, no database
            with tracer.start_as_current_span("compute"):
                status = 500 if order_id % 20 == 0 else 200
            if status == 500:
                span.set_status(Status(StatusCode.ERROR))
            return result, status
        finally:
            requests.labels("GET", "/order/<id>", str(status)).inc()
            latency.labels("GET", "/order/<id>").observe(time.perf_counter() - started)
 
@app.get("/metrics")
def metrics():
    return generate_latest(), 200, {"Content-Type": CONTENT_TYPE_LATEST}
 
if __name__ == "__main__":
    try:
        app.run(host="127.0.0.1", port=5000, debug=False)
    finally:
        provider.shutdown()

This local OTLP channel is intentionally plaintext only on loopback; it is not suitable across a network. BatchSpanProcessor already batches SDK exports, so removing Collector batching does not imply one RPC per span. A real service adds automatic propagation, exception instrumentation, resource identity, privacy policy, TLS/authentication and export-loss monitoring. Do not put personal order data into attributes without a purpose/retention decision.

Collector configuration and all three signals

Receivers accept signals, processors transform them in the listed order, exporters forward or expose them, and service.pipelines enables the wiring. Defining a receiver without referencing it does nothing. In the pinned contrib distribution, the attributes processor supports traces, metrics and logs, but it changes signal attributes rather than resource identity. A resource processor is the appropriate choice for the deployment resource attribute used here. batch is an efficiency control, not durable delivery. Memory limits and queue/backpressure settings need workload testing.

# otel-config.yaml -- Collector contrib 0.104.0
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318
processors:
  resource:
    attributes:
      - key: deployment.environment
        value: local-reference
        action: upsert
  batch:
    timeout: 5s
exporters:
  prometheus:
    endpoint: 127.0.0.1:8889
  debug:
    verbosity: basic
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheus, debug]
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [debug]

The resource processor’s deployment.environment attribute does not automatically become a Prometheus label on every metric. In the pinned exporter, resource_to_telemetry_conversion.enabled defaults to false; enabling it copies resource attributes to metric labels and therefore needs a cardinality review. The direct app metrics do not pass through this resource processor at all.

For logs, configure a language's OTel log bridge/exporter or a supported filelog receiver for structured files; merely printing to stdout does not send OTLP. For durable traces/logs replace debug with supported OTLP/OTLP-HTTP exporters configured for each backend's endpoint, TLS and credentials. Tempo can receive OTLP traces; Loki ingestion depends on its version and OTLP/log configuration. Do not assume a historical exporter name exists in a different distribution or release. The 0.104.0 contrib manifest still includes loki; this reference uses debug and prometheus. The 0.104.0 manifest is the distribution inventory. Test with a known trace and a log carrying its trace ID, then query both backends and verify correlation; none of those service steps was executed here.

Dashboard and alert as inspectable code

Prometheus evaluates this threshold rule. The five-minute rate window and the for duration do different things: the first selects samples; the second requires a continuously active expression across evaluations. A 1% threshold becomes a burn rate only after declaring an objective, eligible population and budget window.

# alert_rules.yml
groups:
  - name: local-reference
    rules:
      - alert: HighErrorRate
        expr: |
          ((sum(rate(http_requests_total{job="app",status=~"5.."}[5m]))
            or 0 * sum(rate(http_requests_total{job="app"}[5m])))
           / sum(rate(http_requests_total{job="app"}[5m]))) > 0.01
        for: 5m
        labels:
          severity: investigate
        annotations:
          summary: "Measured 5xx ratio above 1% for 5m"
      - alert: AppScrapeFailed
        expr: up{job="app"} == 0
        for: 1m
        annotations:
          summary: "App metrics scrape is failing; check user impact separately"
      - alert: AppTargetAbsent
        expr: absent(up{job="app"})
        for: 1m
        annotations:
          summary: "No app target series; inspect discovery and collector health"

AppTargetAbsent detects an empty app-job selection, not one lost replica while another still exports up. A multi-replica service needs an expected-target inventory or an independently justified count/per-target disappearance check. Likewise, the numerator fallback means “no selected 5xx series for an existing total”; it cannot prove error instrumentation is intact. Scrape health, discovery and business correctness remain separate signals.

For a separately run local exercise, request IDs 1 through 100 once: five deterministic failures make the controlled mix 5%, not a random promise about every sample. Verify up, counter labels, nonempty histogram buckets and rule state before interpreting the dashboard. A sparse window or a missed scrape can delay or reset firing. Collector debug output depends on received batches; it is not guaranteed to print every five seconds.

Build RED panels for request rate, errors and duration; USE panels for resource utilization, saturation and errors; the four golden signals add traffic and saturation to the service view. Saturation needs a meaningful denominator: queue depth versus capacity, active workers versus worker limit, or resident bytes versus an explicitly exported memory limit. Dividing Python RSS by a Go heap metric from another process has no defensible meaning.

{
  "type": "timeseries",
  "title": "Request rate by status",
  "targets": [{
    "expr": "sum by (status) (rate(http_requests_total{job=\"app\"}[5m]))",
    "legendFormat": "{{status}}"
  }]
}

This is a panel fragment, not a complete importable dashboard. Add a data-source reference, dashboard UID/schema/version and the other panels when exporting from the chosen Grafana release. Provisioning distinguishes data-source YAML, dashboard-provider YAML and dashboard JSON. Version controlling these makes changes reviewable; ordinary upgrades do not inherently erase click-created dashboards. Check units (seconds versus milliseconds), denominator labels, absent-data presentation and query time range before saving.

Executed population model

This exact standard-library program verifies arithmetic and bucket semantics, not PromQL's extrapolation algorithm or a running server.

from fractions import Fraction
from bisect import bisect_left
 
def error_ratio(populations):
    total = sum(n for bad, n in populations)
    if any(bad < 0 or n < bad for bad, n in populations):
        raise ValueError("invalid population")
    return None if total == 0 else Fraction(sum(b for b, n in populations), total)
 
def cumulative_buckets(samples, bounds):
    if list(bounds) != sorted(set(bounds)):
        raise ValueError("strictly increasing bounds required")
    bins = [0] * (len(bounds) + 1)
    for x in samples:
        if x < 0:
            raise ValueError("negative duration")
        bins[bisect_left(bounds, x)] += 1
    running = 0
    cumulative = []
    for count in bins:
        running += count
        cumulative.append(running)
    return cumulative  # last entry is +Inf
 
assert error_ratio([(1, 10), (10, 10000)]) == Fraction(11, 10010)
assert error_ratio([(0, 0)]) is None
assert cumulative_buckets([.1, .3, .3, .7], [.1, .3, 1.0]) == [1, 3, 4, 4]
assert error_ratio([(1, 10), (10, 10000)]) != (Fraction(1, 10) + Fraction(10, 10000)) / 2
print("weighted ratios and cumulative histogram boundaries: PASS")

Production decisions and recovery practice

For a hypothetical 200-service Kubernetes deployment, choose whether collectors run as node agents, sidecars or gateways based on receivers, tenancy and tail-sampling locality. Scrape application/node metrics; route traces/logs to purpose-built stores; link Grafana data sources through trace IDs/exemplars; send rule alerts to an Alertmanager routing policy. Add availability, authentication, cardinality budgets, queue/drop monitoring, retention and cost controls. “200 services” alone does not justify a particular backend or collector topology.

A cardinality incident exercise: a new route exports raw URL IDs, series increase, and memory pressure rises. Compare scrape sample counts and scoped series inventories; roll back that instrumentation; stop ingesting the offending metric if necessary. Do not collapse labels without checking duplicates. Preserve a small sanitized sample, then add a bounded-route test and series budget to review. A separate exercise migrates a backend: verify semantic conventions, resource-to-label translation, histogram types and alert population parity before saying instrumentation is unchanged. These are worked failure scenarios, not attributed outage statistics.

Optional production reading pointers retained from the original: Prometheus introduction/history for the project's own origin account; cardinality spikes for an external operational perspective; and Mimir for distributed-storage product context. These pointers are not evidence for the old invented scale ceilings, cost multipliers or outage numbers. Migration requires query, label, histogram, retention and alert parity checks, not “zero-code” faith.

Recall and worked answers

1. Why pull? An observer controls scrapes and emits scrape-health metrics for known targets. It cannot distinguish every discovery failure or guarantee the application is healthy; push plus heartbeats is also possible.
2. GET order p99? Use the histogram query above: filter labels that actually exist, rate per bucket series, sum by (le), then histogram_quantile(0.99, ...). The output is a bucket-based estimate.
3. Why not only average latency? A mean hides the distribution but remains useful for work accounting. Report tails, counts and errors too; never average replica quantiles.
4. What is cardinality? Distinct observed name/label sets. More dimensions can multiply them; there is no fixed universal failure threshold.
5. What does the Collector do? Receives, processes and exports configured signals. Protocol support reduces coupling, but vendor migration still needs mapping and behavior validation.
6. Pipeline pieces? Receivers, ordered processors, exporters, enabled separately under metrics/logs/traces pipelines.
7. Error rate above 1% for five minutes? Use HighErrorRate in “Dashboard and alert as inspectable code” above and interpret missing/zero traffic separately. It is a threshold alert, not by itself an SLO definition.
8. Other Grafana stores? Logs, traces, SQL and cloud-monitoring data sources via supported plugins; confirm the specific plugin/version/authentication before use.
9. Why distributed metric storage? Required availability, operational capacity, tenant isolation or durable long-term storage, not a mandatory 15-day boundary.
10. What runs where for 200 services? Instrument applications, collect with a deliberately chosen agent/gateway topology, store each signal, query with Grafana and route actionable alerts. Include pipeline failure monitoring and data/privacy budgets, not just product boxes.

Explain out loud how a counter reset differs from an application error, why a scrape failure is not an application-outage proof, and why backend changes need tests even with OTel.

Prerequisites and study route

Bring HTTP, basic Python, counters/gauges and the distinction between metrics, logs and traces. Continue from instrumentation to SLO policy, then incident response; container networking is an alternative deployment topic, not an assumed environment here.

Preparation links: S077, three pillars supplies signal concepts; Docker, now S065 is optional for a container adaptation, not a prerequisite for the native reference. Continue to S079, SLOs to turn observations into explicit contracts, then S080, incident response to act on evidence. Kubernetes basics, now S066 provides orchestration context for collection topology, and rate limiting and token buckets explores saturation controls; neither is automatically configured by these examples.

The original route labels “S056 Docker”, “S090 Kubernetes” and “S089 Rate limiting” no longer match those numbered articles. The links above follow inspected topic identities instead; the original labels remain here for orientation.

Next: S079 · SLIs, SLOs & Error Budgets — the SRE Math.

Hub: The 6-Month Learning Plan. This retains the original series hub, not a new endorsement of every linked lesson.

Study pacing: this is S078 in the original 130-session series. Use intuition → visual model → bounded practice → production decisions → recall as the learning route. The original 90-minute session and 5/15/25/15/10-minute phase labels are flexible planning cues, not a verified completion time (the phase labels total 70 minutes). Leave time for preparation and discussion; service deployment remains a separate authorized exercise.

Sources and verification boundary

The earlier primary snapshots were captured on 2026-10-08. This body and its governing passages were re-reviewed on 2026-10-10 with exact path/hash matching of inherited snapshots and fresh pinned-source additions where needed. Raw/canonical extraction, quotation offsets, original coverage and bounded fixtures are retained in the batch-42 packet. Historical pins describe their own versions, not current deployment recommendations.

The exact population model was executed; the historical app/Collector/Prometheus/Grafana reference was source/syntax-reviewed, not deployed. No SDK exporter or PromQL engine run is claimed.

• Overview | Prometheus — Mutable official documentation snapshot accessed 2026-10-08T15:53:58.288490+00:00.
• prometheus/prometheus — docs/configuration/configuration.md (v2.53.0) — repository tag v2.53.0.
• prometheus/prometheus — docs/querying/functions.md (v2.53.0) — repository tag v2.53.0.
• prometheus/prometheus — docs/storage.md (v2.53.0) — repository tag v2.53.0.
• Metric types | Prometheus — Mutable official documentation snapshot accessed 2026-10-08T15:35:48.304595+00:00.
• Histograms and summaries | Prometheus — Mutable official documentation snapshot accessed 2026-10-08T15:35:48.537435+00:00.
• When to use the Pushgateway | Prometheus — Mutable official documentation snapshot accessed 2026-10-08T15:35:48.568435+00:00.
• Instrumentation | Prometheus — Mutable official documentation snapshot accessed 2026-10-08T15:35:48.607108+00:00.
• Configuration | OpenTelemetry — Mutable official documentation snapshot accessed 2026-10-08T15:35:48.763829+00:00.
• open-telemetry/opentelemetry-collector-releases — distributions/otelcol-contrib/manifest.yaml (v0.104.0) — repository tag v0.104.0.
• open-telemetry/opentelemetry-collector-contrib — processor/resourceprocessor/README.md (v0.104.0) — repository tag v0.104.0.
• open-telemetry/opentelemetry-collector-contrib — processor/attributesprocessor/README.md (v0.104.0) — repository tag v0.104.0.
• open-telemetry/opentelemetry-collector — processor/batchprocessor/README.md (v0.104.0) — repository tag v0.104.0.
• open-telemetry/opentelemetry-collector-contrib — exporter/prometheusexporter/README.md (v0.104.0) — repository tag v0.104.0.
• open-telemetry/opentelemetry-python — exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/trace_exporter/init.py (v1.25.0) — repository tag v1.25.0.
• open-telemetry/opentelemetry-python — opentelemetry-sdk/src/opentelemetry/sdk/trace/init.py (v1.25.0) — repository tag v1.25.0.
• OpenTelemetry Metrics Release Candidates | OpenTelemetry — OpenTelemetry Metrics Release Candidates, 2022-05-18.
• Provision Grafana | Grafana documentation — Grafana v11.1 provisioning documentation.
• prometheus/prometheus — docs/configuration/alerting_rules.md (v2.53.0) — repository tag v2.53.0.
• prometheus/client_python — prometheus_client/metrics.py (v0.20.0) — repository tag v0.20.0.