MS Stack Ch 15 — Observability

Telemetry paths and versions

The classic examples use ASP.NET Core 8, Serilog.AspNetCore 8.x (8.0.3 source reference) and the classic Application Insights SDK 2.x pipeline. The alternative uses the Azure Monitor OpenTelemetry distro; its current configuration is checked against Microsoft documentation on 7 October 2026. These are alternative instrumentation paths, not two automatic request collectors to install together. KQL examples use the Application Insights requests/traces schema; workspace AppRequests/AppTraces uses different column names. No .NET collector/exporter or Azure service was executed; local sampling/SLI models do not prove end-to-end export.

Chapter 15 of From Novice to Fluent on the Modern Microsoft Web Stack — a 22-chapter self-study plan.

Why this chapter

You ship code. Then you stop seeing it. Without observability, every production deploy is a leap of faith and every incident is a 3am archaeology session through raw text logs. Done well, observability turns "the API feels slow" into "p99 on /checkout for the EU region jumped from 180ms to 1.2s at 14:03, correlated with a Kusto cluster CPU spike, after deploy d4e5f6". Done badly, observability is a six-figure App Insights bill, a dashboard nobody reads, and every alert ignored as noise.

Shipping-level observability means: structured logging via Serilog or ILogger<T>, Application Insights wired up, a handful of standard alerts and a single dashboard per service. Expert-level observability means: a deliberate sampling policy you can defend at cost-review, custom telemetry processors that redact PII before it hits the wire, a per-service SLO with an error budget burn-rate alert, and a fluency with KQL that lets you author new analytic queries in real time during an incident.

You finish this chapter when you can stand up a fresh service with logging, metrics, tracing and correlation IDs flowing through every hop, write the KQL that surfaces the worst-performing endpoint in the last hour, and explain to a teammate how message templates preserve named values for structured queries.

Log structurally — Message templates with named properties, never string interpolation.

Correlate instrumented hops — propagate W3C traceparent across supported boundaries and enrich logs deliberately; verify missing or broken edges.

Pick the right pillar — Logs for narratives, metrics for rates, traces for latency causality.

Sample deliberately — Choose a pipeline-specific policy, weight sampled counts and monitor missing evidence; exceptions are not magically retained.

Alert on signal — Six canonical alerts per service; tune to baseline; respect alert fatigue.

Query like a native — KQL fluency turns App Insights into a microscope, not a graveyard.

Concepts and depth

Logging philosophies: levels, structured vs string, sampling

Levels allow source/provider/sink filtering; events rejected early cannot be recovered by a later sink. Microsoft ILogger uses Trace/Debug/Information/Warning/Error/Critical; Serilog maps the endpoints to Verbose and Fatal:

• Trace — firehose; per-statement breadcrumbs; off in production.
• Debug — useful in dev, off in prod by default.
• Information — business events and lifecycle (request handled, user created, queue dequeued).
• Warning — degraded behaviour that did not fail: a retry, a fallback, a circuit half-open.
• Error — an operation failed; a user-visible problem.
• Critical — process is in trouble (OOM, can't serve at all).

Pick the level at the source, filter at the sink. The classic mistake is logging at Information from inside a hot loop — the next mistake is "fixing" it by raising the minimum level to Warning, which throws away the Information lines you actually need from elsewhere.

Structured logging means the log line is a key-value record, not a flat string. The same event "Created user in 234ms" becomes a structured record { "message": "Created user in {ElapsedMs}ms", "ElapsedMs": 234 }. The sink can filter, group and aggregate on the properties. String interpolation throws all of that away — the engine cannot distinguish "in 234ms" from "in 2.34s" without re-parsing.

Control cost through event selection, bounded dimensions, retention, aggregation and sampling. Classic adaptive sampling targets a rate and adjusts the kept fraction; fixed trace-aware sampling uses a stable trace decision. Exception exclusion is an explicit policy, not a universal default. Arbitrary per-item filtering can break correlation; redact before any export path and document the statistical bias of keeping selected classes.

Good enough to ship

• Information at the source; Warning at the sink for chatty libs. • Structured templates only; never $"..." in log calls. • Adaptive sampling at a defensible rate.

Expert tier

• Per-namespace level overrides codified in config. • Tenant-aware sampling for cost fairness. • Compliance redaction processors before the wire.

Serilog: sinks, enrichers, message templates, LogContext, JSON config

Serilog is the canonical structured-logging library on .NET. Its mental model is small:

• Sink — where logs go (Console, File, ApplicationInsights, Seq, Elasticsearch, …). Multiple sinks per logger.
• Enricher — augments every log event with extra properties (machine name, service name, trace id, correlation id).
• Message template — a string with {Named} placeholders that the sink captures as properties.
• LogContext — a thread/async-local scope of properties; everything inside the scope inherits them.

Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithProperty("Service", "queries-api")
    .Enrich.WithProperty("Env", builder.Environment.EnvironmentName)
    .WriteTo.Console(new Serilog.Formatting.Compact.CompactJsonFormatter())
    .CreateLogger();
builder.Host.UseSerilog();

public sealed class UserService(ILogger<UserService> log, IUserRepository repo)
{
    public async Task<User> CreateAsync(NewUser dto, CancellationToken ct)
    {
        var sw = Stopwatch.StartNew();
        using var scope = LogContext.PushProperty("Operation", "CreateUser");
        log.LogInformation("Creating user"); // no full DTO, email or credential
        var user = await repo.CreateAsync(dto, ct);
        log.LogInformation("Created user in {ElapsedMs}ms", sw.ElapsedMilliseconds);
        return user;
    }
}

IUserRepository, NewUser and User are application types. This fragment illustrates logging around an existing operation, not a full persistence implementation.

Two micro-rules worth internalising:

• In Serilog, {@Object} destructures into structured properties, while ordinary object formatting generally uses scalar/ToString behavior; the sink decides the wire format. Use an allow-listed safe projection, never an arbitrary DTO containing passwords, tokens or personal data.
• LogContext.PushProperty adds a property to every log event for the duration of the using block — perfect for request-scoped or operation-scoped properties.

JSON-configured Serilog (appsettings.json Serilog section) can support reloadable level controls when the configuration provider and Serilog.Settings.Configuration support reload. Not every sink or topology change reloads dynamically. Calling ReadFrom.Configuration is necessary; merely putting JSON in a file does not apply it.

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft.AspNetCore": "Warning",
        "System.Net.Http": "Warning"
      }
    },
    "Enrich": ["FromLogContext", "WithMachineName"],
    "WriteTo": [
      { "Name": "Console" },
      { "Name": "ApplicationInsights", "Args": { "telemetryConverter": "Serilog.Sinks.ApplicationInsights.TelemetryConverters.TraceTelemetryConverter, Serilog.Sinks.ApplicationInsights" } }
    ]
  }
}

ASP.NET Core logging abstraction: ILogger<T> and Serilog plug-in

ILogger<T> is the framework's logging abstraction. You take a dependency on ILogger<MyService> from DI and call log.LogInformation(...). The framework routes the call through whatever ILoggerProviders are registered.

When builder.Host.UseSerilog() runs, Serilog replaces the default providers with its own pipeline. From the consumer's side nothing changes — you keep using ILogger<T>; Serilog catches the call and applies templates, enrichers and sinks. This is why "switch from default logging to Serilog" is a one-line change for the entire codebase.

// Consumer: framework-style; Serilog routes behind the scenes
public class CheckoutHandler(ILogger<CheckoutHandler> log) { /* ... */ }

The corollary: do not call Log.Logger.Information(...) directly from your code. That couples you to Serilog at every call site. Stay on ILogger<T>; the abstraction is the point.

Application Insights data model

App Insights organises telemetry into a small, fixed set of tables:

• requests — collected inbound requests that survive instrumentation/export/sampling/filtering. Columns include name, url, duration, resultCode, success, operation_Id.
• dependencies — collected supported/manual outbound spans (not every SDK operation automatically). target, type, data, duration, success.
• exceptions — uncaught and TrackException-reported exceptions. type, outerMessage, details, severityLevel.
• traces — enabled ILogger/Serilog events after provider filters and sampling; classic Application Insights ILogger defaults commonly retain Warning+ unless overridden. message, severityLevel, customDimensions.
• customEvents — TrackEvent-emitted business events. name, customDimensions, customMeasurements.
• customMetrics — classic TrackMetric/preaggregated metric output, or mapped OpenTelemetry metrics through a configured reader/exporter. name, value, valueCount, valueSum, customDimensions.
• pageViews — browser-side telemetry (only with the App Insights JS SDK).

Request/dependency and correlated log/exception telemetry use correlation columns: operation_Id (the trace ID), operation_ParentId (the parent span ID). Trace ID groups related work; parent/span IDs reconstruct edges. Aggregated metrics and background events need not have a request parent. A join on trace ID can be many-to-many and is not a request counter.

Correlation: operation_Id, operation_ParentId, propagating across services

App Insights uses the W3C Trace Context where instrumented. An inbound request may have no traceparent, or an invalid one, and require a new context:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │   │                                │                │
              │   trace-id (32 hex)                parent-id (16)   flags
              version

When your service handles an inbound request, ASP.NET Core reads traceparent, sets the Activity.Current trace/span ids, and allows configured instrumentation to stamp correlated telemetry with operation_Id = traceId. When your service calls an outbound HTTP API via HttpClient, the HttpClient instrumentation injects a fresh traceparent with a new span id and the original trace id. The downstream service sees the same trace id, links itself as a child of your span, and so on.

var activity = Activity.Current;
log.LogInformation("Processing {TraceId} / {SpanId}", activity?.TraceId, activity?.SpanId);
activity?.SetTag("app.operation", "dashboard-query"); // bounded, non-secret context
activity?.SetTag("query.id", queryId);

A TraceId log property is not automatically the backend operation_Id column. Verify the sink's Activity correlation mapping; then join telemetry on the populated operation_Id and use parent/span IDs for edges. Merely writing a trace ID into message text does not establish that mapping.

Good enough to ship

• Verify Activity/trace propagation through the configured HttpClient and ASP.NET Core instrumentation; proxies, suppressed instrumentation and non-HTTP paths can break it. • Log operation_Id on every interesting line. • Use App Insights "Transaction search" to view a trace.

Expert tier

• Stitch in tracestate for vendor-specific context. • Carry trace context across non-HTTP boundaries (Service Bus, Storage). • Allow-list baggage; it can cross service/trust boundaries and must never contain secrets or substitute for authenticated tenant identity.

ITelemetryInitializer and ITelemetryProcessor

App Insights gives you two pipeline hooks:

• ITelemetryInitializer — runs before sampling, on every telemetry item, and can add or modify properties. Use it to attach context that should appear on every record (current user id, tenant id, deployment slot).
• ITelemetryProcessor — runs after initializers, in a chain, and can drop or transform items. Use it to filter (e.g. drop health-check requests, drop one chatty dependency), or to redact before export. Hashing a guessable email is pseudonymization, not reliable anonymization.

// Only attach approved, validated tenant context under your telemetry access/retention policy.
public class TenantInitializer(IHttpContextAccessor ctx) : ITelemetryInitializer
{
    public void Initialize(ITelemetry t)
    {
        var tid = ctx.HttpContext?.User.FindFirstValue("tid");
        if (tid is not null) t.Context.GlobalProperties["TenantId"] = tid;
    }
}
 
public class HealthCheckFilter(ITelemetryProcessor next) : ITelemetryProcessor
{
    public void Process(ITelemetry t)
    {
        if (t is RequestTelemetry r && r.Url?.AbsolutePath.StartsWith("/health/") == true) return;
        next.Process(t);
    }
}
 
builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<ITelemetryInitializer, TenantInitializer>();
builder.Services.AddApplicationInsightsTelemetryProcessor<HealthCheckFilter>();

Use an initializer for "always add this property"; a processor for "drop this whole class of records".

Adaptive sampling and blind spots

Classic adaptive sampling commonly targets five telemetry items/second per process, adjusting sampling percentage rather than imposing an exact five-record hard limit. Correlation-aware selection aims to retain a trace coherently, but independently configured services, different SDKs, item filters and export loss can still leave partial traces. Exceptions and rare events are not universally exempt; exclusion settings and pipeline ordering must be inspected.

• Stored versus represented counts: count() counts rows. For this classic schema, sum(itemCount) estimates represented traffic and sumif(itemCount, success == false) estimates failures. If one kept row has itemCount 20, count is one and its represented count is twenty.
• Distribution: percentile(duration,95) over retained rows is an observed-sample estimate. Weighting or preaggregated histograms need the correct schema and exporter contract; unequal sampling can distort a naive percentile.
• Rare dimensions: at a 10% independent keep rate, a tenant with one request can disappear from logs. Missing rows may also mean no traffic, filtering, exporter failure or ingest delay. Use an independent telemetry-availability signal.
• Processor ordering: initializers precede the processor chain; sampling is itself a processor in the classic SDK. Do not assert every custom processor always runs before every sampling stage. Inspect the assembled pipeline and avoid duplicate independently configured telemetry clients.

Keeping all checkout events but sampling other routes may be a deliberate policy. Record it in the query denominator and budget; never describe the resulting raw row counts as an unbiased census.

Live Metrics stream

Live Metrics provides a low-latency diagnostic stream while the pane is open. Selected metrics/counters are transmitted, but failures and stack traces are sampled; it is not an unsampled retained copy of every request. It helps inspect a release quickly without waiting for normal ingestion. It does not prove that absent evidence was sampled out rather than lost, and it is not an audit archive. Secure its control channel and compare collector/exporter health with the retained data path.

Writing KQL against App Insights tables

The five queries you will run more than any others:

// Slowest endpoints, p95 + error rate
requests
| where timestamp > ago(1h)
| summarize
    Count = sum(itemCount),
    observed_p95 = percentile(duration, 95),
    observed_p99 = percentile(duration, 99),
    ErrorRate = sumif(itemCount, success == false) * 100.0 / sum(itemCount)
  by name
| order by observed_p95 desc | take 20

// Dependency failures by target
dependencies
| where timestamp > ago(1h)
| where success == false
| summarize Failures = sum(itemCount) by target, type, resultCode
| order by Failures desc

// End-to-end trace for one operation_Id
let opId = "4bf92f3577b34da6a3ce929d0e0e4736"; // synthetic W3C trace ID
union withsource = Table requests, dependencies, traces, exceptions
| where timestamp > ago(1h) and operation_Id == opId
| project timestamp, Table, Name = coalesce(name, type, message), duration, resultCode
| order by timestamp asc

// Per-user timeline
let uid = "synthetic-approved-user-id";
union requests, exceptions, customEvents
| where timestamp > ago(1h) and user_AuthenticatedId == uid
| order by timestamp desc | take 200

// Request × trace join for correlated diagnostics
requests
| where timestamp > ago(1h) and success == false
| join kind=leftouter (
    traces | where timestamp > ago(1h) | project operation_Id, severityLevel, message
) on operation_Id
| project timestamp, name, duration, resultCode, severityLevel, message
| order by timestamp desc

The trace join may produce several rows per request, and a shared operation ID can include several request spans. Use request/span IDs to assign exact parentage; never count joined rows as requests. Restrict access to telemetry, redact URLs/messages and time-bound incident queries. Equivalent workspace tables use TimeGenerated, OperationId, ItemCount and DurationMs; inspect getschema rather than changing only the table name.

Alerts: metric vs log-search, action groups

App Insights supports two alert types:

• Metric alerts — operate on supported platform/application metric time series. Evaluation frequency, dimensions, cost and delay depend on signal and rule; do not promise sub-minute detection for every metric.
• Log-search alerts — operate on KQL queries with supported evaluation frequencies and windows. The scheduled-query service imposes KQL/operator and result-shape restrictions; not every interactive KQL query is valid. Ingestion delay also affects detection. Use when the alert needs joins, filters or custom thresholds.

An action group is the reusable definition of "what to do when an alert fires" — email, SMS, voice, webhook, Logic App, Function. Define them once per environment, attach to many alerts.

Six candidate signals to adapt to a service's SLO and traffic shape:

1. Live failure rate — error rate > 5% for 5 minutes (catches bad deploys).
2. p99 latency — p99 > 2× rolling baseline for 5 minutes (catches downstream degradation).
3. Dependency health — downstream failure rate > 1% for 5 minutes (catches DB/Kusto wobbles).
4. Resource saturation — CPU > 80% for 15 minutes (autoscale masks; alert surfaces).
5. No-traffic detection — request rate == 0 for 10 minutes (catches dead deploys).
6. Cost guard — autoscale at max instances for >30 minutes (capacity headroom is low; this does not itself prove a monetary budget violation).

Tune thresholds against weeks of baseline; an alert that fires three times a day during normal operation is worse than no alert at all.

Alert fatigue is the killer.

The biggest risk to your on-call rotation is not bad alerts; it is too many alerts. Every alert you add must answer "is a human better off being woken up by this than by the consequence". If no, do not add it. A small, actionable alert set is preferable to many duplicate pages; six is a teaching checklist, not a universal optimum.

OpenTelemetry: the converging standard

OpenTelemetry (OTel) is the vendor-neutral observability standard. Backends support different OTel signals, protocol versions and semantic mappings. Select a supported exporter and verify traces, logs and metrics separately; portability does not imply identical data or dashboards everywhere.

// Alternative to the classic SDK. Packages/namespaces:
// Azure.Monitor.OpenTelemetry.AspNetCore, OpenTelemetry.Resources
builder.Services.AddOpenTelemetry()
    .UseAzureMonitor()
    .ConfigureResource(r => r.AddService("queries-api", serviceVersion: "1.2.3"))
    .WithTracing(t => t.AddSource("queries-api"))
    .WithMetrics(m => m.AddMeter("queries-api", "polly-breaker"));

The distro supplies its supported instrumentation/exporters; use its documented configuration and APPLICATIONINSIGHTS_CONNECTION_STRING rather than installing a second independent classic collector alongside it. ActivitySource needs a listener/subscription; Meter needs a metric reader/exporter. For custom log export, use the distro's documented ILogger path or one consciously configured Serilog route, not duplicate sinks. Verify package-specific extensions when assembling a project.

Migration changes more than a service-registration line: compare operation names, resource/service labels, sampling, exception semantics, log levels, histogram aggregation, Live Metrics and custom telemetry support. Choose a cutover with a known fixture and compare counts/trace edges. Similar destination tables do not mean identical telemetry contracts.

Worked examples

Example 1 — Full Serilog + App Insights wiring

using Serilog;
using Serilog.Events;
using Serilog.Context;
 
var builder = WebApplication.CreateBuilder(args);
 
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<ITelemetryInitializer, TenantInitializer>();
builder.Services.AddApplicationInsightsTelemetryProcessor<HealthCheckFilter>();
builder.Host.UseSerilog((context, services, logger) => logger
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithProperty("Service", "queries-api")
    .Enrich.WithProperty("Env", context.HostingEnvironment.EnvironmentName)
    .WriteTo.Console(new Serilog.Formatting.Compact.CompactJsonFormatter())
    .WriteTo.ApplicationInsights(
        services.GetRequiredService<Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration>(),
        Serilog.Sinks.ApplicationInsights.TelemetryConverter.Traces));
 
var app = builder.Build();
 
app.Use(async (ctx, next) =>
{
    using var _ = LogContext.PushProperty("TraceId", Activity.Current?.TraceId.ToString());
    using var __ = LogContext.PushProperty("Operation", ctx.GetEndpoint()?.DisplayName ?? "unmatched");
    await next();
});
 
app.MapGet("/api/whoami", (ILogger<Program> log, ClaimsPrincipal user) =>
{
    log.LogInformation("whoami called"); // avoid logging a direct email identifier
    return new { name = user.Identity?.Name };
});
 
app.Run();

• This example uses code-only sink configuration to avoid duplicating Console/ApplicationInsights sinks from ReadFrom.Configuration. The classic sink explicitly reuses the DI TelemetryConfiguration, including the registered initializers/processors; do not substitute the connection-string overload and assume the same pipeline. In a configuration-driven variant define sinks only once.
• The middleware pushes TraceId and bounded operation context into LogContext; every downstream log line picks them up.
• TenantInitializer and HealthCheckFilter shape the App Insights pipeline.

Example 2 — Custom Meter for metrics

public static class QueryMetrics
{
    public static readonly Meter Meter = new("queries-api", "1.0");
    public static readonly Counter<long> Executed = Meter.CreateCounter<long>("queries.executed");
    public static readonly Histogram<double> Duration = Meter.CreateHistogram<double>("queries.duration_ms");
}
 
app.MapPost("/api/q", async (QueryDto dto, IQueryService svc) =>
{
    var sw = Stopwatch.StartNew();
    var kind = dto.Type is "dashboard" or "search" ? dto.Type : "other";
    try
    {
        var result = await svc.RunAsync(dto);
        QueryMetrics.Executed.Add(1, new("type", kind), new("status", "ok"));
        return Results.Ok(result);
    }
    catch
    {
        QueryMetrics.Executed.Add(1, new("type", kind), new("status", "error"));
        throw;
    }
    finally
    {
        QueryMetrics.Duration.Record(sw.Elapsed.TotalMilliseconds, new("type", kind));
    }
});

• Meter + Counter/Histogram is the .NET 6+ metrics API; exports through a configured .NET metric listener/OpenTelemetry reader; classic SDK registration alone does not subscribe to arbitrary Meters.
• Tags use an explicit finite allow-list. Raw request text, user/tenant IDs and arbitrary dto.Type values are not bounded metric dimensions.
• Configure a Meter subscription/exporter and verify its aggregation. The checked Azure Monitor documentation maps histograms to min/max/average/sum/count, not preserved quantile-capable buckets. Declaring a Meter does not export it automatically, and sum/count cannot recover p95. Another exporter may retain buckets, but that requires its own contract.

Example 3 — End-to-end incident KQL

let problemOp = "POST /api/checkout";
let lookback = 1h;
let bucket = 1m;
 
let problemRequests = materialize(
    requests
    | where timestamp > ago(lookback)
    | where name == problemOp
);
 
problemRequests
| summarize Total = sum(itemCount), Errors = sumif(itemCount, success == false), observed_p95 = percentile(duration, 95)
  by bin(timestamp, bucket)
| render timechart;
 
let problemTraces = problemRequests
| where success == false
| distinct operation_Id;
 
union
    (problemTraces | join kind=inner (traces | where timestamp > ago(lookback)) on operation_Id | project timestamp, operation_Id, severityLevel, message),
    (problemTraces | join kind=inner (exceptions | where timestamp > ago(lookback)) on operation_Id | project timestamp, operation_Id, type, outerMessage)
| order by timestamp desc
| take 100

• materialize caches the first sub-query because we reference it twice.
• First query produces the time-series chart; second walks just the failing operations through traces and exceptions.
• The predicates reduce eligible telemetry; no composite-index or query-performance result is asserted. Counts use classic itemCount weights; observed_p95 is explicitly the stored-sample percentile, not an unsampled latency guarantee.

Hands-on exercises

1. Serilog wiring. Wire Serilog + App Insights into a fresh minimal API. Log a business event with a structured template and LogContext.PushProperty. Confirm it lands in the traces table with the pushed properties as custom dimensions.

   • You are done when the property appears as a column in the KQL result.

2. Two-service trace. Stand up two services (A and B). Service A calls B over HTTP. Confirm a single operation_Id propagates through and that the App Insights Transaction Search shows the waterfall.

   • You are done when you can point to the parent-child relationship in the trace UI.

3. Custom Meter. Add queries.executed and queries.duration_ms per Example 2. Generate 100 requests and confirm they appear in the customMetrics table with the tags as dimensions.

   • You are done when counter totals and histogram min/max/sum/count (or actual supported aggregation) match the chosen exporter contract; do not expect bucket export from the documented Azure Monitor mapping. A sum/count metric row cannot recover an exact p95; use retained request durations or supported histogram quantile aggregation.

4. Five KQL queries. Write the five canonical queries from the "Writing KQL" section against your own telemetry. Pin them to an Azure Workbook dashboard.

   • You are done when you have measured query and display latency against a stated target and verified denominators, missing data and access scope.

5. Alert + action group. Configure a log-search alert: error rate > 5% for 5 minutes → email action group. Deliberately introduce a 500 in 10% of requests and wait for the alert to fire.

   • You are done when an authorized test action fires and you record detection plus delivery delay; a window alone is not a delivery-time guarantee.

6. OpenTelemetry migration. Replace AddApplicationInsightsTelemetry with the OTel block from the OTel section. Compare fixture counts, span parentage, resource labels, log levels, sampling and custom metric representation; do not run both automatic collectors simultaneously.

   • You are done when any semantic/schema differences are reconciled, not merely when charts look similar.

Self-check questions

1. Why message templates instead of string interpolation?
2. What's the difference between an ITelemetryInitializer and an ITelemetryProcessor?
3. Explain LogContext.PushProperty and the canonical use cases.
4. What does traceparent look like and which two ids does it carry?
5. How does adaptive sampling decide what to keep and what to drop?
6. Why is “adaptive sampling always keeps exceptions” false, and how would you verify an intentional exception-retention policy?
7. When do you reach for a metric and when for a log?
8. Walk through the six canonical alerts and the failure each catches.
9. What is Live Metrics good for that the regular dashboards are not?
10. Why is OTel the default for new services?
11. What's the difference between a metric alert and a log-search alert? Which is faster?
12. Why is alert fatigue worse than missing one alert?

SLOs, burn rate and worked assessment answers

Define an SLI as good events / eligible events over a stated window, excluding only deliberately specified traffic. For a 99.9% request-success objective, the allowed bad fraction is 0.001. If a recent window has 1% failures, burn rate is 0.01 / 0.001 = 10: the service consumes its allowed error fraction ten times as fast as budgeted. With one million eligible requests, the monthly budget is 1,000 bad requests; 600 failures have used 60%, not “600 minutes”. A time-based SLI needs a different denominator.

Use short and long windows together to detect sustained rapid burn without paging on one tiny low-volume spike. Separate no-traffic, no-telemetry and true zero-error outcomes. A collector outage should not draw a reassuring zero line; label unknown and alert on telemetry health.

1. Message templates keep a stable event pattern and named values, making numeric grouping/filtering possible without parsing strings. They do not sanitize values, DTOs or exception messages; use an allow-listed projection.
2. Classic ITelemetryInitializer enriches telemetry before the processor chain. ITelemetryProcessor can transform/drop records; sampling is itself a processor, so inspect ordering. Neither hook automatically controls a separate OTel pipeline.
3. LogContext.PushProperty scopes values in the execution context when FromLogContext is enabled. Use a bounded operation label or permitted trace ID, dispose nested scopes in reverse order, and avoid leaking identifiers into unrelated background work.
4. Version-00 traceparent is version, 32-lowercase-hex trace ID, 16-lowercase-hex parent ID, flags. All-zero IDs are invalid. The parent is the caller's span, not the downstream service's newly created span. Flags are tracing hints, not authentication or a promise of retained telemetry.
5. Classic adaptive sampling adjusts the retained fraction toward a target per-process rate; it is not a hard cap of exactly five records/second. Correlation-aware selection aims for coherent operations, but mismatched collectors, filters and export loss can fragment them.
6. “Always keeps exceptions” is false. Inspect the configured sampling exclusions and their pipeline placement, then send an approved known exception fixture and compare observed/represented counts. A missing exception can reflect filtering, sampling or export failure.
7. Metrics summarize rates/distributions with bounded dimensions; logs explain individual events; spans relate timed operations. Do not add arbitrary tenant IDs or query text as metric dimensions. A flattened histogram's sum/count gives a mean, not p95.
8. Failure-rate and latency signals detect user impact; dependency failures and resource saturation help diagnose it; no-traffic detection must distinguish quiet demand from outages; max-instance capacity indicates headroom, not a bill. Choose thresholds and ownership from the SLO rather than copying six pages unchanged.
9. Live Metrics is a low-latency, ephemeral diagnostic path with selected counters and sampled failure details. It helps inspect a release before normal ingestion, but is not an unsampled retained audit archive or proof the normal exporter works.
10. OTel separates instrumentation from exporter choice and is a useful default when the needed signals are supported. It is not universally interchangeable: compare backend schemas, exception mapping, sampling, labels and histogram aggregation before migration.
11. Metric alerts evaluate supported time series; log-search alerts evaluate supported scheduled queries with result/operator constraints. Actual detection/delivery depends on collection, ingestion, evaluation frequency/window and action delivery. Neither is always faster in every configuration.
12. Repeated nonactionable pages teach responders to ignore real incidents. Require an owner, action and tested delivery; consolidate duplicates. This does not mean missing one critical alert is always less harmful—judge user impact and blind spots explicitly.

Exercise 1 solution: Configure one Serilog sink route using the DI TelemetryConfiguration, emit a synthetic Operation property inside a disposed LogContext scope and query traces | extend Operation=tostring(customDimensions.Operation). If absent, check source/provider levels, FromLogContext, sink wiring, sampling and ingestion independently; never fix it by logging an entire DTO.

Exercise 2 solution: Emit one synthetic request A → B; verify equal trace IDs and distinct server/client span IDs, with B's parent equal to A's outgoing client span. Matching operation_Id alone is insufficient. Test missing/malformed incoming traceparent and a disabled-instrumentation hop; neither should be mistaken for authorization failure.

Exercise 3 solution: Subscribe to the exact queries-api Meter name, export a known 100-operation fixture and compare counter deltas, status partitions, duration count and sum after the export interval. Azure Monitor's checked histogram mapping flattens to min/max/average/sum/count; do not infer p95 from that row. Account for failed/cancelled operations in the chosen SLI denominator, and compare request durations separately if needed.

Exercise 4 solution: Run the five queries with bounded time/access scopes. For endpoint/failure totals, use itemCount weights; label retained-row percentiles as observed. A two-request/three-log trace can yield six join rows, so joined row count is not request count. The trace timeline needs span IDs for parentage; the user timeline uses only an approved synthetic identifier. Pin only after schema, denominator and missing-data behavior are understood.

Exercise 5 solution: For an approved fixture with 100 eligible requests and 10 failures, the unsampled rate is 10%; the threshold exceeds 5%. For sampled data use represented counts and monitor uncertainty/missing telemetry. Configure the supported query shape, evaluation/window and recipient-approved action group, then measure detection and delivery separately. Do not send test notifications to unsuspecting recipients.

Exercise 6 solution: Snapshot a fixed small request fixture under classic instrumentation, remove duplicate automatic collectors and configure the supported OTel distro. Compare represented counts, log severity, trace parentage, resource labels and metric aggregation—not raw row equality. Missing custom metrics first suggests a Meter subscription/export problem, not proof the operation never ran.

These are reference solutions and acceptance designs, not an executed collector/exporter or Azure test.

High-signal resources

Official docs

• Application Insights overview.
• Azure Monitor OpenTelemetry distro.
• Serilog documentation.
• W3C Trace Context spec.
• Adaptive sampling in App Insights.

Books or courses

• Observability Engineering — Charity Majors, Liz Fong-Jones and George Miranda; optional reading. The publisher page was not accessible for this review, so confirm this edition and author list before relying on the recommendation.
• Site Reliability Engineering + The Site Reliability Workbook — Google. The chapters on SLOs and monitoring are essential.

Practitioner posts

• charity.wtf — optional practitioner reading; evaluate each dated post independently rather than treating this index as evidence for a particular technical claim.
• Honeycomb engineering blog — practical observability deep-dives.
• Azure SDK blog — a broader SDK publication, not exclusively the App Insights team; check individual author, package version and migration scope.

Weekly milestones

1. Day 1. Read the App Insights overview + the W3C Trace Context spec. Do exercise 1. Self-check questions 1–3.
2. Day 2. Distributed tracing (exercise 2). Self-check questions 4 + 9.
3. Day 3. Custom metrics (exercise 3). Self-check question 7.
4. Day 4-5. KQL queries + dashboards (exercise 4). Self-check questions 5–6 + 11.
5. Day 6-7. Alerts (exercise 5) and OTel migration (exercise 6). Self-check questions 8 + 10 + 12.

How it shows up in the capstone

The capstone design chooses one supported log-export route plus OpenTelemetry → App Insights. An incoming request may lack traceparent; instrumented code creates/continues an Activity, and explicit enrichers bridge its trace ID to logs. A Kusto ClientRequestId or SQL tag is diagnostic context, not automatically a W3C parent-child edge. Custom Meters are queries.executed, queries.duration_ms, cache.hits. Configure and test the chosen action-group/Teams integration separately; no notification delivery is demonstrated here.

A proposed Azure Workbook dashboard groups six teaching tiles (live health, request rate by op, p50/p95/p99, dependency health, exception clusters, infra utilisation). Choose the OTel sampler deliberately; the classic adaptive sampler/ITelemetryProcessor example is not silently reused in the distro. A health filter can remove request rows without removing all associated telemetry or all billing. Pre-prod telemetry goes to a separate App Insights resource so capacity planning is honest.

Previous chapter → Ch 14 — Identity for cloud apps Next chapter → Ch 16 — Resilience patterns

Corrected contracts and failure analysis

Classic sampled event counts need sum(itemCount), not just count(). An observed sampled latency distribution is not automatically an unbiased unsampled percentile; use the documented aggregation/weights for the schema and collect preaggregated metrics where suitable. Sampling and ingestion filtering can remove evidence, so absence of an event is not proof a request never occurred.

Correlation IDs link spans; they are not authentication. Never place tokens, request bodies or raw tenant secrets in baggage or metric labels. High-cardinality dimensions can overwhelm a metric backend. Structured logging improves queryability but does not redact exception messages automatically. Choose either a documented OTel-based integration or the classic SDK path deliberately; duplicate instrumentation can emit duplicate requests and inflate apparent traffic.

Boundary exercise with solution

A sampled request row has itemCount=20. Does count() return twenty, and does a missing trace prove no request happened?

Source-backed review notes

• Telemetry sampling in Azure Application Insights - Azure Monitor | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “If you write Analytics queries, you should take account of sampling. In particular, instead of simply counting records, you should use summarize sum(itemCount).”
• Logging in C# - .NET | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “The logger methods have overloads that take an exception parameter:”

Versioned source contracts

Official passages were checked on 7 October 2026; immutable tags are used where available, while service docs remain dated snapshots. The saved evidence records include exact passages, URL, source date/revision and snapshot hash. These references support the stated contracts, not an unperformed service deployment.

• sampling — Classic sampled counts need represented-event weighting, not count().
• live-metrics — Live Metrics is not an unsampled retained copy of every event.
• otel-enable — Azure Monitor distro is a supported distinct instrumentation path.
• otel-add — Custom Meters require subscription/export configuration.
• serilog8 — Serilog can use DI-aware configuration; avoid independent duplicated pipelines.

Additional contract references

Checked 9 October 2026 as dated official snapshots or the explicit source tags below; these support the stated mechanisms, not a live-service reproduction.

• Trace Context — trace-context.
• Add and Modify OpenTelemetry in Application Insights - Azure Monitor | Microsoft Learn — otel-metrics.
• serilog-config — serilog-config.
• serilog-sink — serilog-sink.
• Google SRE - Prometheus Alerting: Turn SLOs into Alerts — slo.