MS Stack Ch 16 — Resilience patterns with Polly v8

Pipeline and version contract

This chapter uses the Polly v8 strategy API and Microsoft.Extensions.Http.Resilience 8.0.0 integration; defaults are distinguished from the explicitly chosen examples. Strategies are source-reviewed against official Polly documentation and the HTTP options source. No .NET pipeline, remote service, chaos test or retry timing benchmark was executed. Local state/interleaving models verify arithmetic and safety invariants, not cancellation timing or distributed exactly-once behavior.

Chapter 16 of From Novice to Fluent on the Modern Microsoft Web Stack — a 22-chapter self-study plan.

Why this chapter

Production systems fail. Networks blip, downstreams degrade, queues back up, certificates expire, the SQL connection pool exhausts, a noisy neighbour starves your CPU, an EU region falls offline at 03:14 UTC because someone in another team rolled out a bad config. The difference between a 30-second blip and a 30-minute outage is whether your callers handle failure with grace. The difference between a 30-minute outage and a 30-hour incident is whether your retries are bounded, whether your timeouts are tight, whether your circuit breaker is configured at all.

Shipping-level resilience means: every outbound call has a timeout, every retry has jitter, every circuit breaker has a MinimumThroughput, and you never retry a mutation without a demonstrated repeatability or durable idempotency contract. Expert-level resilience means: you reason about your service's failure mode taxonomy, you size timeouts against percentiles rather than guesses, you have run a chaos exercise that deliberately fails the downstream you most rely on, you can justify why this lesson restricts hedging to safe repeatable reads, and you know exactly which Polly exception means "the breaker is open" versus "the request itself timed out".

You finish this chapter when you can build a complete resilience pipeline from memory, defend the ordering of every layer, and explain the failure mode each one catches.

Classify failures — Transient vs persistent; retriable vs not; surface the cause.

Bound everything — Max attempts, max delay, sampling windows, break durations — nothing unbounded.

Compose deliberately — Outer-to-inner pipeline ordering you can defend at code review.

Idempotency-first — Retry mutations only with a demonstrated repeatability/idempotency contract, not merely a header.

Pick the right pattern — Timeout / retry / breaker / bulkhead / hedge / fallback — each for its purpose.

Wire via HttpClient — AddResilienceHandler applies to calls through that registered client, not every SDK or database call.

Concepts and depth

Failure taxonomy: transient vs persistent, retriable vs not

The first decision in any failure is "do I retry?". The classifier has two axes:

• Transient — the same call again has a real chance of succeeding (a connection reset, a 503, a 429). The downstream is healthy or recovering.
• Persistent — the same call again will fail the same way (a 400, a 404, a 401, a SQL constraint violation). The state, the request or the auth is wrong.

Cross with:

• Retriable — the operation is safe to attempt again (idempotent GETs, idempotent PUTs with If-Match, POSTs with a correctly enforced durable idempotency protocol).
• Non-retriable — repeating it would do harm (a POST that charges a card, a SendEmail call, an unscoped UPDATE).

Only (transient AND retriable) belongs in a retry policy. Everything else fails fast and surfaces the real cause. Retrying a 400 is wasted budget; retrying a non-idempotent POST is a double-charge incident waiting to happen.

Candidate retry statuses (not proof that repeating the operation is safe): 408 (Request Timeout), 429 (Too Many Requests; honour Retry-After), 500 (Internal Server Error), 502 (Bad Gateway), 503 (Service Unavailable), 504 (Gateway Timeout). Some network faults are transient, but a certificate-validation/TLS configuration failure is usually persistent: do not bypass validation or retry blindly. Caller cancellation is not a transient timeout. Honor the operation's deadline and known side-effect semantics.

Good enough to ship

• Retry 5xx + 408 + 429 only. • An idempotency-key header is insufficient: require durable ownership/replay semantics before retrying a mutation. • Surface the original exception in logs.

Expert tier

• Per-endpoint retry budgets based on actual failure-rate baselines. • Read-vs-write retry policies live in code; reviewable. • Reconcile durable effect records under concurrent retries and crashes; replay logs alone cannot prove no duplicate effect occurred.

Retry: max attempts, backoff strategies, jitter

A retry policy is parameterised by:

• Retry count — Polly MaxRetryAttempts counts retries after the original: 3 allows up to 4 total executions. Pick from the latency/capacity budget, not a universal fast/slow default.
• Backoff strategy — how the delay grows between attempts: constant, linear, exponential.
• Base delay — the unit of delay (Delay = TimeSpan.FromSeconds(1)).
• Jitter — random spread to avoid synchronised herd retries.

Backoff strategies:

• Constant — same delay every time. Use only for very fast operations.
• Linear — delay grows linearly (1s, 2s, 3s). Modest backoff.
• Exponential — delay doubles (1s, 2s, 4s, 8s). The standard choice.

Jitter is the single most important knob. Without it, every client that hit the failing downstream waits the same fixed delay and retries at the same instant. The recovering downstream is hit with a synchronised herd of retries that pushes it back into failure. With jitter (Polly's UseJitter = true), delays follow the selected strategy's distribution (Polly's exponential jitter uses DecorrelatedJitterBackoffV2, not a universal 0.5×–1.5× multiplier) and the herd spreads out across the window. It reduces synchronization; the resulting recovery time still depends on offered load and downstream capacity.

.AddRetry(new HttpRetryStrategyOptions
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .Handle<TimeoutRejectedException>()
        .HandleResult(r => (int)r.StatusCode is >= 500 or 408 or 429),
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Exponential,
    UseJitter = true,
    OnRetry = args =>
    {
        log.LogWarning("Retry {Attempt}; status {Status}", args.AttemptNumber, args.Outcome.Result?.StatusCode);
        // Polly disposes discarded retry results after OnRetry; inspect here, do not retain them.
        return default;
    }
})

This HTTP-specific options type honors a valid Retry-After header by default in the pinned integration; the generic RetryStrategyOptions does not. Use this fragment only for an operation already known to be safe to repeat. Its broad HttpRequestException match is a starting point, not a proof that every transport error is transient: exclude known persistent certificate/configuration failures and never bypass TLS validation.

Worked synchronization example: 100 callers fail together and all wait exactly one second. The next attempt is another burst of 100. Jitter spreads those retries, but if each call permits four attempts, total offered work can still approach 400 executions. Add admission control and a total deadline; random delay is not a capacity plan.

Timeout: overall vs per-attempt, propagating cancellation

A timeout cancels an operation that takes too long. Two flavours:

• Per-attempt timeout — each individual retry attempt has its own deadline.
• Overall timeout — the entire pipeline (all retries + waits + work) has a deadline.

Use an overall budget derived from the caller's deadline and a smaller per-attempt budget. The overall budget may intentionally prevent all configured retries; it need not exceed the sum of every attempt and backoff. Without an overall timeout, three retries with a 5-second per-attempt timeout plus exponential backoff can easily eat 30 seconds — by which point your caller has already given up.

new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddTimeout(TimeSpan.FromSeconds(30)) // overall
    .AddRetry(/* ... */)
    .AddTimeout(TimeSpan.FromSeconds(5))  // per-attempt
    .Build();

Polly timeouts are cooperative: they signal via CancellationToken. Your inner code must honour the token. Many BCL/SDK APIs accept cancellation, but support and abort behavior vary by operation/provider; passing a token does not prove a remote side effect stopped. Work that ignores the token may continue and delay completion of Polly's timeout path; Polly waits for the callback rather than forcibly terminating it. Hand-rolled Task.Run work is not made cancellable merely by passing a token to the outer pipeline.

The right wiring inside your service: take a CancellationToken parameter on every async method, pass it down to every awaited call, propagate to HttpClient.SendAsync(req, ct) and friends.

Circuit breaker: closed, open, half-open

The circuit breaker is the "stop knocking on a broken door" pattern. It has three states:

• Closed — all calls pass through. Failures are counted in a sliding window. If FailureRatio is met or exceeded over MinimumThroughput calls in SamplingDuration, the breaker trips.
• Open — all calls fail immediately with BrokenCircuitException. Saves both your service (no wasted threads) and the downstream (no wasted load while it recovers). After BreakDuration elapses, the breaker moves to Half-Open.
• Half-Open — the next single call is a probe. If it succeeds the breaker returns to Closed; if it fails the breaker returns to Open for another BreakDuration.

.AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .Handle<TimeoutRejectedException>()
        .HandleResult(r => (int)r.StatusCode >= 500),
    FailureRatio = 0.5,
    MinimumThroughput = 10,
    SamplingDuration = TimeSpan.FromSeconds(30),
    BreakDuration = TimeSpan.FromSeconds(30),
    OnOpened = args => { log.LogError("Breaker opened; status {Status}; error type {ErrorType}", args.Outcome.Result?.StatusCode, args.Outcome.Exception?.GetType().Name); return default; },
    OnClosed = args => { log.LogInformation("Breaker closed"); return default; },
    OnHalfOpened = args => { log.LogInformation("Breaker half-open"); return default; }
})

When the breaker helps and when it hurts:

• Helps when the downstream is overloaded; fail-fast removes the load and lets it recover.
• Hurts when the downstream is having a brief blip; opening the breaker turns a 2-second blip into a 30-second outage. Tune MinimumThroughput and SamplingDuration so a single failure cannot trip the breaker.

MinimumThroughput is the safety: the breaker will not trip without at least N calls in the window, so a single early failure during low traffic does not cause cascading damage.

Good enough to ship

• Example choice (not library defaults): FailureRatio=0.5, MinimumThroughput=10, Sampling=30s, Break=30s. • Combine with fallback so callers see graceful degradation. • Log every state transition.

Expert tier

• Tune from production baselines, not from defaults. • Per-endpoint or per-tenant breakers. • A shared overload signal requires a separate coordination design; Polly does not acquire a distributed breaker merely by adding Redis.

Bulkhead: isolating resource pools

The bulkhead pattern (named after ship compartments) caps admitted concurrent operations against a resource so a slow downstream cannot accumulate unbounded outstanding work in that pool. Polly v8 calls it the concurrency limiter:

.AddConcurrencyLimiter(new ConcurrencyLimiterOptions
{
    PermitLimit = 50,  // max 50 concurrent
    QueueLimit = 10    // queue up to 10 more; reject the rest
})

Async HTTP waits do not reserve one thread per request. The resources at risk are outstanding work, sockets/connections, queued continuations, memory and downstream capacity. A 50-permit limiter bounds admitted operations in this pipeline instance; it does not reserve 50 threads or enforce a fleet-wide cap. QueueLimit 10 adds waiting work; a total deadline should include that wait. With four replicas, 50 permits each can admit 200 calls unless a separate global budget exists.

Bulkhead and rate limiter are different. Bulkhead caps in-flight (concurrency); rate limiter caps issuance rate (calls per second). Use bulkhead to isolate, rate limiter to throttle.

Hedging: racing duplicate requests for tail-latency

Hedging issues a second (and optionally third) request to the same endpoint after a short delay, taking the first acceptable unhandled outcome, not necessarily the first completed failure. It is the tail-latency mitigation pattern: when a small fraction of calls hang for unknown reasons, racing a parallel attempt usually catches the fast tail.

.AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .Handle<TimeoutRejectedException>()
        .HandleResult(r => (int)r.StatusCode is >= 500 or 408 or 429),
    MaxHedgedAttempts = 1, // one extra execution; two total
    Delay = TimeSpan.FromMilliseconds(300),
    ActionGenerator = args => () => args.Callback(args.ActionContext)
})

The predicate defines an acceptable outcome: without an HTTP result predicate, a completed 503 is an unhandled result and can win. The example permits one extra execution, not two extra executions. If all attempts fail, preserve the documented failure outcome rather than manufacturing success.

This lesson restricts hedging to safe repeatable reads. An intrinsically idempotent write can theoretically be hedged under a stronger protocol, but a header alone is not that protocol. Hedging a POST means you might hit the downstream twice with side effects; a side-effect-free search may be eligible only if its capacity, permissions and cancellation contract are suitable. It is most promising for fast-when-healthy calls with variable tail latency and independent spare capacity; a slow shared backend can make both attempts slow. No fixed five-second duration proves benefit or harm.

The cost: extra downstream load. If exactly 1% of calls launch one extra attempt and no other call is hedged, attempt volume grows by 1%. A handled failure can trigger another attempt before the delay, so the slow-call fraction alone does not determine the hedge rate. Measure whether the tail-latency benefit justifies that work; no order-of-magnitude improvement follows from the 1% assumption.

Fallback: safe defaults, subtle-wrong is worse than obviously-wrong

When everything else in the pipeline fails, fallback substitutes a safe response so the caller sees a graceful degradation rather than an exception.

.AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<BrokenCircuitException>()
        .Handle<TimeoutRejectedException>()
        .Handle<HttpRequestException>()
        .HandleResult(r => (int)r.StatusCode >= 500),
    FallbackAction = args =>
    {
        args.Outcome.Result?.Dispose();
        // Application helper returns a NEW response and verifies tenant, age and permission.
        return Outcome.FromResultAsValueTask(GetAuthorizedCachedResponse());
    }
})

A fallback must not convert caller cancellation, authorization errors or programming defects into an apparent success. An empty list can be as misleading as stale data: it asserts “there are no records”. Return explicitly labelled stale data with age/source metadata only when the business contract allows it; otherwise return a clear unavailable response. Never synthesize a cached authorization decision or payment success.

Put fallback outside the strategies whose failures it should handle. Its own retrieval must be bounded too: an outermost unbounded cache fallback can outlive the inner overall timeout. Reserve part of the caller budget for fallback or place a final request deadline around the entire composition.

Rate limiting and throttling

Server-side rate limiting protects your service from being overwhelmed; client-side rate limiting prevents you from being a noisy neighbour. Both belong in the pipeline:

// Create ONE shared limiter with the service/pipeline, not a fresh limiter per call.
// Its owner disposes it only after all in-flight work has stopped.
var limiter = new FixedWindowRateLimiter(new FixedWindowRateLimiterOptions
{
    PermitLimit = 100,
    Window = TimeSpan.FromSeconds(1),
    QueueLimit = 0,
    QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
    AutoReplenishment = true,
});
var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRateLimiter(new RateLimiterStrategyOptions
    {
        RateLimiter = args => limiter.AcquireAsync(1, args.Context.CancellationToken),
    })
    .Build();

Creating a limiter inside the per-call delegate resets its budget on every request and leaks its lifetime; that original example imposed no shared rate bound. The limiter above is process-local. A shared service quota needs budget partitioning or a supported distributed enforcement point.

The .NET 7+ rate limiter primitives include FixedWindow, SlidingWindow, TokenBucket and Concurrency limiters. Pick:

• TokenBucket for "average rate X, burst up to Y" — the most common shape.
• FixedWindow for "at most N permits in each aligned window" — not an exact rate or any rolling-minute bound. N just before a reset and N just after can create a 2N boundary burst.
• SlidingWindow for segmented rolling accounting — smoother than fixed windows, but the .NET segmented implementation is not an exact timestamp log of every possible 60-second interval.

When the downstream returns 429 with Retry-After, honour the header — respect its minimum delay within your overall deadline; if the delay exceeds the remaining budget, fail/defer rather than retry early. Generic Polly RetryStrategyOptions does not parse HTTP headers automatically; HttpRetryStrategyOptions supplies its HTTP-specific Retry-After behavior.

Polly v8: ResiliencePipelineBuilder, strategy ordering, options, AddResilienceHandler

Polly v8 redesigned the API around ResiliencePipelineBuilder. The mental model:

• A pipeline is an ordered chain of strategies.
• Each strategy is strongly-typed options (RetryStrategyOptions<T>, CircuitBreakerStrategyOptions<T>).
• Strategies are added with .Add<Strategy>(options) and the order matters: outer strategies wrap inner ones.

Outer-to-inner ordering:

1. Fallback — outside selected failures; do not swallow cancellation or authorization errors.
2. Overall timeout — budgets the inner operation (admission, attempts and waits); an outer fallback is outside it and still needs a caller/final deadline.
3. Bulkhead / rate limiter — admission control before doing real work.
4. Retry — the work that runs N times.
5. Circuit breaker — protects the actual call from being made when the downstream is broken.
6. Per-attempt timeout — innermost; caps each individual attempt.
7. The action — the HTTP/SQL/Kusto call.

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddFallback(/* ... */)
    .AddTimeout(TimeSpan.FromSeconds(30)) // overall
    .AddConcurrencyLimiter(new() { PermitLimit = 50 })
    .AddRetry(/* ... */)
    .AddCircuitBreaker(/* ... */)
    .AddTimeout(TimeSpan.FromSeconds(5))  // per-attempt
    .Build();

For HttpClient consumers, the canonical wiring is AddResilienceHandler on HttpClientFactory:

builder.Services.AddHttpClient<IUpstreamClient, UpstreamClient>()
    .AddResilienceHandler("upstream", b =>
    {
        var retry = new HttpRetryStrategyOptions { MaxRetryAttempts = 3 };
        // Pinned 8.0.0 example: retry only explicit GET response failures.
        // Current package releases also offer DisableForUnsafeHttpMethods;
        // do not backport that newer helper into 8.0.0.
        var defaultPredicate = retry.ShouldHandle;
        retry.ShouldHandle = async args =>
            args.Outcome.Result?.RequestMessage?.Method == HttpMethod.Get
            && await defaultPredicate(args);
        b.AddRetry(retry);
        b.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5, MinimumThroughput = 10,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(30)
        });
        b.AddTimeout(TimeSpan.FromSeconds(10)); // per-attempt; caller/outer budget still required
    });

Every outbound call through the registered HttpClient inherits the policy without per-call wrapping.

Exception semantics: TimeoutRejectedException, BrokenCircuitException, surfacing the cause

Polly throws specific exceptions when its strategies fail:

• TimeoutRejectedException — a timeout strategy elapsed.
• BrokenCircuitException — the breaker was Open when a call attempted to pass through.
• RateLimiterRejectedException — the rate limiter denied admission.

These are Polly exceptions, not the underlying cause. When you catch them, log both the Polly exception type and the original cause (where available in recorded breaker transition telemetry and an inner exception; a result-triggered opening may have no inner exception. RetryAfter is a delay hint, not the original cause). Without surfacing the original cause, your incident response sees "BrokenCircuitException" 1000 times and never learns why the breaker opened.

try
{
    using var result = await pipeline.ExecuteAsync(async ct => await client.GetAsync(url, ct));
}
catch (BrokenCircuitException ex)
{
    log.LogError("Breaker open for {Downstream}; cause type {InnerType}",
        "pricing", ex.InnerException?.GetType().Name); // sanitize detailed exceptions separately
    return Results.StatusCode(503);
}
catch (TimeoutRejectedException ex)
{
    log.LogWarning("Timeout on {Downstream}", "pricing");
    return Results.StatusCode(504);
}

Worked examples

Example 1 — Full HttpClient pipeline

builder.Services.AddHttpClient<IPricingClient, PricingClient>(c =>
{
    c.BaseAddress = new Uri("https://pricing.internal/");
    c.DefaultRequestHeaders.UserAgent.ParseAdd("queries-api/1.0");
    c.Timeout = Timeout.InfiniteTimeSpan; // pipeline/caller owns the time budget
})
.AddResilienceHandler("pricing", b =>
{
    b.AddTimeout(TimeSpan.FromSeconds(30)); // overall budget
 
    var retry = new HttpRetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromSeconds(1),
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        OnRetry = args =>
        {
            logger.LogWarning(
                "pricing retry {Attempt} after {Outcome}",
                args.AttemptNumber, args.Outcome.Result?.StatusCode);
            return default;
        }
    };
    // Pinned 8.0.0 example: retry only explicit GET response failures.
        // Current package releases also offer DisableForUnsafeHttpMethods;
        // do not backport that newer helper into 8.0.0.
        var defaultPredicate = retry.ShouldHandle;
        retry.ShouldHandle = async args =>
            args.Outcome.Result?.RequestMessage?.Method == HttpMethod.Get
            && await defaultPredicate(args);
    b.AddRetry(retry);
 
    b.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        MinimumThroughput = 10,
        SamplingDuration = TimeSpan.FromSeconds(30),
        BreakDuration = TimeSpan.FromSeconds(30)
    });
 
    b.AddTimeout(TimeSpan.FromSeconds(5)); // per-attempt
});

• Overall + per-attempt timeouts, with retry + breaker between.
• HTTP retry handles the documented transient statuses/exceptions; this pinned example admits only GET response failures. Exceptions with no response (including connection/timeout failures) are deliberately not retried by this predicate; they still fail through the timeout/breaker path. A newer package’s unsafe-method helper supports a broader documented HTTP policy. Do not infer repeatability from a POST header. Own only one retry layer: SDK retries plus this handler can multiply attempts.
• Logging callback emits a warning per retry so you see them in App Insights.

Example 2 — Idempotent POST with idempotency key

The previous distributed-cache middleware was unsafe: a get-then-set race allowed duplicate effects, it omitted tenant/request digest, lost status/headers on replay and did not restore the response stream after exceptions. Do not reuse it as a POST safety mechanism.

Use a durable state machine instead:

key = (tenant, authorized_principal, operation, caller_idempotency_key)
request_hash = hash(canonical validated request)
atomic unique insert(key, request_hash, state=in_progress)
  existing different hash -> conflict
  existing complete -> original status + safe headers + body
  existing in_progress -> documented retry/status response
transaction owner:
  apply the database mutation
  persist the response and complete the key
  commit together

If the effect is outside that transaction, use the downstream system's idempotency protocol or a durable outbox and reconciliation. A worker crash between an external charge and recording completion is an ambiguous outcome, not permission to charge again. Authorization runs before looking up a prior response. Expiration must exceed the supported client retry horizon; reusing expired keys can repeat effects.

Exercise: interleave two callers at the insert and crash the winner after mutation. Solution: the unique insert chooses one owner; an atomic transaction either rolls back mutation plus record or commits both. An external side effect needs its own reconciliation and cannot be solved by response caching alone.

Example 3 — Hedging on a search endpoint

Use hedging only for safe repeatable reads with independent capacity. Each speculative attempt needs its own request/message instance, its own cancellation path and a measured concurrency budget. The v8 ActionGenerator returns a zero-argument delegate invoking args.Callback(args.ActionContext), as shown earlier. The action itself must allocate independent request state; a shared HttpRequestMessage cannot simply be sent concurrently.

A correct design nests overall timeout outside hedging and per-attempt timeout inside. If A starts at 0 ms, B at 300 ms and B succeeds at 450 ms, choose B if its outcome is accepted and signal cancellation to A. Polly waits for pending hedged actions to finish cancellation before returning the accepted result, so selecting B at 450 ms does not guarantee a 450 ms caller response. Cooperative cancellation may not stop remote work; record both attempts in telemetry. If both share a saturated backend, hedging may make tail latency worse. Exercise: choose a maximum extra-attempt budget for 100 requests/s. Solution: one hedge per request permits up to 200 attempts/s before retries, which must fit downstream capacity; hedging and retries multiply unless one layer owns the combined budget.

Example 4 — Breaker state to App Insights

public static class BreakerMetrics
{
    public static readonly Meter Meter = new("polly-breaker", "1.0");
    public static readonly Counter<long> Opened = Meter.CreateCounter<long>("breaker.opened");
    public static readonly Counter<long> Closed = Meter.CreateCounter<long>("breaker.closed");
}
 
.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
    /* thresholds */
    OnOpened = args =>
    {
        BreakerMetrics.Opened.Add(1, new("downstream", "pricing"));
        return default;
    },
    OnClosed = args =>
    {
        BreakerMetrics.Closed.Add(1, new("downstream", "pricing"));
        return default;
    }
})

• Records an increment on open and closed transitions; the shown code does not record half-open transitions, and collection/export still needs a subscribed Meter.
• Subscribe/export the polly-breaker Meter, then correlate breaker transitions with traffic and user-impact SLIs. Paging on every opening can create noise; an open counter is not a universal pre-user outage detector.

Hands-on exercises

1. Flaky-server pipeline. Stand up a test server that returns 503 for 50% of calls. Wrap a client with timeout + retry + circuit breaker. Run 1000 requests through and verify each pattern triggers.

   • You are done when deterministic failure/recovery fixtures exercise retry, open rejection and a successful half-open probe, alongside the random trial; random 50% failures alone do not guarantee every transition.

2. Jitter demonstration. Run 100 clients concurrently against a downstream that fails for 5 seconds then recovers. Compare median recovery time with and without UseJitter = true.

   • You are done when you compare attempt-time histograms and load across repeated trials, explain any synchronized bursts and report the observed outcome without requiring jitter to win every run.

3. Idempotent POST. Implement Example 2's durable state machine on a POST endpoint that creates a row. Retry the POST three times with the same Idempotency-Key. Confirm only one row exists.

   • You are done when sequential and concurrent same-key retries produce one committed effect, differing digests conflict, caller scope is enforced, and crash/expiry cases follow the documented protocol.

4. Fallback chain. Add an outer Fallback layer that returns a stale cached response when the breaker is Open. Force the breaker open. Confirm callers see the cached response with a "served-from-fallback" header.

   • You are done when eligible reads get a labelled, age-bounded fallback and ineligible/unauthorized/too-stale cases fail explicitly.

5. Hedging tuning. Add hedging at 300ms to a read endpoint. Inject a 1-second pause on 20% of calls. Measure p99 before and after.

   • You are done when p99, total attempts, cancellation/disposal and downstream saturation are measured against the same workload; reject hedging if the extra load has no justified benefit.

6. Wire with HttpClientFactory. Migrate any hand-rolled pipeline.Execute(...) calls to AddResilienceHandler and consume via typed HttpClient.

   • You are done when no business code directly references the pipeline.

Self-check questions

1. Give the four-cell transient × retriable matrix and an example of each.
2. Why is jitter essential, and what is the failure mode without it?
3. Explain the three circuit-breaker states and the transition triggers.
4. Why is MinimumThroughput important on a circuit breaker?
5. What is the difference between an overall timeout and a per-attempt timeout?
6. When is hedging worth it and when is it harmful?
7. Which failures should fallback wrap, and how is fallback work itself bounded?
8. Explain the difference between bulkhead and rate limiter.
9. Walk through the canonical outer-to-inner Polly pipeline ordering.
10. Why is BrokenCircuitException insufficient to diagnose an incident on its own?
11. What does an idempotency key identify, and why is a durable protocol—not just the key—needed to safely retry an otherwise non-repeatable POST?
12. When does retrying make an outage worse instead of better?

Worked budget and state-machine answers

With MaxRetryAttempts 3, per-attempt timeout 5s and non-jittered delays 1s, 2s, 4s, the worst planned path is 4 × 5 + 1 + 2 + 4 = 27s, excluding overhead/queue time. A 12s overall budget deliberately prevents completing that plan. Caller cancellation stops admission and signals in-flight work; it does not prove a timed-out charge never happened. Two layers each permitting four attempts can make 16 downstream attempts, before hedging.

A breaker with MinimumThroughput 10 and FailureRatio 0.5 does not open on four failures among eight executions: throughput is too low. At five failures among ten eligible executions in the window, the threshold is met. The retry-inside/outside choice changes whether the breaker observes attempts or final operations; the shown retry-outside-breaker arrangement observes attempts. Reuse the pipeline so state persists across calls; creating it per request prevents useful history.

1. Transient/safe: repeatable GET with temporary 503. Transient/unsafe: an ambiguous charge response without a durable key protocol. Persistent/safe: malformed filter returning 400. Persistent/unsafe: repeating an invalid state-changing request. Retry only the bounded transient-safe cell after considering the actual endpoint contract.
2. Jitter disperses retry start times and reduces synchronized bursts; it does not reduce the maximum number of attempts or create capacity. Compare time histograms and recovery across trials, not a universal latency improvement claim.
3. Closed admits/counts eligible outcomes; open rejects with BrokenCircuitException; after the break interval an admitted half-open probe tests recovery. An accepted probe closes and a handled failure reopens. Sharing the pipeline preserves state; constructing a new one per call does not.
4. MinimumThroughput prevents a high ratio among too few calls from opening the circuit. Four failures among eight do not meet a threshold of ten; five failures among ten meet a 0.5 ratio. The selected failure predicate and window still matter.
5. An outer timeout budgets inner admission, retries and waits; an inner timeout applies to one attempt. Fallback outside that timeout needs a separate remaining budget. Cancellation is cooperative and cannot establish whether a remote side effect already happened.
6. Hedging can help safe repeatable reads with rare slow attempts and independent capacity. It hurts when duplicate work competes at the same bottleneck. One extra hedge permits two total executions; inspect outcome predicates, cancellation and losing-result cleanup.
7. Put fallback outside only the failures it should handle. Use a labelled age-bounded authorized cached read or explicit unavailable result, not fabricated empty/payment success. Do not swallow caller cancellation or authorization errors. Bound fallback itself with the caller/final deadline.
8. A concurrency limiter caps admitted in-flight operations per shared instance; a temporal limiter caps permits over a specified window. Neither is a fleet-wide quota without coordination. Fixed-window limits permit bursts across a boundary and do not promise exactly N calls.
9. The shown order is fallback → inner overall timeout → admission → retry → breaker → per-attempt timeout → action. Retry outside breaker means the breaker sees attempts. Admission outside retry holds a permit across waits and limits logical calls, not separately every attempt's rate; move/add attempt-level throttling when that is the actual quota.
10. BrokenCircuitException explains current rejection, not necessarily why the circuit opened. Record sanitized original status/error type at the transition; a result-triggered opening may have no inner exception. Avoid logging whole response objects or credentials.
11. A key names a scoped logical operation; it is not magic deduplication. Atomically claim tenant/principal/operation/key with a canonical request digest, enforce authorization, and persist effect plus replay response together when transactional. Reconcile external effects separately; key expiry and request conflicts require explicit rules.
12. Retries/hedges worsen overload when extra offered work consumes recovery capacity, or when nested retry layers multiply attempts. Two four-attempt layers allow sixteen downstream executions before hedging. Respect Retry-After and a total deadline; fail/defer instead of retrying earlier than allowed.

Exercise 1 solution: Use deterministic GET response sequences (503 then 200; enough handled failures to reach ten observations at ratio ≥0.5; then a successful probe after the break). A timeout fixture must respect cancellation. The pinned response-only predicate does not retry response-less exceptions; test that exclusion explicitly. Random 1000-request statistics supplement rather than prove state transitions.

Exercise 2 solution: Start 100 simulated or authorized test clients together and record every attempt timestamp. Compare fixed delays against jittered delays over repeated runs at identical failure/capacity settings. Maximum four attempts still allows 400 executions. A lower burst peak in a toy model is not measured Polly timing or service recovery.

Exercise 3 solution: Use a unique durable key on tenant/principal/operation/caller-key, compare canonical request digests and commit the mutation and replay response in one transaction. Interleave callers at insertion, crash before/after commit and reject mismatched payloads. One committed row plus a replayed original status/body is the success case; an external charge requires its own idempotency/reconciliation protocol.

Exercise 4 solution: Force the shared breaker open, then allow only authorized age-bounded cached reads with stale/source metadata and a fresh response object. Test wrong tenant/principal, expired cache, caller cancellation and a slow cache backend; these must not become apparent successful live results. Budget fallback outside or inside the appropriate final deadline explicitly.

Exercise 5 solution: With one extra hedge, 100 requests/s can mean 200 attempts/s before retries. For the illustrative A-at-0ms/B-at-300ms sequence where B returns an acceptable result at 450ms, select B and signal A's cancellation; if B returns handled 503 first, it must not win as success. Use independent request instances, record losing-work/disposal and report measured p99 plus capacity cost—even if hedging loses.

Exercise 6 solution: Register a reusable named resilience handler on the intended typed HttpClient, remove duplicate wrapping/retry layers and pass the caller token into SendAsync. Confirm another unregistered client/SQL/Kusto SDK is not implicitly covered. The overall budget and response-only GET predicate remain part of the contract after migration.

These are acceptance designs and bounded decision models; no .NET service execution, cloud fault injection or timing benchmark occurred here.

High-signal resources

Official docs

• Polly v8 documentation — the resilience pipeline model and every strategy.
• Microsoft.Extensions.Http.Resilience — AddResilienceHandler integration.
• .NET 7 rate limiting — the primitives behind Polly's rate limiter.

Books or courses

• Release It! (2nd ed) — Michael T. Nygard; optional stability-pattern reading, with edition/authorship checked against the publisher listing.
• Site Reliability Engineering (Google) — the chapters on cascading failures and overload.

Practitioner posts

• AWS Builders' Library — timeouts, retries, and backoff with jitter — the canonical jitter writeup.
• brooker.co.za blog index — optional practitioner reading; the index is not a verified citation to a specific “fail open vs fail closed” article.
• .NET team — resilient HTTP apps — release notes and migration guides for Polly v8 + HTTP resilience.

Weekly milestones

1. Day 1. Read the Polly v8 docs + the AWS jitter post. Wire Example 1. Self-check questions 1–3.
2. Day 2. Jitter demonstration (exercise 2) + circuit breaker tuning. Self-check questions 4 + 12.
3. Day 3. Timeouts (overall + per-attempt). Self-check questions 5 + 9.
4. Day 4-5. Fallback + durable idempotency state machine (exercises 3 + 4). Self-check questions 7 + 10–11.
5. Day 6-7. Hedging (exercise 5) + AddResilienceHandler migration (exercise 6). Self-check questions 6 + 8.

How it shows up in the capstone

The capstone design uses AddResilienceHandler on its factory-managed HTTP clients where appropriate. SQL/EF Core and the Kusto SDK are not automatically intercepted by that HttpClient registration; configure their supported retry/cancellation layers separately. The HTTP example uses: 30-second overall timeout, up to 3 retries for GET response failures with jittered exponential backoff or the HTTP Retry-After delay on 5xx/408/429, circuit breaker at 50% failure ratio over 10+ calls in 30 seconds, 5-second per-attempt timeout. For an eligible critical read such as the main chart query, an outer fallback may return an authorized, age-bounded cached result with an X-Served-From-Cache header; its retrieval still needs the caller/final deadline.

Write endpoints use the durable ownership/request-digest protocol from Example 2; response caching is not deduplication. A candidate safe-read search policy hedges after 300ms only if measured tail benefit, repeatability and spare capacity justify the extra attempt. Breaker state transitions emit custom metrics for diagnosis and capacity control. Correlate them with user-facing error/latency SLIs; they do not guarantee an earlier warning.

Previous chapter → Ch 15 — Observability Next chapter → Ch 17 — Caching

Corrected contracts and failure analysis

A retry repeats effects as well as work. The standard HTTP handler retries all methods by default; disable unsafe-method retries unless an actual durable idempotency protocol makes them safe. A cache get-then-set is not that protocol: concurrent misses both execute, keys can cross tenants, and lost responses need original status and headers as well as body.

A durable key combines tenant/principal, operation and caller key, stores a request digest, and claims ownership atomically with a unique constraint. Reusing a key with a different digest is a conflict. Persist the effect and result in one transaction where possible; otherwise reconcile the external effect with its own idempotency mechanism. Reserve bounded retention and define in-progress responses.

Strategies nest in registration order. An outer timeout budgets the whole operation; an inner timeout bounds each attempt. Hedging consumes extra capacity and requires idempotent work plus cancellation of losers. Circuit breakers are generally process-local unless explicitly coordinated; their purpose is failing fast under known failure, not guaranteeing availability.

Boundary exercise with solution

Two identical POSTs arrive simultaneously and both miss a response cache. What property is absent?

Source-backed review notes

• Build resilient HTTP apps: Key development patterns - .NET | Microsoft Learn — accessed 2026-10-07. Exact supporting passage: “By default, the standard resilience handler is configured to make retries for all HTTP methods. For some applications, such behavior could be undesirable or even harmful. For example, if a POST request inserts a new record to a database, then making retries for such a request could lead to data duplication. If you need to disable retries for a given list of HTTP methods you can use the DisableFor(HttpRetryStrategyOptions, HttpMethod[]) method:”

Versioned source contracts

Official passages were checked on 7 October 2026; immutable tags are used where available, while service docs remain dated snapshots. The saved evidence records include exact passages, URL, source date/revision and snapshot hash. These references support the stated contracts, not an unperformed service deployment.

• resilience — Default standard HTTP retries need explicit unsafe-method policy.
• polly-http8 — HTTP-specific retry supports Retry-After; generic RetryStrategyOptions does not inherently parse HTTP headers.
• polly-retry — Exponential jitter is not a universal 0.5–1.5 multiplier.
• polly-timeout — Timeout is cooperative cancellation, not proof remote work stopped.
• polly-hedge — ActionGenerator returns a delegate invoking the action context.
• polly-rate — A reused limiter supplies shared admission state; per-call new limiter resets the budget.
• polly-breaker — Failure ratio is evaluated only with sufficient throughput.

Additional contract references

Checked 9 October 2026 as dated official snapshots or the explicit source tags below; these support the stated mechanisms, not a live-service reproduction.

• Hedging resilience strategy | Polly — polly-hedge.
• polly-retry-source — polly-retry-source.
• http-predicates — http-predicates.
• Rate limiting middleware in ASP.NET Core | Microsoft Learn — fixed-window.