08 · Retries: a timeout is not a verdict

The notebook accepts an event, stores it, and returns success. Now lose the reply. The client sees a timeout; the server has already changed state. Retrying is reasonable, but a naive “increment on every arrival” implementation counts twice. Declaring that the transport is reliable does not settle the application-level question: did the requested effect occur?

This lesson uses deterministic, in-memory models. It opens no sockets and measures no network. Its purpose is to make one failure history visible, not to simulate every real distributed-system behavior.

Separate the messages from the effect

A request has a method, target, fields and possibly a body. A response has a status and fields and may carry a representation. The status communicates what the server reports; absence of a response does not prove absence of a committed effect. The failure could happen before the server receives the request, during processing, or after a successful commit but before the reply is observed.

RFC 9110 section 9.2.2 defines an idempotent method by its intended server effect under multiple identical requests. It names PUT, DELETE and safe methods as idempotent. The responses may differ, and the server may log each attempt separately. Deleting an already deleted resource can return a different status without violating the intended repeated effect.

The RFC says a client should not automatically retry a non-idempotent request unless it knows the request semantics are idempotent or knows the original request was never applied. A POST endpoint can implement an application-specific idempotency contract, but POST itself does not confer that property. “Safe” is another distinction: safe methods have essentially read-only requested semantics; idempotent DELETE is not thereby safe.

Execute the failure the explanation claims

# Model 1: a reply is lost AFTER the first effect.
naive_total = 0
 
def naive_receive(amount):
    global naive_total
    naive_total += amount
    return naive_total
 
lost_reply = naive_receive(5)  # server committed; client never observes this value
retry_reply = naive_receive(5)
assert naive_total == 10
assert retry_reply == 10
 
# Model 2: sequential in-memory processing with a stable request identity.
records = {}
total = 0
 
def receive(key, amount):
    global total
    if key in records:
        original_amount, original_response = records[key]
        if original_amount != amount:
            raise ValueError("same key with a different request")
        return original_response.copy()
    total += amount
    response = {"accepted_amount": amount}
    records[key] = (amount, response.copy())
    return response
 
lost_reply = receive("e1", 5)
retry_reply = receive("e1", 5)
assert total == 5
assert retry_reply == {"accepted_amount": 5}
receive("e2", 5)  # same amount, different event: must count separately
assert total == 10
try:
    receive("e1", 9)
except ValueError:
    pass
else:
    raise AssertionError("key reuse conflict was silently accepted")
assert total == 10

This model actually repeats a request after the first effect, unlike a “retry” example that only retries attempts rejected before processing. Deduplication cannot demonstrate value if no duplicate effect is ever attempted. The test checks three distinct cases: same key/same content, different key/same content, and same key/different content.

The notebook's event ID is a logical identity chosen before the first attempt and reused for retries. It is not a fresh random number on every attempt. In a multi-user API, the key would normally be scoped to an authenticated caller and operation so one user's key cannot collide with another's. A key is not an authorization credential.

Find the hole in the model

The second model still has a dangerous interval between changing total and writing records[key]. A crash there loses the deduplication record after applying the effect. Concurrent callers can also both observe the key as absent. The example's sequential execution and in-memory state deliberately exclude those failures; it is not a production exactly-once implementation.

If the effect and the request record live in one transactional database, store them in one transaction with a uniqueness constraint on the scoped key. A retry checks the stored canonical request, then returns the stored response or a defined conflict. The SQLite transaction contract supports grouping changes within one database; it does not group a database change and a remote charge into one atomic operation. An outbox/inbox or provider-supported idempotency protocol may be necessary for external effects, with its own retention and recovery rules.

The capstone implements the local transactional boundary and injects failures before commit. It does not claim a distributed transaction with another service.

A retry policy spends resources

Unlimited retries can turn a transient failure into sustained overload. Bound attempts and elapsed time, distinguish invalid requests from transient failures, and introduce delay rather than an immediate tight loop. Randomized delay can reduce synchronized retries; it does not make a non-idempotent effect safe. A server's Retry-After instruction and the endpoint's documented semantics matter more than a generic list of “retryable” status codes.

A deduplication record also needs a retention policy. Delete it after one hour, then replay a request after two hours, and the old identity can look new. The guarantee is limited to the retention window and the stable scope of the key. Keeping records forever has a storage and privacy cost. A cryptographic hash can compact a request fingerprint, but hashing a string is not a substitute for specifying canonicalization: field ordering, defaults and relevant identity fields need a contract.

Exercises

1. Draw the timeline for “effect committed, acknowledgement lost, retry arrives.” Mark what the client knows at each point.
2. Move records[key] before the effect. Did that remove the crash gap?
3. Should a duplicate response include the current global total or the original result?
4. A GET handler increments a business counter as its primary action. Is its method label enough to make the action safe?

Exit artifact: a failure timeline, a same-key conflict test, and a written boundary around the guarantee. No network fault-injection, transport verification, payment processing or concurrency correctness is claimed here.

Next: Bounded work and resources.