Dinesh/ Blog
systems

Designing for Scale · Notification Systems

The provider acknowledgement is an uncertainty boundary. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.

9 min read
Checking device speech…
Lesson preparation & details

Level: advanced

Designing for scale · Lesson 8 of 20 ↗
Loading this browser’s progress…

By the end, you should be able to

  • The provider acknowledgement is an uncertainty boundary
  • A provider times out after possibly accepting an email. Can a local unique constraint alone prove one external send?

Bring with you

Sources checked: . A source check does not change the publication or update date.

Editorial review: · What review means

In this article · 15 sections

The problem this post solves

A provider's successful HTTP response is not necessarily proof that a person received, opened or read a notification. Separate durable notification intent, provider acceptance, device delivery and user interaction. A receipt, password-reset link and promotional campaign also have different freshness, consent and duplicate-tolerance requirements.

Consider a transactional receipt generated when an order commits. If the order succeeds but notification intent is lost, a later worker cannot reconstruct the missing work unless the system retained another authoritative event. If a send succeeds but its response is lost, a retry may send twice. These two gaps—not simply wrapping an API—drive the design.

First principles

A queue decouples request latency from provider latency and permits controlled retries. It is a common resilience mechanism, not mandatory for every third-party call. An asynchronous HTTP client need not block one operating-system thread per request, but it still consumes bounded sockets, memory and in-flight work. A direct call with strict deadlines can be a reasonable small-workload choice if its failure semantics are acceptable.

Queueing does not guarantee eventual delivery. The destination can remain invalid, a reset link can expire, consent can be withdrawn or retention can run out. Define “accepted for processing” separately from “delivered,” and expose states that preserve uncertainty rather than converting every timeout into known failure or every enqueue into known success.

Architecture

Architecture
Figure text equivalent — labels and relationships
  • Node “Business transaction” (square).
  • Node “Notification intent and outbox” (cylinder).
  • Node “Bounded channel and priority queues” (square).
  • Node “Workers with rate and concurrency budgets” (square).
  • Node “Current consent and destination check” (square).
  • Node “Provider request” (square).
  • Node “Accepted or receipt observed” (square).
  • Node “Ambiguous timeout” (square).
  • Node “Retryable error with next attempt” (square).
  • Node “Permanent failure or expiry” (square).
  • Node “Receipt lookup or explicit retry policy” (square).
  • Node “Auditable dead-letter or cancellation” (square).
  • “Business transaction” leads to “Notification intent and outbox”. Line: normal; ends: arrow point.
  • “Notification intent and outbox” leads to “Bounded channel and priority queues”. Line: normal; ends: arrow point.
  • “Bounded channel and priority queues” leads to “Workers with rate and concurrency budgets”. Line: normal; ends: arrow point.
  • “Workers with rate and concurrency budgets” leads to “Current consent and destination check”. Line: normal; ends: arrow point.
  • “Current consent and destination check” leads to “Provider request”. Line: normal; ends: arrow point.
  • “Provider request” leads to “Accepted or receipt observed”. Line: normal; ends: arrow point.
  • “Provider request” leads to “Ambiguous timeout”. Line: normal; ends: arrow point.
  • “Provider request” leads to “Retryable error with next attempt”. Line: normal; ends: arrow point.
  • “Provider request” leads to “Permanent failure or expiry”. Line: normal; ends: arrow point.
  • “Ambiguous timeout” leads to “Receipt lookup or explicit retry policy”. Line: normal; ends: arrow point.
  • “Retryable error with next attempt” leads to “Bounded channel and priority queues”. Line: normal; ends: arrow point.
  • “Permanent failure or expiry” leads to “Auditable dead-letter or cancellation”. Line: normal; ends: arrow point.

Commit durable intent with the business event where a shared transaction is available. An outbox publisher can then enqueue it repeatedly until acknowledged. The API may report that the business operation committed and a receipt is queued; it must not claim the email was delivered before that happened. Where a shared transaction is unavailable, design an explicit recoverable event/reconciliation boundary instead of ignoring the dual write.

The notification service can have APIs for intent ingestion, preferences and status as well as workers for delivery. It is not defined by “having no API.” Separate transactional and bulk traffic so a campaign cannot exhaust every worker or provider quota. Queue capacity, age, retention and storage cost are finite; a provider outage can still affect upstream availability when those limits are reached. Define admission/degradation behavior then.

The Notification Workers

A worker claims a durable intent under a lease/attempt identifier, then resolves the current recipient destination and allowed channel. It checks consent, unsubscribe/quiet-hour rules, expiration, template version and payload constraints. Sensitive content and attachments need scoped access; do not place permanent public links or credentials in queue bodies/logs. Persist the provider request/response identifier and classification without storing unnecessary personal data. Clear ownership and retention rules apply to logs and dead-letter payloads too.

An attempt that crashes after calling the provider is not necessarily safe to resend merely because the worker lease expired. Mark it uncertain and use a documented provider idempotency or receipt-query contract if one exists. “Accepted” is still not a read receipt. A later provider callback also needs authentication, deduplication and ordering/version checks before it updates local state.

Deduplication and Rate Limiting

Deduplication

Use a stable identity such as (tenant, business_event, recipient, channel, template_version) for the notification intent. Compare equality-defining content if that identity repeats. A resend requested as a new business action may legitimately need a new identity, while a transport retry of the same action must keep its identity. A hash of user/type/current minute can merge distinct events or split one event across a minute boundary; it is not a durable semantic definition of sameness.

An in-memory SETNX cache can coalesce recent duplicate requests, but cannot alone make an external send exactly once. Set the key first and a crash before sending may suppress delivery. Send first and a crash before setting the key may duplicate delivery. Retain the durable intent and explicit attempt states. If a provider supports an idempotency key, use the same key across retries within its documented retention/payload rules; do not infer support from an arbitrary request-ID header. If no such contract resolves ambiguity, choose and document whether duplicate risk or loss risk is preferable for that message class.

Provider Rate Limiting

Model provider/account/device/topic quotas separately where applicable. One local bucket per worker multiplies the global allowance unless budgets are coordinated or partitioned. A token bucket permits its configured burst; that can violate a provider's strict rolling window if the provider uses a different policy. Combine the actual quota contract with bounded in-flight concurrency and retry budget rather than promising one algorithm can never exceed every provider limit.

FCM's HTTP v1 error documentation distinguishes QUOTA_EXCEEDED (429) from UNAVAILABLE (503). For exceeded overall/topic rate it specifies exponential backoff with a minimum initial delay of one minute. For 503 it says to honor Retry-After when present, use exponential backoff and consider jitter. This is provider-specific guidance, not a universal one-second retry schedule. Read the error details to identify the quota; reducing one account's total rate may not fix a per-device hot spot.

The Retry Architecture (Dead Letter Queues)

Store attempt count, next eligible time, last outcome, expiry and original intent identity. Delayed queues, a scheduled retry table or separate queues can all implement backoff; a particular chain of one-minute/five-minute queues is not mandatory. Separate queues/worker budgets by provider and priority where needed so one failing destination does not repeatedly consume all capacity. Maintain fairness for healthy new transactional messages.

Suppose 10,000 intents arrive per minute while a provider is down for ten minutes: 100,000 backlog items accumulate before expiration/cancellation. After recovery, a worker capacity of 15,000/min with continuing arrivals of 10,000/min drains at a net 5,000/min, taking twenty minutes in this idealized constant-rate scenario. Retrying at full speed without accounting for new arrivals can prolong the outage. Real time varies with quota changes, payload cost and failure rates; the arithmetic is a capacity model, not a benchmark.

Stop on permanent invalid destinations, exhausted policy or expired purpose. A password-reset link that expired during the outage should not be delivered blindly hours later; the product may require a new user-initiated reset. Dead-letter records need reason, owner, alerting and an inspected replay operation that retains identity and rechecks consent. Moving messages to a DLQ is not successful delivery and is not a reason to retain sensitive payloads forever.

Failure modes

Provider-specific token lifecycle

FCM HTTP v1 UNREGISTERED is a 404-class response indicating that the app-instance token is no longer valid; its documentation says to remove that registration token from the app server and stop using it. Do not confuse it with SENDER_ID_MISMATCH or treat every failure as evidence that the user uninstalled the app.

Apple's archived HTTP/2 APNs guide documents 410 with reason Unregistered for a token inactive for the specified topic. Its timestamp indicates when APNs confirmed invalidity; stop pushing until the device registers a token with a later timestamp. Scope invalidation to the relevant token/topic and compare registration generation/time so a delayed error for an old token does not delete a newly registered destination. These are distinct provider contracts, not a universal legacy string NotRegistered. The archive is used for that narrow protocol behavior; consult current APNs requirements before implementing a new client.

Queue limits depend on the selected product; “all queues support only tiny byte payloads” and “a 5 MB payload always crashes the queue” are unsupported generalizations. Keeping large attachments in object storage can reduce repeated transfer and broker pressure. Queue an object identifier, content hash and access policy; generate a suitably scoped recipient link when needed. Ensure an expired link, deleted object or changed permission produces a defined failure rather than a permanently inaccessible receipt. Do not execute provider sends while running a teaching test.

Uncertain acceptance

A timeout after a provider may have accepted the message is neither a definite failure nor proof of delivery. A local uniqueness constraint prevents two intent rows, not two provider side effects. If receipt lookup or idempotency cannot resolve it, retain an auditable uncertain state and apply the explicitly chosen policy. Consent and expiry are checked again before any retry; an earlier queued preference snapshot is not enough for a later marketing send.

Worked state-machine exercise

Intent receipt:order17:user3:email:v2 is durably recorded. A worker sends it, times out, then crashes. Its queue delivery is repeated. Meanwhile, another user action creates a different receipt in the same minute. Explain why both a minute hash and a blind resend are wrong.

Identity, uncertainty and independent events

The repeated queue delivery resolves to the existing intent, but its external outcome stays uncertain. Retry with the provider's supported same-key contract or perform receipt lookup; otherwise apply the message-class duplicate/loss policy rather than inventing exactly-once. The other order has a different business-event identity and must not be dropped merely because its user/type/minute matches. Before sending either, verify current destination, consent and expiry. Mark provider acceptance separately from delivery/read status and keep the original identity during inspected dead-letter replay.

What to carry forward

Persist intent, bound work and preserve uncertain outcomes. Deduplication, provider throttling, priority isolation and destination lifecycle each solve a different problem. Queues and retries improve recovery options; they do not guarantee delivery or make consent optional. No provider message, cloud queue or device lifecycle was executed for this conceptual review.

Sources and review boundary

Checked 2026-10-07: SQS standard duplicate-delivery guidance, FCM HTTP v1 error codes, and Apple archived APNs HTTP/2 request/response guide. These are official access-dated snapshots; the APNs source is explicitly archival and FCM claims are scoped to HTTP v1, not every SDK. RFC 6585 (2012) defines 429 and optional Retry-After generically; provider-specific retry requirements take precedence in an integration. Architecture, backlog arithmetic and failure traces are design analysis, not claims of a deployed notification service.

Pause / Recall / Apply

Can you explain it without the page?

Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.

Stored in this browser only. No account, no sync. Clearing browser data removes your record.