systemsadvanced 22m read

Designing for Scale · Notification Systems

Why sending a push notification is not a fire-and-forget API call. Handling rate limits, provider outages, deduplication, and the retry queues required to make delivery reliable.

The problem this post solves

A notification system looks like a simple wrapper around third-party APIs (Apple APNs, Firebase FCM, SendGrid). The naive implementation takes a request, formats a JSON payload, calls the provider, and returns success.

That naive implementation falls apart immediately. What happens when the provider is down? What happens when a user rapid-fires ten actions that trigger ten identical emails? What happens when the marketing team decides to send a push notification to five million users, and the system attempts to open five million outbound HTTP connections simultaneously?

This post moves from the naive wrapper to the asynchronous, queue-driven architecture required to guarantee delivery, respect rate limits, and prevent self-inflicted denial-of-service attacks.

First principles

From first principles

Start with the question

Why can't the API server just call the notification provider directly?

1
Calling an external provider requires a synchronous network hop.
forced by · You must wait for their API to respond with success or failure.
2
External APIs are slow, rate-limited, and occasionally unavailable.
forced by · They are complex distributed systems themselves.
3
If the API server waits for the provider, it consumes a thread/connection while waiting.
forced by · The thread cannot serve other users until the HTTP call returns.
4
If the provider slows down, the API server's connection pool fills up with waiting threads.
forced by · New requests keep arriving, but old ones aren't finishing.

⇒ Therefore

Synchronously calling external APIs from your primary web servers is a recipe for cascading failure. Notifications must be asynchronous.

Architecture

The fundamental shift is decoupling the intent to send a notification from the act of sending it.

When a microservice (e.g., the Billing Service) needs to send a receipt, it does not call the Notification Service API and wait. It drops a message onto a queue and immediately returns success to the user.

The Notification Workers

The Notification Service is not an API; it is a fleet of background workers pulling off the queue.

When a worker pulls a message:

It queries the database to find the user's contact info and device tokens.
It checks the user's preferences (e.g., "Did they disable promotional emails?").
It formats the payload according to the specific provider's requirements.
It executes the HTTP call to the provider.

Mental modelThe Delivery Queue separates concerns

A conveyor belt. The upstream services (Billing, Social) just dump boxes on the belt. They don't know how to drive the delivery trucks. The Notification workers take boxes off the belt, look at the address, and put them in the correct truck (APNs, SendGrid).

Upstream services never know about device tokens or email templates.
If Apple's API goes down, the boxes pile up on the conveyor belt. The Billing Service is completely unaffected.
When Apple comes back online, the workers catch up.

🔔 Fires when you see

Use message queues whenever you integrate with a third-party API that you do not control.

Deduplication and Rate Limiting

If a user repeatedly clicks "Resend Verification Email" 10 times in two seconds, the naive system will send 10 emails. This annoys the user, costs money, and risks your domain being marked as spam.

Deduplication

Before sending, the worker calculates a hash of the notification (e.g., hash(user_id, email_type, timestamp_truncated_to_minute)). It attempts to insert this hash into Redis with a short TTL (e.g., 5 minutes) using SETNX (Set if Not eXists). If Redis returns 1, it's the first time; proceed with sending. If Redis returns 0, it's a duplicate; drop the message.

Provider Rate Limiting

Providers enforce strict limits (e.g., "Maximum 500 emails per second"). If you have a backlog of 5 million marketing emails on your queue, your workers might pull them and hammer the provider with 10,000 requests per second. The provider will return HTTP 429 (Too Many Requests), and you will have to retry them all, wasting resources.

The workers must use a Token Bucket rate limiter internally, ensuring they never exceed the provider's known limits, even if the internal queue is overflowing.

The tradeoff

Handling provider failures (HTTP 500 or 429)

Drop the message

+ you gain Simple. Prevents the queue from backing up permanently.

− you pay Data loss. The user never gets their password reset link.

pick when Only for low-priority, ephemeral notifications (e.g., 'Someone liked your post').

Exponential Backoff and Retry

+ you gain Guarantees eventual delivery.

− you pay Requires a complex Dead Letter Queue architecture. A failing provider can cause millions of messages to cycle endlessly, burning CPU.

pick when Mandatory for transactional notifications (receipts, password resets, security alerts).

The Retry Architecture (Dead Letter Queues)

When a worker attempts to send an email and the provider times out, the worker cannot just put the message back on the main queue. If it does, the main queue will fill up with failing messages, blocking the fresh, healthy messages from getting through (Head-of-Line blocking).

Failed messages must be moved to a separate Retry Queue.

If the message fails, it goes to the 1-minute retry queue. A separate worker checks that queue. If it fails again, it moves to the 5-minute queue. Eventually, it moves to the Dead Letter Queue (DLQ), where it sits until an engineer investigates the systemic failure.

Failure modes

Missing the unregister event. When a user uninstalls your app, Apple/Google will eventually return a specific error code ("NotRegistered") when you try to push to that device token. If your workers ignore this error, you will permanently waste resources pushing to dead tokens, and the providers will eventually throttle or ban you. The worker must catch this specific error and delete the token from the database.

The mega-payload. Attaching a 5MB PDF receipt directly to the queue message. Message queues are designed for tiny payloads (bytes or low kilobytes). Massive payloads cause the queue to thrash to disk and crash. Store the PDF in an Object Store (S3), and put the URL in the queue message.

What to carry forward

Key points

You can now

Explain why asynchronous queues are mandatory for third-party integrations.
Design a deduplication mechanism using an in-memory cache.
Architect a retry system that prevents Head-of-Line blocking.
Handle the 'NotRegistered' lifecycle of push notification device tokens.