Message Queues — SQS, RabbitMQ, Kafka as Queue

Learning goal: Pick the right queue for a job (SQS, RabbitMQ, Kafka), design for at-least-once delivery, and avoid the classic poison-message and reprocessing-flood failures.

Why this session exists

“Put it on a queue” separates accepting work from completing it. It is useful for asynchronous email, image resizing and long-running refund workflows. With the right durability, retention and recovery contract, components can fail and resume without losing the work covered by that contract. Get it wrong (wrong delivery semantics, no dead-letter queue, no idempotency) and you'll spend Fridays chasing "why did we send this email 47 times?" mysteries. This session gives you the mental model and the operational discipline.

You will be able to

• Explain the difference between a queue (point-to-point) and a log (publish-subscribe) in one minute.

• Compare SQS, RabbitMQ, and Kafka on delivery semantics, ordering, and throughput.

• Design a consumer that is idempotent + safe against duplicates + robust to poison messages.

• Configure a dead-letter queue and retry policy without creating a retry storm.

• Estimate network, storage and consumer bottlenecks at 10 K/sec, 100 K/sec and 1 M/sec from explicit assumptions—not product names.

Prerequisites

• S055 · HTTP fundamentals — you know what synchronous requests look like; this is the async cousin.
• S071 · Replication — queue durability = replication under the hood.
• S072 · Consistency models — delivery, ordering and atomic effects are distinct contracts to name.

Optional preparation and pacing

The original session suggested a 90-minute study block: preparation, intuition, visual reasoning, a local exercise, production failure analysis and recall. Treat all durations as planning cues, not measured completion times; pause and revisit the worked answers as needed. Optional resources are not prerequisites for running the offline examples. Videos have not been watched or timed here; their availability, creator attribution, contents and quality remain unverified. Search links are searches, not selected talks.

• Kafka in 100 seconds — Optional queue/log orientation; historical title and creator label, not a verified recording description.
• How Message Queues Work — Optional message-delivery models; historical title and creator label, not an endorsement of the old diagram praise.
• Kafka internals — Hussein Nasser — Search for Kafka internals and log-design discussions, not an identified recording or verified speaker.
• Amazon SQS · FIFO queues — Historical FIFO announcement pointer; use the version/access-bounded primary documentation below for present deduplication and delivery contracts, not the old “best explanation” endorsement.
• Kafka · Design — General Kafka design entry point; the Kafka 4.1 design source below governs this lesson, not an unspecified latest version.
• The Log: What every software engineer should know — Optional log-abstraction essay; the earlier claim that this essay inspired Kafka is not established here.

(a) Intuition · 5 min

A ticket window vs a movie theatre

Ticket window (queue): customers form a line. Each customer is served by one teller. Once served, they leave — the next customer takes their place. A second customer cannot be served by the same ticket as the first. This models competing consumers: one delivery attempt is assigned to a worker, but timeouts/redelivery can cause multiple workers to process the same logical message.

Movie theatre (log): 200 people watching the same film. If someone shows up late they miss the earlier scenes (or, on a DVR, they can rewind). Every seat sees every frame. This is a log: many independent consumers can read the same messages, each at their own pace.

SQS and RabbitMQ are ticket windows: workers receive delivery attempts and successful acknowledgment permits removal; failures can cause redelivery. Kafka is a movie theatre: the message stays in the log, and many consumer groups read it independently at their own offsets.

You'll be tempted to think one model is "better". They aren't — they solve different problems. Changing the consumption and replay model later can require substantial application work.

The five semantic decisions every queue makes

• Delivery: at-most-once permits loss without retry; at-least-once permits repeated attempts under retention, durability and progress assumptions. Exactly-once effects need a defined transactional/deduplication boundary.
• Ordering: none, per-key/group/partition, or a global order. Parallel preparation can coexist with ordered commit; global order does not require every stage to be single-threaded.
• Consumption: competing workers versus independent subscriptions/groups. A second worker in the same group divides work; a separate group/subscription gets its own logical copy.
• Retention: queues commonly remove acknowledged work; logs retain history independently of progress, subject to time/size/compaction policies. Neither promises infinite replay.
• Acknowledgment: producer confirms and consumer acknowledgments answer different questions. Batching and auto/manual acknowledgment affect failure windows.

Kafka transactions can atomically commit consumed offsets and produced Kafka records, with appropriate transactional producers and read_committed consumers. An external database can also achieve a single effect by atomically recording a stable event ID with its state change, or coordinating an offset in that destination. It is false that exactly-once effects are possible only inside Kafka. An external email/card provider requires its own idempotency or reconciliation protocol; a local seen-set alone is insufficient.

A history — from single-node queues to global logs

Traditional enterprise messaging emphasized routing and acknowledged work; log-based designs emphasize retained history and independent consumption. Kafka's versioned design documentation explains batching, partition ordering and transaction boundaries. RabbitMQ exposes routing exchanges, classic/quorum queues and streams; SQS exposes Standard and FIFO queues. These overlapping feature sets are more useful to compare than an unsupported chronology or a fixed throughput hierarchy.

Queue vs log — the fundamental picture

Before choosing a broker, decide whether a message represents work to finish or history to retain. Compare the queue's competing consumers with the log's independent group offsets.

In the queue, acknowledging work permits its removal; in the log, a group advances its position while retention remains separate. Neither model removes the gap between doing work and recording that it was done.

The producer→consumer flow with all the failure points

Assume each participant can crash immediately after a successful action. The two marked gaps ask the same question: can the sender know that its action took effect before it decides to retry?

The dangerous consumer gap is between changing the database and acknowledging the message. A redelivery is then reasonable broker behaviour, so correctness has to survive repeated execution of the consumer's effect.

The four canonical failure modes

Poison message

A malformed or unprocessable message. Consumer errors and does not ack; redelivery can consume attempts until a configured limit or retention boundary. Fix: max-receive-count + DLQ.

Duplicate delivery

Consumer processed but crashed before ack. Broker redelivers. Fix: idempotent processing (dedup by msg_id, use INSERT ON CONFLICT).

Slow consumer / backlog

Producer outpaces consumer. Backlog grows until capacity, retention, throttling or admission limits intervene. Fix: back-pressure, autoscaling consumers, or drop policy.

Reprocessing storm

Consumer bug fixed, replay 3 days of messages. Suddenly downstream API gets 1000x traffic. Fix: rate-limited replays, staged rollouts.

Where each system sits

The five layers of a queue system

Producer SDK

Batches, retries, chooses partition, computes idempotency token.

Broker protocol

AMQP (RabbitMQ), Kafka wire protocol, HTTP+JSON (SQS). Determines what SDKs exist.

Broker cluster

Where messages live. Replicated across nodes. Handles ordering, retention, ack.

Consumer SDK

Pull vs push model, ack strategy, offset commit, dead-letter routing.

Observability

Depth per topic, consumer lag, DLQ size, oldest-message-age. These metrics are your queue's vital signs.

Side by side — the three defaults

SQS Standard is a managed competing-consumer queue with at-least-once delivery and best-effort ordering. Its approximately 120,000 in-flight-message quota is not a 120,000 messages/second throughput cap. Current quotas document very high/nearly unlimited API throughput for Standard and region/configuration-specific FIFO limits. Payload, batching, concurrency, client limits and downstream work all matter.

SQS FIFO orders messages within MessageGroupId and suppresses duplicate sends within its documented five-minute deduplication interval. A consumer that fails to delete a received message before its visibility timeout may receive it again. “Exactly-once processing” in the producer-dedup documentation does not make an external card charge transactional. Group count and distribution affect parallelism; high-throughput FIFO is not universally capped at 3,000 messages/s.

RabbitMQ provides direct, topic, fanout and other routing patterns. Quorum queues replicate and require a majority for availability; publisher confirms and consumer acknowledgments are independent. A confirmed publish is not a completed business effect. Manual acknowledgment, prefetch, queue type, replication and payload influence performance. Push delivery alone does not establish lower end-to-end latency than pull, and streams offer a different retained-log option.

Kafka 4.1 retains partition logs and independent group offsets. Classic consumer groups assign a partition to one member at a time, but application concurrency/retries must preserve the processing order they claim. Records can be replayed only while retained and readable; compaction may remove earlier values. Partition count, replication, acknowledgments, batch size, compression, storage and consumer work determine capacity—not “a million per broker on modest hardware.”

Capacity at 10K, 100K and 1M events per second

Assume each record has 1,000 bytes of payload, no compression and three durable copies. Ignore metadata, indexes, protocol overhead and reads for this lower-bound planning calculation:

These decimal volumes apply to the assumed retained stream, not to SQS's internal replica count or billing. Seven days at 100K/s is 60.48 TB raw, or 181.44 TB with three copies before overhead. A worker taking 20 ms per message can do at most 50/s if strictly sequential; the three rates need at least 200, 2,000 and 20,000 such workers' worth of concurrency, before headroom and retries. If one ordering key carries half the rate, adding unrelated partitions does not parallelize that serialized key.

Thus “which product bottlenecks at 100K/s?” has no answer without a configuration. Test message sizes, burst distribution, durable acknowledgments, per-group skew, retention, consumer CPU and destination limits. Record p95/p99 latency and recovery under backlog; do not turn this arithmetic into a broker benchmark.

The mental model to hold

Make the effect boundary explicit. A producer times out after the broker persisted M; retry may publish another attempt. A consumer commits an effect then crashes before ack; redelivery is reasonable. Deduplication must use the stable business event identity, not a new random ID on each retry. Store the ID and effect in the same transaction, and reject reuse of an ID with a different payload. A TTL cache can expire before replay and is not by itself an accounting ledger.

For email, “send, then mark processed” can resend after a crash. “Mark processed, then send” can omit the email after a crash. Neither solves the external atomicity gap. Use provider idempotency if available; otherwise preserve the ambiguous state and reconcile provider receipts rather than claiming exactly-once delivery.

Why logs can be efficient. Sequential appends, batches and shared retained records can reduce per-message overhead. Classic offset tracking represents a contiguous completed prefix: committing offset 6 says earlier relevant records are processed, so do not commit past unfinished message 3 just because message 5 finished. Parallel workers may track holes and advance only the safe prefix. Additional groups still consume network, CPU and I/O; fan-out is not free. Traditional queues can batch acknowledgments and log state sequentially too, so “queue acknowledgments must cause random I/O” is not a valid universal proof of performance.

Backlog is a rate difference. If arrivals are 1,200/s and completion capacity is 1,000/s, ten minutes add 120,000 messages. After arrivals fall to 800/s, the net drain is 200/s, requiring another ten minutes. Sustained overload cannot drain itself, but a temporary burst can. Alert on oldest age, arrival/completion rates, retention margin and failure counts, not depth alone.

Retries need a shared budget. At 1,000 new tasks/s and a 10% independent per-attempt failure chance with at most three attempts, expected work is 1 + 0.1 + 0.01 = 1.11 attempts/task, or 1,110 attempts/s. A correlated downstream outage violates the independence assumption and can be much worse. Use capped exponential backoff with jitter, bounded in-flight work, destination-wide rate limits and a circuit breaker. A separate retry loop per worker can collectively overwhelm the destination.

Close the producer gap with an outbox

Consumer deduplication cannot recover an event the producer never published. Suppose the order database commits order-7, then the process crashes before publishing order.created. Reversing those two writes is no cure: a published event can refer to an order whose database transaction later rolls back.

Instead, commit the order change and an outbox row with a stable event ID in one local transaction. A polling relay or CDC reader publishes committed rows and marks progress only after the broker confirms the configured durability boundary. If publishing succeeds but recording progress fails, the relay publishes again: keep the same event ID and let the destination's atomic dedup/effect transaction absorb the repeat. Do not mark an outbox row sent before publishing. Persist failed/ambiguous attempts, monitor oldest unsent age, and retain enough outbox/dedup history for the maximum authorized replay window.

Outbox atomicity covers local business state plus intent, not the broker and not an external payment provider. Concurrent relays need a claiming/ordering protocol. Per-order event versions and downstream stale-version rejection can be necessary even with one stable ID per event; dedup prevents repeats of one event, not reversal of two distinct events. See the Debezium outbox event router for the documented event-ID and aggregate-key roles; this is a design explanation, not a configured connector test.

Failure exercise and answer: in a paper trace, suppose a relay successfully publishes but crashes before persisting its progress. A retry sends the same event ID twice; the destination should record one effect. If the local order transaction rolls back, neither the order nor its outbox row should remain. If the destination is an email provider without a usable idempotency contract, preserve uncertainty for reconciliation rather than declaring the email exactly once.

Choosing synchronous or asynchronous boundaries

Use a synchronous call when the caller needs a result now and can handle timeout ambiguity. Use durable asynchronous acceptance when work can finish later, but return a request ID only after its intent is durably stored. Model pending/succeeded/failed states and expose status polling or notifications. A durable queue decouples producer and consumer schedules only within capacity/retention and shared-dependency limits.

If three independent required synchronous dependencies each succeed with probability 0.999 per request, the simple combined model gives 0.999**3 = 0.997003. Correlated failures invalidate independence, and retries/caching change the model. Three calls are not automatically wrong; use the workload contract rather than a rule that every side effect must be asynchronous.

(c) Hands-on · 25 min

This deterministic queue uses a logical clock, bounded receive attempts and delivery-generation receipts. It deliberately has no networking, durable broker storage or production concurrency. The SQLite consumer demonstrates an atomic local database effect and dedup record; the in-memory SQLite instance is not a crash-durability test. The code's strict stale-receipt rule is the toy's contract, not an assertion of every detail of SQS receipt handling.

from dataclasses import dataclass
import sqlite3
 
@dataclass
class Message:
    event_id: str
    amount: int
    attempts: int = 0
    visible_at: int = 0
    receipt: int = 0
 
class Queue:
    def __init__(self, timeout=5, max_attempts=3):
        self.timeout, self.max_attempts = timeout, max_attempts
        self.pending, self.dead = {}, []
        self.next_id = 0
 
    def send(self, event_id, amount):
        self.next_id += 1
        self.pending[self.next_id] = Message(event_id, amount)
        return self.next_id
 
    def receive(self, now):
        for delivery_id, m in list(self.pending.items()):
            if m.visible_at > now:
                continue
            if m.attempts >= self.max_attempts:
                self.dead.append(m)
                del self.pending[delivery_id]
                continue
            m.attempts += 1
            m.receipt += 1
            m.visible_at = now + self.timeout
            return delivery_id, m.receipt, m.event_id, m.amount
        return None
 
    def ack(self, delivery_id, receipt):
        m = self.pending.get(delivery_id)
        if m is None or receipt != m.receipt:
            return False
        del self.pending[delivery_id]
        return True
 
class Consumer:
    def __init__(self):
        self.db = sqlite3.connect(':memory:')
        self.db.executescript("""CREATE TABLE seen(id TEXT PRIMARY KEY, amount INTEGER);
          CREATE TABLE totals(id INTEGER PRIMARY KEY, amount INTEGER);
          INSERT INTO totals VALUES(1, 0);""")
 
    def apply(self, event_id, amount):
        with self.db:
            old = self.db.execute('SELECT amount FROM seen WHERE id=?', (event_id,)).fetchone()
            if old is not None:
                if old[0] != amount:
                    raise ValueError('event ID reused with different payload')
                return False
            self.db.execute('INSERT INTO seen VALUES(?,?)', (event_id, amount))
            self.db.execute('UPDATE totals SET amount=amount+? WHERE id=1', (amount,))
        return True
 
q, c = Queue(), Consumer()
q.send('e7', 20)
first = q.receive(0)
assert c.apply(first[2], first[3])  # local transaction succeeds
# Crash-before-ack is modeled by omitting ack. Redeliver at t=5.
second = q.receive(5)
assert second[:2] != first[:2]
assert not q.ack(first[0], first[1])  # old delivery cannot retire new lease
assert not c.apply(second[2], second[3])
assert q.ack(second[0], second[1])
assert c.db.execute('SELECT amount FROM totals').fetchone()[0] == 20
try:
    c.apply('e7', 999)
except ValueError:
    pass
else:
    raise AssertionError('conflicting retry was accepted')
 
q.send('poison', -1)
for now in (10, 15, 20):
    assert q.receive(now)[2] == 'poison'  # handler declines it; no ack
assert q.receive(25) is None
assert len(q.dead) == 1 and not q.pending
assert q.dead[0].event_id == 'poison' and q.dead[0].attempts == 3
c.db.close()
print('effect=20 once; stale receipt rejected; poison quarantined after 3 attempts')

What each block does

receive advances the attempt generation and visibility deadline. At t=5 the previous delivery can overlap with a new one; a real worker may still be running when its lease expires. ack checks the current receipt generation. Consumer.apply commits a unique event ID and balance change together; a repeated matching event has no second effect, while conflicting payload reuse raises an error. Production consumers need concurrent-writer-safe uniqueness/transaction handling and durable storage—this sequential fixture does not certify either deployment property.

The poison event reaches the local DLQ after three failed attempts. A random 30% failure probability would not guarantee zero dead letters: three independent failures occur with probability 0.3**3 = 2.7%, so among 50 messages the expected count is 1.35 under that simplified model. Counting successful unique effects is also not counting handler attempts. Keep separate counters.

Exercise: what if the visibility timeout is shorter than the handler? Answer: another attempt may begin before the first finishes. Increase the initial lease based on measured processing distribution, extend it while healthy within provider limits, and preserve idempotency anyway. A p99-based timeout is not a correctness guarantee. In the toy, t=0 and t=5 explicitly reproduce overlapping ownership generations.

(d) Production reality · 15 min

Retry and dead-letter operating contract

For an SQS reference configuration, create a separate appropriate-type DLQ, grant the source permission to use it, and set a redrive policy naming its ARN and maxReceiveCount (for example 5). Match Standard-to-Standard or FIFO-to-FIFO requirements. Choose retention longer than the source retention where appropriate; Standard DLQ expiry retains the original enqueue age, while FIFO resets the enqueue timestamp on the move. Read the provider's policy before setting a retention window.

A starting five-attempt budget is a workload assumption, not an industry guarantee. Alert on oldest DLQ age and count, record error class, event identity and attempt history, and assign an owner. Repair code or payload before a bounded replay that preserves original event IDs; monitor the destination's capacity. Moving a FIFO message to a DLQ lets later messages progress and can break an application's required end-to-end sequence. Sometimes quarantine the whole entity/group instead. A DLQ isolates failures; it does not ensure work succeeded or will never expire.

For RabbitMQ, enable the intended durable queue type, persistent publishing where required, publisher confirms and consumer acknowledgments; handle negative acknowledgments and unroutable publishes. For Kafka, define acknowledgment/replication policy, consumer commit boundaries, retention, transactional read mode when needed, and a poison-record strategy. A log offset at the head does not prove a valid business effect if offsets were committed too early.

Where this shows up in the rest of the plan

CDC connects database intent to replayable changes. Sharding explains ordering-key hotspots. Multi-region asks whether broker state, consumers and destinations share a failure domain.

(e) Recall + stretch · 10 min

1. Queue versus log? A work queue distributes delivery attempts among workers and commonly retires acknowledged work; a retained log permits independent group progress and replay within retention. Product features overlap.

2. Does FIFO dedup make a charge exactly once? No. Deduplication of producer sends and transactional external effects have different boundaries. Preserve a stable charge ID and use the destination's supported idempotency/reconciliation contract.

3. What does a visibility timeout do? Temporarily hides a delivery; expiry can cause another attempt even while the old handler is active. It is not proof the old worker stopped.

4. How do ordered users work? SQS FIFO groups or Kafka keys can supply a per-group/partition sequence, but processing concurrency, retries, repartitioning and DLQ policy must preserve the intended order. RabbitMQ routing alone is not an end-to-end ordering guarantee.

5. What is consumer lag, and does zero mean healthy? A common per-partition offset-lag measure is the end offset minus the group's committed next offset. State which end position and isolation level your tool uses; offset distance is not necessarily an exact count of deliverable business records. Zero does not prove health: inspect business completion/errors, age and early-commit bugs; producers may also have stopped. Offset lag and time lag are different measures.

6. How does a database dedup table work? Commit its unique event key and all local effects in one transaction. A dedup insert followed by a separate transaction for the effect leaves another crash gap.

7. Which broker is fastest? No defensible ranking follows from names alone. Use the 1 KB planning table and measure the actual configuration. The old 120K in-flight number was not throughput.

8. What does a DLQ guarantee, and how many receives? It quarantines work after the configured receive budget; it does not promise success or indefinite retention. Choose the budget from transient-failure tolerance and destination capacity, then monitor age, error class and replay ownership. Three or five is an example, not a universal default.

9. How does a five-minute downstream outage become a retry storm? Attempts accumulate while capacity is unavailable, then compete with new traffic on recovery. Bound total destination concurrency/rate, use capped backoff with jitter and circuit breaking, and drain replay within spare capacity. Per-worker limits alone need not enforce a destination-wide budget.

Food-delivery exercise and answer

Order processing can use an acknowledged work queue with durable intent and stable event IDs. Per-order assignment changes need an ordered version and stale-update rejection, not just a FIFO label. Location events may suit a retained partitioned log because dispatch, maps and analytics need independent progress; choose retention from actual replay/privacy needs. Order history should live in an authoritative store rather than disappear when the task is acknowledged. This is a proposed design, not an attribution to DoorDash or Uber Eats.

Explain-out-loud test

Trace a crash after effect but before ack, explain what the unique event row protects, compute the 120,000-message backlog and 1,110-attempt/s budget, then explain how a DLQ replay can violate ordering if identity and group state are ignored.

Continue with multi-region recovery. Learning plan.

Optional case-study pointers and onward study

The original Netflix Keystone pointer and Uber consumer-proxy pointer remain optional historical reading, not support here for the old SQS throughput ceiling, migration speedup or Mumbai outage story. Kafka MirrorMaker, MSK Replicator and Confluent Cluster Linking are optional replication topics, not implemented by this fixture or guarantees of identical regional semantics. RabbitMQ consistent-hash exchange is likewise an optional routing topic: routing a key does not alone prove ordered effects.

Queue depth/lag connects to observability; retry storms and DLQ overflow to incident response; partition/ISR behavior to Kafka internals; event sourcing, CQRS and sagas to event-driven design; destination budgets to rate limiting; and explicit asynchronous boundaries to system-design reviews. These connections preserve the original learning route without endorsing its outdated session numbers or making linked lessons part of this review.

Sources and review boundary

Full current-body review on 2026-10-10 reverified saved primary passages captured on 2026-10-07 through 2026-10-09 (not a fresh network retrieval); mutable documentation is bound to saved source hashes in the accompanying claim evidence. Numerical examples are explicit teaching assumptions. The bounded Python examples were run locally; no cloud/distributed deployment, PostgreSQL cluster, broker, object store, load benchmark or regional drill was executed. Compilation and diagram parsing do not establish deployed-system correctness.

• Sqs Visibility — Visibility expiry permits redelivery.; 120,000 is an in-flight-message quantity, not messages-per-second throughput.
• Sqs Quotas — SQS Standard documentation does not support the old 120K/s cap.; FIFO throughput depends on mode rather than one universal 3K/s cap.
• Sqs Fifo — SQS FIFO producer deduplication has a five-minute interval.
• Sqs Dlq — SQS redrive policy bounds receives before DLQ transfer.; Standard DLQ retention preserves original enqueue age.
• Sqs Create — SQS source and DLQ types must match.
• Rabbit Confirms — Publisher confirms and consumer acknowledgments have separate boundaries.
• Rabbit Quorum — RabbitMQ quorum queues use majority agreement.
• Apache Kafka 4.1 design — Exactly-once external effects need destination cooperation, not necessarily an all-Kafka boundary.; Destination output and consumption progress can be committed together.