S075 · Message Queues — SQS, RabbitMQ, Kafka as Queue

Module M08: Distributed Systems · Session 75 of 130 · Track: SYS 📨

What you'll be able to do after this session

• Explain Message Queues to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S048 — Kafka — Topics, Partitions, Consumer Groups

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Message Queues Explained (Fireship — 100 Seconds series, extended) — RabbitMQ in 100 seconds + why queues exist.
• 🎥 Deep dive (30–60 min) · SQS, Kafka, RabbitMQ — What's the Difference? (Hussein Nasser) — 45-minute deep-dive comparing the three big players.
• 🎥 Hands-on demo (10–20 min) · RabbitMQ in 20 Minutes (TechWorld with Nana) — set up producer + consumer + queue live.
• 📖 Canonical article · AWS SQS Developer Guide — How Amazon SQS Works — canonical reference for the queue model.
• 📖 Book chapter / tutorial · RabbitMQ Tutorial 1 — Hello World (rabbitmq.com) — the official six-tutorial series is the best hands-on intro.
• 📖 Engineering blog · Uber — Building Reliable Reprocessing and Dead Letter Queues with Kafka — production-grade DLQ patterns.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Decouple producers from consumers.

Imagine a busy restaurant kitchen. Customers place orders at the counter, but they don't stand there watching the chef cook — the order goes on a ticket, joins a line of tickets, and the chef picks the next one when free. If the chef gets hit by a bus mid-service, another chef walks in and picks up where they left off — the tickets are the source of truth. That queue of tickets decouples the fast, bursty front-of-house from the slow, careful kitchen.

Message queues exist because production systems are made of parts that operate at wildly different speeds. Your web server handles 10 000 requests/sec; your PDF-generation service handles 5/sec. Directly connecting them means the web server blocks or drops traffic. Put a queue in the middle and the web server just drops the job in, returns instantly, and the PDF service consumes at its own pace.

Beyond speed matching, queues give you retries (if the consumer crashes, the message is redelivered), fan-out (one message to many consumers), buffering (survive traffic spikes), and decoupling (producers know nothing about consumers). The three big players you'll meet: SQS (AWS-managed, dead-simple, at-least-once), RabbitMQ (self-hosted, rich routing with exchanges, mature), and Kafka (log-based, replayable, high throughput — S076 revisits this).

Gotcha to carry forever: almost every queue delivers at-least-once, which means your consumer will occasionally see the same message twice. Design consumers to be idempotent (processing twice = processing once) — this is not optional, and this one insight prevents 90% of production queue bugs.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — the visibility-timeout dance (SQS style).

1. Producer sends message M1 to the queue.
2. Consumer A calls ReceiveMessage. SQS returns M1 and marks it invisible for 30 seconds (the visibility timeout).
3. Consumer A starts processing (say, generating a PDF that takes 20 s).
4. If A finishes and calls DeleteMessage within 30 s → message is gone. Done.
5. If A crashes at second 15 → nothing calls DeleteMessage → at second 30, M1 becomes visible again and Consumer B picks it up. At-least-once redelivery.
6. If M1 fails 5 times (all consumers crash on it — usually a bad message), SQS moves it to the Dead Letter Queue for a human to investigate.

Comparison of the big three:

Rule of thumb: SQS if you're on AWS and want zero ops. RabbitMQ if you need rich routing (topics, headers, priorities) or aren't on AWS. Kafka if you need replay, high throughput, or event sourcing (that's the "queue as log" pattern — S076 covers this).

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Run RabbitMQ in Docker, write a producer + a worker, and see redelivery in action.

# consumer.py — pip install pika
import pika, time, os, sys, json, random
 
conn = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
ch = conn.channel()
ch.queue_declare(queue='pdf_jobs', durable=True)
 
def callback(ch_, method, properties, body):
    job = json.loads(body)
    print(f"[worker {os.getpid()}] processing job {job['id']}")
    time.sleep(2)
    # Simulate a crash 20% of the time BEFORE acking
    if random.random() < 0.2:
        print(f"[worker {os.getpid()}]  CRASH before ack for job {job['id']}!")
        sys.exit(1)
    ch_.basic_ack(delivery_tag=method.delivery_tag)
    print(f"[worker {os.getpid()}]  done job {job['id']}")
 
ch.basic_qos(prefetch_count=1)  # fair dispatch: one at a time
ch.basic_consume(queue='pdf_jobs', on_message_callback=callback)
print(f"[worker {os.getpid()}] waiting...")
ch.start_consuming()

# producer.py
import pika, json
conn = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
ch = conn.channel()
ch.queue_declare(queue='pdf_jobs', durable=True)
for i in range(20):
    ch.basic_publish(
        exchange='',
        routing_key='pdf_jobs',
        body=json.dumps({'id': i, 'title': f'invoice-{i}'}),
        properties=pika.BasicProperties(delivery_mode=2),  # persistent
    )
print('20 jobs queued')
conn.close()

# Terminal 1: broker
docker run -d --name rabbit -p 5672:5672 -p 15672:15672 rabbitmq:3-management
sleep 8
 
# Terminal 2 and 3: two workers
python consumer.py &
python consumer.py &
 
# Terminal 4: produce 20 jobs
python producer.py

Observe (checklist):

1. Both workers pick up jobs alternately (fair dispatch thanks to prefetch_count=1).
2. When a worker "crashes" (simulated sys.exit), the un-acked message is redelivered to the other worker — no message is lost.
3. Open http://localhost:15672 (guest/guest), watch the queue drain in real time.
4. Kill both workers mid-run, then restart them — jobs still there because durable=True + delivery_mode=2 persist to disk.
5. Because of the crash simulation, the same job_id will occasionally be processed twice. This is why your process() must be idempotent — track job_ids you've already handled in a DB.

Try this modification: add a Dead Letter Exchange: ch.queue_declare('pdf_jobs', arguments={'x-dead-letter-exchange': 'dlx'}), then send a job with a deliberately un-processable payload and confirm it lands in the DLQ after N nacks.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Message queues are simple until they aren't. The pain points are always the same three: duplicate delivery, poison pills, and back-pressure.

Duplicate delivery. SQS says "at-least-once," Kafka says "at-least-once (or effectively-once with idempotent producer + transactional consumer)," RabbitMQ says "at-least-once with acks." All of them will occasionally redeliver the same message. Your consumer must be idempotent — track processed message IDs in a DB (with an index) and skip repeats. Stripe's famous idempotency-key API works exactly this way, powered internally by a queue.

Poison pills. One malformed message that crashes every consumer that tries to parse it. Without a DLQ, this message gets redelivered forever, blocking the queue and eating consumer capacity. Always configure a max-redelivery count and a DLQ. Then alert on DLQ.depth > 0.

Back-pressure & queue length as a health signal. If your queue length is growing, your consumers can't keep up — full stop. Monitor queue.depth and oldest_message_age. A queue that grows unboundedly is heading for OOM (RabbitMQ) or throttling (SQS). Autoscale consumers on queue depth, not CPU.

Real war stories:

• Cloudflare 2019 had a Kafka consumer lag disaster: a bad regex in WAF rules caused every worker to spin up 100% CPU; consumer lag on the config-distribution Kafka topic grew until the queue paged them. Rollback was 30 min.
• Uber's Kafka DLQ pattern (linked in pre-read) uses tiered retry topics — retry after 1 min, then 10 min, then to a "manual review" topic. This prevents transient errors from filling the DLQ.
• RabbitMQ split brain: a cluster partition causes queues to diverge. RabbitMQ's mirroring modes have subtle failure modes; use quorum queues (Raft-based) in modern versions.

Ordering pitfalls. SQS standard queues are unordered; if you need FIFO, use SQS FIFO queues (10× more expensive, lower throughput). Kafka only guarantees order within a partition — if you shard by user_id, all events for one user go to one partition and stay ordered.

Golden rule from ops: the queue is not your database. Don't try to query messages, sort them, or store business state in them. Move state to a DB; use the queue for delivery only.

(e) Quiz + exercise · 10 min

1. What is a "visibility timeout" and why is it needed for at-least-once delivery?
2. Why must consumers be idempotent even when the queue promises "exactly once"?
3. What is a Dead Letter Queue and what problem does it solve?
4. When would you choose Kafka over SQS?
5. Give one concrete metric that indicates your consumers can't keep up with producers.

Stretch (connects to S048 · APIs and prior "async" work): Design an async job API where the client POSTs a job, gets a job-id, and can poll GET /jobs/{id} for the result. Sketch the tables and queue topology, and specifically explain how you make the whole thing survive a consumer crash mid-processing.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Message Queues? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S076): Multi-Region — Active-Passive, Active-Active, Failover
• Previous (S074): Sharding & Partitioning Strategies
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.