S128 · Design a Chat System — WebSockets, Delivery, Presence
Realtime bidirectional messaging at billions/day scale — WebSockets, routing tables, at-least-once delivery, ordering, presence, offline pushes, and the reconnect thundering herd that took Slack down.
🎯 Design a WhatsApp/Slack-class chat system end-to-end, covering realtime delivery, ordering, presence, offline fallback, and reconnect-storm safety.
Why this session exists
A chat system is where system design gets fun — and dangerous. It's the first problem that forces you to reason about stateful services, at-least-once delivery, ordering, presence, offline behaviour, flaky mobile networks, and message storage at billions-per-day scale. WhatsApp, Slack, Discord, and Messenger all solve the same core problem with different bets. This session gives you the shared skeleton and the specific traps that took down real systems in the last five years.
- Explain why WebSocket load-balancing is fundamentally different from HTTP load-balancing.
- Draw the five-layer chat architecture (gateway, chat servers, routing, queue, store) from memory.
- Design at-least-once delivery + client-side dedup using message IDs and ACKs.
- Choose between fanout-on-write and inbox-pull for group chat given group size + activity.
- Explain and defend against the reconnect thundering herd that took Slack down.
Prerequisites
- S126 · System Design Framework
- S075 · Message Queues (Kafka, SQS, RabbitMQ)
- S063 · Caching · S066 · Load Balancers · S070 · Sharding
(a) Intuition · 5 min
The postal service (HTTP) is one-shot: address an envelope, drop it in a box, forget about it. Each letter is independent. Great for websites; terrible for a conversation. Imagine texting your friend by mailing letters and waiting for a reply mailing back.
A phone line (WebSocket) is a live circuit. You dial once, keep the line open, and either party can talk instantly. Now a conversation flows. That's the whole reason WebSockets exist.
The fundamental question: how does a server push a message to a client in real time? Long polling wastes CPU. Server-Sent Events are one-directional. WebSockets — a bidirectional persistent TCP connection upgraded from HTTP — are the industry default.
Each chat server holds ~100k–1M open sockets. Users are sharded across many chat servers via a routing layer (usually Redis or ZooKeeper) that knows which server holds which user's live connection.
- The routing layer — 'which server holds Alice?' — is the beating heart. Every major chat outage of the last decade has touched it.
- The reconnect storm — millions of clients that lost their socket try to reconnect in the same second. Without jitter, this DoSes your own control plane.
- The group-chat fanout — a naive 'send to 500 members' explodes writes; the inbox-pull model saves you.
- 2009WhatsApp launches on Erlang~50 engineers scale to hundreds of millions of users on Erlang's actor model + FreeBSD. Sets the industry template.
- 2011WebSocket RFC 6455The protocol is standardised. HTML5 clients can hold live bidirectional connections without hacks (long polling, Comet, Flash sockets).
- 2015Discord launchesElixir + Cassandra + custom voice stack. Later migrates message store to ScyllaDB then to a custom Rust engine.
- 2020COVID surge · every chat system 3–5×Slack, Zoom, Teams all hit unprecedented concurrent-connection counts. Reconnect storms become the top on-call topic.
- 2021Slack outageA network blip triggers a client reconnect flood that DoSes the routing layer. Jittered reconnects become an industry-wide fix.
- 2024Realtime + AI overlayEvery chat now embeds AI (Copilot, Claude in Slack). The chat system becomes the transport for LLM streams too.
(b) Visual walkthrough · 15 min
The five-layer architecture
Alice sends "hi" to Bob — the full path
Client sends {to: bob, text: 'hi', client_seq: 42} over her WebSocket. ~5–30 ms.
CS1 assigns a Snowflake message_id (sortable by time), writes to Cassandra. ~5–15 ms.
For durability + async fanout (notifications, analytics). ~2–5 ms.
CS1 looks up Bob in Redis → 'Chat Server 2'. ~1–2 ms.
'Deliver this to Bob.' Internal gRPC or Kafka topic. ~2–5 ms.
CS2 sends over Bob's WebSocket. ~5–30 ms.
Bob's client ACKs receipt; CS2 → CS1 → Alice. +30–60 ms.
End-to-end: 80–150 ms for online users on good networks.
The offline path — never lose a message
Live WebSocket path
- Redis says 'Bob on CS2'
- RPC CS1 → CS2
- Push over WebSocket
- ACK closes the loop
Push notification + inbox pull
- Redis says 'Bob has no connection'
- Store to Cassandra
- APNs / FCM notification
- Bob opens app → pulls messages > last_read_id
Idempotent replay
- Client's local last_read_id is source of truth
- Server returns everything > last_read_id
- Client deduplicates by message_id
- No double-delivery visible to user
Group chat — inbox pull is the win
Every serious chat platform (WhatsApp, Slack, Discord) uses inbox-pull for groups. Pure fanout-on-write multiplies storage by average group size (often 50–500×).
Message ID choice — Snowflake
Why Snowflake IDs won
Sharding the message store
"WebSockets give a persistent connection, so message delivery is reliable. If the socket is open, the message arrived."
A successful send() means the bytes entered the kernel buffer — not that the peer received them, and certainly not that the application processed them. TCP connections die silently (NAT timeouts, radio handoffs, sleeping phones) and can stay "open" from the sender's view for minutes. Reliable delivery requires application-level acknowledgements and per-message IDs, exactly as if the transport were unreliable.
Because TCP genuinely does guarantee reliable, ordered delivery within a connection, and that guarantee is what everyone learned. What it does not guarantee is that the connection still exists or that the application on the other side did anything with the bytes. On a wired LAN failures are fast and loud; on mobile networks they are slow and silent, which is exactly the environment a chat app lives in.
Reason through the delivery chain and locate where the guarantee actually ends:
sender app -> kernel buf -> network -> server kernel -> server app
-> persisted to DB -> fanout -> recipient kernel
-> recipient app -> rendered on screen
# send() returns after step 1 of 9.
# TCP ACK covers up to step 4.
# Only an APPLICATION ack from the recipient covers the whole chain.
#
# Which is why every real chat protocol carries three states:
# sent (server persisted) / delivered (device acked) / read (user saw)
# Those checkmarks are not UI decoration -- they are the protocol.Why must a chat system assign message ordering on the server, and why is a wall-clock timestamp insufficient?
- 1Participants in a conversation must see messages in the same order, or the conversation becomes incoherent — replies appearing before what they reply to.forced by · message semantics depend on sequence; this is a correctness requirement, not cosmetics
- 2Client clocks cannot supply that order: they drift, are user-settable, and span timezones. Two clients can disagree by minutes.forced by · there is no trusted global clock across untrusted devices
- 3Server clocks are better but still insufficient in a distributed deployment, because two servers handling two participants have clocks that differ by some skew, and NTP bounds that skew only loosely.forced by · physical clock synchronisation has irreducible uncertainty; you cannot totally order events by timestamps you cannot trust to the millisecond
- 4What is actually required is a total order per conversation, not globally across the whole system. Two messages in different chats have no ordering relationship anyone can observe.forced by · observers are scoped to a conversation, so consistency only needs to hold within that scope
- 5Scoping to a conversation makes the problem tractable: route all messages for a conversation to one partition and assign a monotonically increasing per-conversation sequence number there.forced by · a single assigner per partition gives a total order with no distributed coordination at all
- 6Clients then sort by sequence number, and a gap in the sequence is a positive signal that a message is missing — which is what enables reliable catch-up after reconnect.forced by · a dense integer sequence makes absence detectable, which timestamps never do
Therefore per-conversation sequence numbers, assigned server-side at a single partition, give both ordering and gap detection — neither of which timestamps can provide.
And note what this predicts: partitioning by conversation ID also solves fanout locality (all participants' delivery state lives together) and creates the one real hot-key risk (a very large group chat concentrating on one partition). Both consequences follow from the same partitioning decision, which is why "partition by conversation" is the single highest-leverage choice in this design.
A chat system is two loosely-coupled subsystems. The first is storage: an append-only, sequence-numbered log per conversation — simple, partitioned, and the source of truth. The second is delivery: getting new entries to a set of devices that are mostly offline, on flaky networks, behind push services you do not control.
Storage is the easy half and it is where people spend their time. Delivery is where the actual engineering is, and every hard requirement — ordering, dedupe, offline catch-up, read receipts — lives there.
- Connection state (which user is on which gateway) is ephemeral and belongs in a fast store like Redis, keyed by user, with TTL. Never in the durable database.
- Client-generated message IDs enable idempotent retries. Without them, a retry after an ambiguous timeout duplicates the message — and ambiguous timeouts are the normal case on mobile.
- Offline delivery is a queue per recipient plus a push notification, not a stored socket. Assume every device is offline by default.
- Group chats change the cost model entirely: one send becomes N deliveries. Above a few hundred members, fanout-on-write stops making sense.
Fire this model the moment you see: any real-time delivery requirement · duplicate messages after network hiccups · messages appearing out of order · a design that treats an open socket as delivery confirmation · a group-size limit that seems arbitrary (it is a fanout limit).
How do clients receive new messages — long polling, server-sent events, or WebSockets?
WebSockets for chat, with long polling as a fallback, because the fallback is not optional in the real world. The part people underestimate is the operational cost of stateful connections: a deploy that drops every socket produces a reconnection thundering herd, so you need staggered draining, jittered client backoff, and a gateway tier you can restart independently of application servers.
The transferable lesson is that the protocol choice is the easy half of this decision. Persistent connections move complexity from the request path into your deployment and capacity model — and that is where you should evaluate them.
(c) Hands-on · 25 min
Build a minimal but real chat server with FastAPI + WebSockets. Two clients exchange messages, offline-queue works, ACKs flow.
Test client (run in two terminals):
#!/usr/bin/env python3
# client.py — a tiny CLI chat client.
# pip install websockets
import asyncio
import json
import random
import sys
import websockets
BACKOFF_BASE_MS = 500
BACKOFF_MAX_S = 30
async def reader(ws) -> None:
async for m in ws:
print(f"\n<< {m}\n>>> ", end="", flush=True)
async def chat(user_id: str, peer: str) -> None:
attempt = 0
seq = 0
while True:
try:
async with websockets.connect(f"ws://localhost:8000/ws/{user_id}") as ws:
attempt = 0
asyncio.create_task(reader(ws))
while True:
line = await asyncio.to_thread(input, ">>> ")
seq += 1
await ws.send(json.dumps({"to": peer, "text": line, "client_seq": seq}))
except Exception as e:
attempt += 1
# Jittered exponential backoff — the fix Slack shipped in 2021.
base = min(BACKOFF_MAX_S * 1000, BACKOFF_BASE_MS * 2 ** attempt)
wait_ms = random.uniform(0, base)
print(f"\n[reconnect in {wait_ms/1000:.1f}s: {e}]")
await asyncio.sleep(wait_ms / 1000)
# Usage:
# python client.py alice bob (terminal 1)
# python client.py bob alice (terminal 2)
asyncio.run(chat(sys.argv[1], sys.argv[2]))What each block is doing
Anatomy of the chat server
Add a /group/{group_id} endpoint that accepts a member list and fans out one message to every online member's WebSocket. Time the send with time.perf_counter() for groups of size 10, 100, 1000. Note how the sender's tail latency grows linearly with group size. Then implement an inbox-pull variant: one write to a group_messages list, each client polls "give me messages after X." Compare tail latencies + storage.
(d) Production reality · 15 min
A maintenance-triggered network blip disconnected millions of WebSocket clients simultaneously. Clients hit the reconnect endpoint in a synchronised thundering herd — millions of retries per second — which DoSed the routing layer that was supposed to reassign connections.
Every retry made the outage worse. The routing layer couldn't converge.
Added randomised jitter to client reconnect delays: delay = min(30 s, 500 ms · 2attempt) × rand(0, 1). Staggered reconnect windows on the server side; per-IP rate limits on connection attempts.
Every major chat client shipped a similar change in the following six months.
conversation_id worked well for most channels. But their public write-up documented that one very busy channel (a support/general channel with 10+ msgs/s) became a hot partition — read/write latency degraded and compaction fell behind.Changed partition key to (conversation_id, time_bucket) where time_bucket is a week-index. A busy channel now spreads across a new partition every week; old partitions become read-only and compaction-friendly.
Discord later moved the message store from Cassandra to ScyllaDB and then to a custom Rust engine, but the bucketing insight carried over.
Root cause: WebSocket died mid-send on a flaky network, the server thought the message was delivered, and no client-side ACK was required. Fix: at-least-once delivery — server retries until it gets an ACK; client deduplicates by message_id. Client keeps a small local "outbox" of unacked messages and replays on reconnect.
Common failure modes
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- Why is chat load-balancing fundamentally different from HTTP load-balancing?
- When you send a message, walk through the full delivery path in one breath.
- What is the one client-side change that prevents reconnect storms?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.