S128 · Design a Chat System — WebSockets, Delivery, Presence

Module M15: System Design · Session 128 of 130 · Track: SYS 💬

What you'll be able to do after this session

• Explain Design a Chat System to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S075 — Message Queues — SQS, RabbitMQ, Kafka as Queue

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · System Design: Chat Application — clean 15-min chat system overview.
• 🎥 Deep dive (30–60 min) · Design WhatsApp or Facebook Messenger — full interview walkthrough.
• 🎥 Hands-on demo (10–20 min) · Build Real-time Chat with WebSockets — end-to-end demo.
• 📖 Canonical article · WebSocket Protocol (RFC 6455) — the actual spec that powers real-time chat.
• 📖 Book chapter / tutorial · ByteByteGo — Design a Chat System — Alex Xu's canonical chapter on this.
• 📖 Engineering blog · Discord Engineering — How Discord Stores Billions of Messages — real production story from Discord.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Real-time at scale.

A chat system is where system design gets fun: it forces you to reason about real-time communication, at-least-once delivery, ordering, presence, offline behaviour, flaky mobile networks, and message storage at billions-per-day scale. WhatsApp, Slack, Discord, and Messenger all solve the same core problem with different trade-offs. The interview reduces to: "1-to-1 and group chat, message persistence, delivery/read receipts, online presence, mobile + web clients."

The fundamental question is: how does a server push a message to a client in real time? Long polling works but wastes CPU. Server-Sent Events (SSE) are one-directional. WebSockets — a bidirectional persistent TCP connection upgraded from HTTP — are the industry default. Once a client connects, the server keeps a socket open and can push messages the instant they arrive. Each chat server holds ~100k-1M open sockets; you shard users across many chat servers via a routing layer that knows which server holds which user's live connection.

The gotcha you must carry forever: WebSockets are stateful, and stateful services scale differently than stateless APIs. You cannot round-robin load-balance them. A message for user Alice must go to the specific server that holds Alice's socket. That routing table (usually Redis or a service like ZooKeeper) is the beating heart of every chat system, and it's the first thing that breaks under load. Every chat outage in the last decade — Slack, Discord, WhatsApp — has at some point involved either the routing layer or the connection layer failing to reconverge after a network blip.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Every serious chat system decomposes into 5 layers:

Worked example — Alice sends "hi" to Bob:

Total end-to-end: usually 80-150ms for online users on good networks.

Offline delivery: if Redis says "Bob has no live connection", CS1 sends a push notification through APNs/FCM instead. When Bob opens the app, his client does a "give me all messages after last_seen_id" pull from Cassandra.

Group chat trick: don't fan-out at message-send time to 500 group members. Instead, store one copy of the message keyed by (group_id, message_id). Each user maintains a per-group last_read_message_id. On open, they pull from last_read forward. This is called the inbox pull model and it's what WhatsApp, Slack, Discord all do — pure fan-out-on-write would multiply your storage by the average group size (often 50-500×).

Message ID choice: Snowflake IDs (Twitter's design) — 64 bits: timestamp(41) + machine_id(10) + sequence(12). Sortable by time, globally unique, generated locally without coordination. Every chat system uses some variant of this.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Build a minimal chat server with WebSockets in Python using FastAPI. Two clients can exchange messages in real time.

# pip install fastapi uvicorn
import asyncio, json, time
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
 
app = FastAPI()
 
# In-memory equivalent of Redis routing table:
# user_id -> WebSocket connection
connections: dict[str, WebSocket] = {}
# In-memory equivalent of Cassandra message store:
message_log: list[dict] = []
 
def snowflake_id() -> int:
    # Very simplified: real Snowflake shifts bits differently.
    return int(time.time() * 1000)
 
@app.websocket("/ws/{user_id}")
async def ws_endpoint(websocket: WebSocket, user_id: str):
    await websocket.accept()
    connections[user_id] = websocket
    print(f"[connect] {user_id}, total online: {len(connections)}")
    try:
        # Deliver any offline messages first (inbox pull)
        offline = [m for m in message_log if m["to"] == user_id and not m.get("delivered")]
        for m in offline:
            await websocket.send_text(json.dumps(m))
            m["delivered"] = True
 
        while True:
            raw = await websocket.receive_text()
            msg = json.loads(raw)
            msg["id"] = snowflake_id()
            msg["from"] = user_id
            msg["ts"] = time.time()
            message_log.append(msg)  # persist first
 
            recipient = msg["to"]
            target = connections.get(recipient)
            if target:
                await target.send_text(json.dumps(msg))
                msg["delivered"] = True
                # ACK back to sender
                await websocket.send_text(json.dumps({"ack": msg["id"], "status": "delivered"}))
            else:
                await websocket.send_text(json.dumps({"ack": msg["id"], "status": "queued"}))
    except WebSocketDisconnect:
        print(f"[disconnect] {user_id}")
        connections.pop(user_id, None)
 
# Run: uvicorn chat:app --reload --port 8000

Test client (run in two terminals):

# pip install websockets
import asyncio, json, sys, websockets
 
async def chat(user_id: str, peer: str):
    async with websockets.connect(f"ws://localhost:8000/ws/{user_id}") as ws:
        async def reader():
            async for m in ws:
                print(f"\n<< {m}\n>>> ", end="", flush=True)
        asyncio.create_task(reader())
        while True:
            line = await asyncio.to_thread(input, ">>> ")
            await ws.send(json.dumps({"to": peer, "text": line}))
 
asyncio.run(chat(sys.argv[1], sys.argv[2]))
# Usage: python client.py alice bob   (in one terminal)
#        python client.py bob alice   (in another)

Observe when you run:

1. Both clients connect; server prints their user_ids.
2. Type in Alice's terminal → appears in Bob's within ~10ms. ACK arrives back to Alice.
3. Kill Bob's terminal, type in Alice's — Alice gets status: queued. Reconnect Bob → he receives the queued message.
4. Watch server logs: each connect/disconnect updates the routing table.
5. If you kill the server, both clients disconnect — that's the stateful nature of WebSockets biting you.

Try this modification: add a group chat endpoint. Instead of to: single_user, accept to: group_id. On send, look up group members, then fan out through their live WebSocket connections. Notice how a group with 500 members × 1 message = 500 WebSocket writes. Discuss where fanout-on-write breaks down (hint: celebrity users, huge groups).

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the Slack outage of 2021. Slack went down for hours because their WebSocket layer failed to reconnect after a maintenance-triggered network partition. Clients hit the reconnect endpoint in a synchronised thundering herd — millions of retries per second — which took down the routing layer that was supposed to reassign connections. Root cause: no jittered exponential backoff on client-side reconnect logic. Fix: they added randomised jitter (1s + rand(0, 30s)) and staggered reconnect windows. Every real chat client since then does the same.

Common bugs top teams handle for:

• Reconnect thundering herd — always add jitter to client reconnect delays; server-side, apply per-user rate limits on connection attempts.
• Message ordering — if Alice sends messages M1, M2, M3 back-to-back and M2 hits a slower path, Bob might see M1, M3, M2. Fix: attach a monotonic client_seq per (user, conversation) so the recipient can reorder or request retransmit.
• At-least-once vs exactly-once — TCP guarantees delivery on one hop, but a WebSocket dying mid-send can drop a message. Client must ACK each message; server retries until ACK. Deduplicate on client using message_id.
• Read receipts + privacy — sending a "seen" receipt on every render costs bandwidth AND has social implications (WhatsApp made blue ticks toggle-able for a reason). Batch receipts (e.g. every 5 seconds), never on scroll events.
• Presence at scale — naive presence ("who's online?") requires broadcasting to everyone who has you as a contact — O(N × avg_contacts) writes on every status change. Solution: only compute presence lazily when a user opens a chat (pull, not push).
• Cassandra hot partitions — partitioning by conversation_id is standard, but if one group is very active (e.g. a support channel with 10 msgs/sec), that partition becomes a hotspot. Discord's actual fix: change partition key to (conversation_id, bucket) where bucket is time-based (e.g. bucket per week), so writes spread over time.
• APNs / FCM rate limits — Apple caps ~1000 notifications/sec per app. For breaking-news bulk sends, you must precompute and stagger, or you'll drop notifications silently.

Real-world scale for reference: WhatsApp reportedly handles ~100 billion messages/day (2020). Their infrastructure ran on ~50 engineers for years — testament to how far Erlang + smart architecture can get you. Discord: ~4 billion messages/day, stored in Cassandra (later ScyllaDB, then a custom Rust store). Slack: ~10 billion messages sent through its infrastructure per week. Same core patterns underlie all of them.

(e) Quiz + exercise · 10 min

1. Why can't you round-robin load-balance WebSocket connections the way you load-balance stateless HTTP APIs?
2. Describe the "inbox pull" model for group chat and why it's usually better than fan-out-on-write for large groups.
3. What is a Snowflake ID and why is its property of being sortable by time useful for a chat system?
4. Describe two ways message ordering can be violated between two clients, and how you'd fix each.
5. Design the reconnect logic for a mobile chat client that just lost network. What steps happen, in order?

Stretch problem (connects to S075 — Networking): In S075 you learned about TCP, UDP, and long-lived connections. Explain why WebSockets use TCP even though real-time applications often prefer UDP (games, video). What would break if you tried to build a chat system on top of UDP directly?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Design a Chat System? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S129): Design a Newsfeed / Recommender — Pull vs Push, Ranking
• Previous (S127): Design a URL Shortener — the Classic Warm-Up
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.