Designing for Scale · Chat and Real-time Communication

The problem this post solves

A chat application is a system where the server pushes data to the client, rather than waiting for the client to ask.

In a standard web application, the client opens a connection, asks for data, receives it, and closes the connection. In a chat application, millions of clients open connections and hold them open permanently, waiting in silence for a message that might arrive hours later.

This flips standard web architecture upside down. Load balancers must route by connection state rather than by request. Servers are bound by memory, not CPU. And when a message is sent in a large group channel, one write turns into a million immediate outbound network calls.

First principles

Connection Management

A standard stateless API server handles requests. A stateful WebSocket server holds connections. This distinction breaks naive scaling.

If you have 10 million active users, you have 10 million open TCP connections. Each connection consumes a few kilobytes of RAM. A single modern server can hold about 1 million concurrent connections (the C10K problem is long dead; we are in the C10M era). The bottleneck is memory and file descriptors, not CPU.

The Load Balancer Trap

With stateless HTTP, a load balancer can route every single request to a different backend server.

With stateful WebSockets, the connection is bound to a specific server. The load balancer's job changes: it must balance the connections evenly, and then get out of the way. Once established, the client speaks directly to that specific WebSocket server.

The Delivery Architecture

When Alice sends a message to Bob, the architecture must answer one question: Which server is holding Bob's connection?

1. Alice sends the message to her connected server (W1).
2. W1 forwards it to the stateless Chat API to handle validation, persistence, and routing.
3. The Chat API queries the Session Service: "Where is Bob?"
4. The Session Service says Bob is connected to W3.
5. The Chat API publishes the message to a message bus (like Redis Pub/Sub), targeted at W3.
6. W3 receives the message and pushes it down the open WebSocket to Bob.

The Session Service

The Session Service is a fast key-value store (Redis) mapping user_id to websocket_server_id. Every time a user connects, the WebSocket server writes to Redis. When they disconnect, it deletes the key.

Scaling Group Chat (The Fan-out Problem)

1-to-1 chat is simple: one write, one read. A group chat with 100,000 members is a fan-out problem: one write, 100,000 immediate outbound pushes.

If the Chat API tries to find all 100,000 users in the Session Service and send 100,000 individual messages to the message bus, the API server will choke.

Storage Selection

Chat history is an extreme write-heavy workload with an unusual read pattern.

• Recent messages are read constantly.
• Old messages are almost never read, unless a user explicitly searches or scrolls back.

Because the write volume is relentless, a standard B-Tree relational database will suffer from write amplification. This is the textbook use case for a Wide-Column store (LSM Tree) like Cassandra or ScyllaDB.

The schema is optimized entirely for the read path: SELECT * FROM messages WHERE channel_id = X ORDER BY message_id DESC LIMIT 50;. The message_id must be sortable by time (e.g., Snowflake IDs), allowing the database to serve recent history sequentially from disk without a separate sorting step.

Failure modes

Connection limits. Forgetting to increase ulimit for file descriptors on the Linux servers. The server has plenty of RAM, but the OS refuses to open more than 1,024 connections.

The Thundering Herd. When a WebSocket server crashes, 1 million clients disconnect simultaneously. If their code says onClose() { reconnect(); }, 1 million clients will immediately hit the load balancer at the exact same millisecond. The clients must use exponential backoff and jitter (adding a random delay) before reconnecting.

Ordering guarantees. If Alice sends message A then message B, Bob must see A then B. In a distributed system, B might arrive at the message bus before A. The client UI must sort incoming messages by their globally generated ID before rendering them, rather than relying on network arrival time.

What to carry forward