Designing for Scale · Chat and Real-time Communication
Transport, durable acceptance and per-conversation order. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: advanced
By the end, you should be able to
- Transport, durable acceptance and per-conversation order
- The database commit succeeds, live Pub/Sub delivery fails, and the sender retries. How do you avoid both a lost message and a duplicate?
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 12 sections
The problem this post solves
A chat system must distinguish accepting a message, routing it to a connected device, persisting it on that device and showing it to a person. “Sent” cannot mean all four at once. Assume a product needs private conversations, group rooms, recent history and reconnect recovery. Specify maximum group size, payload size, retention, offline behavior and the meaning of each acknowledgment before choosing a transport or database.
Many long-lived connections change capacity planning, but do not make CPU irrelevant. TLS, compression, encryption, fan-out and authorization can be expensive. Connection memory, file descriptors, network bandwidth and downstream storage are separate budgets. A claim such as “one million connections per server” needs a measured workload and configuration; it is not a property of chat architecture.
First principles
HTTP does not necessarily close its connection after each response. Polling can reuse connections; long polling waits for an event or timeout; server-sent events stream server-to-client messages over HTTP and can pair with HTTP requests for sending. WebSocket provides two-way framed communication after its handshake, making it useful for interactive bidirectional traffic. RFC 6455 specifies that transport; it does not mandate WebSockets for every real-time product. Choose based on directionality, browser/proxy support, message rate and recovery semantics, not the false premise that every poll repeats TCP/TLS.
A persistent channel alone does not make delivery durable. Redis Pub/Sub's documented at-most-once behavior loses a publication missed by a disconnected subscriber. Keep a recoverable message log/history when the product promises offline delivery or replay. Transport liveness and application progress are different: a TCP ACK is not a read receipt.
Connection Management
Estimate open connections by concurrently connected devices, not registered accounts or daily users. As a scenario, 100,000 devices using an assumed 20 KiB of application/kernel state each consume about 1.91 GiB before other processes, buffers and headroom. The 20 KiB is a sizing input to measure, not an industry benchmark. File-descriptor limits and ephemeral ports depend on which side owns which sockets; inspect the actual proxy and process topology rather than assuming a universal limit of 1,024.
The Load Balancer Trap
An established WebSocket session stays attached to its backend while that connection lives, but a reverse proxy can remain in the data path throughout. Its own memory, timeout and buffer limits matter. Balance new connections using measured load, and drain old instances without treating socket count as equivalent to work: one busy room can consume more bandwidth than thousands of idle sockets. A reconnect may land on another server; resumable session state must not depend on the original process still existing.
Use bounded exponential backoff and random jitter on reconnect. If 100,000 clients spread one retry uniformly across 30 seconds, the mean retry arrival is about 3,333/s, not a guarantee about peak arrival or server capacity. Cap retries, honor overload signals and budget the re-authentication/history-read work as well as connection establishment. Heartbeat frequency trades detection latency against network/battery cost and must fit intermediary idle timeouts.
The Delivery Architecture
Figure text equivalent — labels and relationships
- Node “Authenticated sender and operation ID” (square).
- Node “Conversation authority” (square).
- Node “Message history plus outbox” (cylinder).
- Node “Durable acceptance ACK” (square).
- Node “Outbox delivery worker” (square).
- Node “Session directory” (square).
- Node “Recipient gateways” (square).
- Node “Authorized recipient devices” (square).
- Node “Resume cursor and application ACK” (square).
- “Authenticated sender and operation ID” leads to “Conversation authority”. Line: normal; ends: arrow point.
- “Conversation authority” leads to “Message history plus outbox”. Line: normal; ends: arrow point.
- “Message history plus outbox” leads to “Durable acceptance ACK”. Line: normal; ends: arrow point.
- “Message history plus outbox” leads to “Outbox delivery worker”. Line: normal; ends: arrow point.
- “Outbox delivery worker” leads to “Session directory”. Line: normal; ends: arrow point.
- “Session directory” leads to “Recipient gateways”. Line: normal; ends: arrow point.
- “Recipient gateways” leads to “Authorized recipient devices”. Line: normal; ends: arrow point.
- “Authorized recipient devices” leads to “Resume cursor and application ACK”. Line: normal; ends: arrow point.
- “Resume cursor and application ACK” leads to “Message history plus outbox”. Line: normal; ends: arrow point.
- Authenticate the sender and authorize the conversation action. Scope the operation ID to the tenant/sender and compare content on retries so a changed payload is not hidden.
- At the conversation authority, commit the message, its sequence and a recoverable delivery record atomically. Only then return the product's durable-acceptance acknowledgment.
- Delivery workers find current device sessions and publish to the appropriate gateways. A live Pub/Sub path may be an optimization, not the only copy of the message.
- Gateways recheck the applicable membership/authorization policy and bound per-device buffers. Devices deduplicate repeated message IDs and track a contiguous received sequence.
- On reconnect, fetch missing history after the acknowledged cursor. If sequence 42 arrives before 41, do not advance a contiguous cursor to 42 and permanently skip 41.
A sender retry after commit returns the stored message/sequence rather than creating another message. A delivery retry can resend that message without creating a second conversation entry. The outbox's delivery semantics and the device's acknowledgment semantics are separate; exactly one database row does not imply exactly one network transmission or one display.
The Session Service
Represent a user's potentially many devices as sessions with a connection ID, gateway and session generation. Treat presence as a lease-like hint with expiry, not a perfect statement about whether someone is watching. If session 7 disconnects after session 8 has replaced it, conditional removal must not delete session 8. Otherwise an old teardown can erase a live route.
Push presence changes only to interested observers or batch/pull visible contacts. For a social graph, update work is proportional to the subscribers actually notified; it is not universally quadratic. Ten thousand changes with 50 interested observers each imply 500,000 notifications before coalescing. Pulling 50 visible contacts every ten seconds shifts work to reads and may show stale state. Neither choice produces perfectly accurate green dots under partitions; state the staleness and privacy policy.
Scaling Group Chat (The Fan-out Problem)
Topic/gateway fan-out publishes once per relevant gateway and lets each gateway distribute to its local sessions. For 100,000 online recipients across 100 gateways, routing can use roughly 100 gateway deliveries rather than 100,000 independent directory lookups. It still requires delivering the recipient payloads: a 1 KiB message to 100,000 devices is about 97.7 MiB of payload before framing, TLS, retries and replication. Topic routing changes where the multiplication happens; it does not eliminate it.
Keep subscriptions aligned with authorization changes. A removed member's stale gateway subscription must not continue receiving private messages. Isolate hot rooms, pace backlog recovery, and choose whether slow clients receive a bounded queue, a dropped live hint with history recovery, or a disconnect requiring resume. Never allow one client to accumulate an unbounded memory buffer. Large rooms may use different product semantics—such as delayed history or limited presence detail—rather than promising every event instantly to everyone.
Storage Selection
Model the access pattern first: recent messages for one tenant/conversation in descending
sequence, plus edits, deletion, search and retention. A relational starting schema might
use (tenant_id, conversation_id, sequence) as a unique ordered key, with a separate unique
sender operation ID for retries. A matching B-tree can support ordered limited reads;
PostgreSQL 16 documents that index ordering can avoid sorting all matching rows. Thus chat
is not inherently incompatible with relational storage.
A wide-column design may partition by conversation and time bucket to bound partition size, with sequence/time as clustering order. It changes cross-partition reads, secondary queries, uniqueness and transaction options. LSM designs also incur compaction/write amplification; “B-tree has amplification, therefore Cassandra solves it” is not a workload proof. Compare measured write/read load, hot conversations, recovery, operator experience and required atomic boundaries. Search may be a derived index rather than the message source of truth.
Sortable timestamps or Snowflake-like IDs are identifiers, not an automatic conversation ordering authority. Clocks can skew and concurrent senders have no single natural order. Specify the chosen scope—per sender, per conversation or causal dependencies—and allocate or validate sequence accordingly. A single conversation sequencer needs its own failover and fencing design; adding a column does not establish distributed safety.
Failure modes
- Commit succeeds, ACK is lost: retry the same sender operation ID and return the committed message; compare payloads to reject conflicting reuse.
- History commit succeeds, live publish fails: replay the outbox and let offline devices resume from history. Redis Pub/Sub alone cannot recover a missed publication.
- Gateway dies: expire its sessions, reconnect with jitter and resume. Restore authentication and permissions, not only the old device cursor.
- History retention passes the cursor: return an explicit history-gap/resync result. Do not pretend the device received data that has been removed.
- Recipient membership changes: enforce the product's access policy on replay as well as live delivery; a cached session route is not authorization.
Worked delivery exercise
Alice's operation op7 commits as conversation sequence 41. The acceptance reply and live
publication are lost. Her retry reaches a new API server. Bob reconnects with contiguous
cursor 40 but receives a live copy of sequence 42 first. Explain each recovery step.
Acceptance, replay and ordering answer
The new server looks up Alice's scoped op7, verifies the payload and returns sequence 41.
It does not allocate 43. Durable delivery retries sequence 41. Bob buffers or marks 42 as
out of order and requests history after 40, obtaining 41 and possibly another copy of 42.
He deduplicates by stable message ID and advances the contiguous cursor only after closing
the gap. The UI may show a temporary gap indicator, but must not equate transport arrival
order with conversation order. None of this implies that Bob has read the message; that
requires a separately defined application receipt.
What to carry forward
Separate transport choice, durable acceptance, session routing and conversation ordering. A directory is a live-routing hint; the message history is the recovery authority. Choose storage and fan-out by access patterns and measured budgets, not a universal database or connection-count slogan. This is a conceptual architecture with checkable arithmetic and failure traces, not a deployed chat service or throughput benchmark.
Sources and review boundary
Checked 2026-10-07: RFC 6455 (2011) for bidirectional WebSocket framing; RFC 9293 (2022) for the TCP transport boundary; WHATWG server-sent events for HTTP event streams; Redis Pub/Sub semantics for disconnected-subscriber loss; Kafka 4.1 design for external-effect boundaries; PostgreSQL 16 ordered indexes for ordered limited reads. WHATWG and Redis pages are access-dated snapshots, not claimed release pins. No socket server, broker, database cluster or reconnect load test was run.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.