S061 · Networking I — TCP/IP, DNS, Sockets
How bytes actually cross the internet — from getaddrinfo() to the 3-way handshake to the router hop that decides your latency. The mental model every backend engineer needs before they can debug ‘why is this slow?’
🎯 Walk through a curl request end-to-end — DNS lookup, TCP handshake, packet routing, socket buffers — and be able to point at every layer where it can break.
Why this session exists
Every backend engineer eventually gets paged for a system that's "up" but nobody can reach it. The DB says green, the load balancer says green, the app process is running — and yet requests time out. Nine times out of ten the bug is somewhere in the seven layers below your code, in a place your framework has been quietly hiding from you. This session peels those layers open so the next time you see EHOSTUNREACH, ETIMEDOUT, or a 500 ms latency spike, you know exactly which layer to look at first.
- Walk through what happens between typing curl example.com and getting HTML back — DNS, TCP handshake, TLS, HTTP, FIN — naming every packet.
- Explain why TCP is a byte stream (not a message stream), and design a length-prefixed framing protocol that handles the resulting bug.
- Read the output of ss -tnp, tcpdump, and dig well enough to diagnose which layer a failure lives in.
- Recognise the top-5 production networking gotchas (TIME_WAIT flood, half-open connections, DNS TTL, Nagle+delayed-ACK, FD ceiling) in under 30 seconds.
- Write a working TCP echo server + client from memory in under 20 minutes.
Prerequisites
- S060 — OS Basics — Processes, Threads, Memory, FDs — sockets are file descriptors; the FD ceiling is the same ulimit.
- Basic comfort with the shell (S001–S003) — you'll be running
curl,ss,dig,tcpdump.
(a) Intuition · 5 min
Think about mailing a physical letter across the country. You write the message, put it in an envelope, address it, drop it in a postbox — and trust an invisible chain of trucks, planes, and sorting centres to hand it to the right person. Nobody in the middle knows the whole journey; each sorting centre just knows "for that ZIP code, throw it on this truck."
If the letter is important, you pay for registered mail: someone signs for it, you get a tracking number, and if the postal service loses it they'll retry. If you're just sending a birthday card, you drop it in the box and hope.
The internet works the same way. IP is the postbox + sorting-centre network — it delivers packets to an address, but never promises they arrive, in order, or only once. TCP is registered mail sitting on top of IP: sequence numbers = tracking, ACKs = signatures, retransmissions = the second delivery attempt. UDP is the birthday card — just send it and move on. DNS is the phonebook that turns google.com into 142.250.192.14 so IP knows which "ZIP code" to route to.
A socket is your program's grip on that whole machinery — a file descriptor the kernel gives you, that you can read() and write() just like a file, except the bytes travel through six intermediate networks to reach the other side.
The four ideas that unlock everything else
- The network is not reliable, not ordered, not fast, and not private. TCP + TLS just do a good enough job hiding that from you most of the time.
- Every socket is a file descriptor. Everything you learned about FDs (ulimit, epoll, close-on-exec) applies. The FD table is the same table.
- TCP is a byte stream, not a message stream. If you send(‘hello’) then send(‘world’), the receiver may recv(‘helloworld’) or recv(‘hell’) then recv(‘oworld’). You must add message boundaries yourself.
- DNS is a distributed, cached, TTL-driven database — not a live query. Anything you change is stale somewhere for at least the TTL.
A quick history so you know why the world looks like this
- 1969ARPANET · first packet-switched networkFour US universities connected via IMPs (Interface Message Processors). ‘LOGIN’ is the first message; it crashes after ‘LO’.
- 1974TCP/IP proposed · Cerf & Kahn‘A Protocol for Packet Network Intercommunication’ — the paper that becomes RFC 675, then the internet.
- 1983ARPANET flag day → TCP/IPThe entire network cuts over from NCP to TCP/IP on Jan 1. Every host had to upgrade. Never done again.
- 1983DNS invented · Paul MockapetrisReplaces the single hosts.txt file that everyone was maintaining by hand. RFC 882/883.
- 1994SSL 1.0 → 3.0 · NetscapeTLS's ancestor. First serious attempt to encrypt TCP for e-commerce.
- 2015HTTP/2 goes RFC (7540)Multiplexed streams over one TCP connection. Fixes app-layer head-of-line blocking.
- 2022HTTP/3 goes RFC (9114)Runs over QUIC (over UDP). Escapes TCP's transport-layer head-of-line blocking entirely.
(b) Visual walkthrough · 15 min
What actually happens when you type curl https://example.com
That single curl invocation touched every layer of the TCP/IP stack. If any one of them misbehaves, the request fails — but the error message you see (curl: (7) Failed to connect) tells you nothing about which one. That's what the rest of this session teaches you to disambiguate.
The layer cake — the model you actually use
TCP/IP stack (top = your code, bottom = electrons on a wire)
The 3-way handshake, spelled out
Client picks a random seq number (say 1000) and sends SYN(seq=1000). This defends against ancient stray packets from a previous connection.
Server picks its own random seq (say 5000) and replies SYN(seq=5000), ACK(ack=1001). ‘I got up to 1000, next expected is 1001.’
Client replies ACK(ack=5001). Both sides now agree on starting sequence numbers.
The kernel marks the socket ESTABLISHED. Your app's connect() call returns. First byte of data can flow.
How DNS actually resolves blog.example.com
The resolver caches every answer for the TTL. Every subsequent lookup within TTL is served from cache — which is why DNS changes take time to propagate and why lowering a TTL after an outage starts is too late.
TCP vs UDP — when to pick which
Reliable, ordered, congestion-controlled byte stream
- 3-way handshake before any data (1 RTT of latency tax)
- Guaranteed delivery via ACKs + retransmissions
- In-order delivery — recv() gives bytes in send order
- Congestion control (slow start, cubic) — plays nice with the network
- Use for: HTTP/1.1+2, gRPC, SSH, SMTP, databases, anything correctness-sensitive
Fire-and-forget datagrams — you build the reliability you need
- No handshake, no state — first packet is the message
- No delivery guarantee, no ordering, no retransmission
- Message-oriented — one send = one recv (or drop)
- You get to invent the semantics: DNS did, QUIC did, gaming/voice do
- Use for: DNS, DHCP, NTP, video/voice, gaming, custom protocols (QUIC, WireGuard)
The port number picture
"TCP guarantees delivery. If send() returns successfully, the data reached the other side."
A successful send() means the bytes were copied into the kernel's socket send buffer. Nothing more. They may not have left the machine, and the peer may already be gone. TCP guarantees that data is delivered in order and without gaps, or the connection fails — it cannot guarantee delivery across a network that has stopped forwarding packets, and it certainly cannot tell you the application on the other end processed anything.
The myth is sticky because in a healthy datacenter it is effectively true a very high proportion of the time, so it survives every test you run. It breaks exactly when it matters: during a partition or a peer crash, the sender happily writes into its buffer for as long as the window allows, and the failure surfaces seconds or minutes later — or never, if no keepalive is configured. This is why a client can appear connected to a server that was terminated ten minutes ago.
Watch data sit in the kernel with the application none the wiser:
ss -tin
# Send-Q shows bytes accepted by the kernel but NOT yet acknowledged.
# A large, non-draining Send-Q means the peer is gone or stalled,
# while your application's send() calls kept returning success.
ss -tan | awk '{print $1}' | sort | uniq -c
# count sockets per state - a pile of CLOSE-WAIT means YOUR app
# is not calling close() on connections the peer already closed.Why does TCP deliberately slow itself down at the start of every connection instead of sending at full speed immediately?
- 1The network between two endpoints is shared by every other connection, and no endpoint knows its capacity or how many others are using it.forced by · there is no signalling channel telling a sender the available bandwidth on a path it does not control
- 2Routers have finite buffers. When arrival rate exceeds forwarding rate, buffers fill and packets are dropped.forced by · a queue with a fixed bound must discard once full; there is nowhere else for the packet to go
- 3If every sender transmitted at full speed on connect, aggregate demand would exceed capacity, buffers would overflow, and mass loss would trigger mass retransmission — which adds yet more load.forced by · retransmission under congestion is positive feedback, and this is precisely how the 1986 Internet congestion collapse happened
- 4Therefore each sender must independently estimate its fair share, and the only observable feedback available is whether packets are being acknowledged.forced by · the network provides no explicit rate signal by default; loss and delay are the only universally available signals
- 5So a sender starts conservatively and increases its window while acknowledgements keep arriving, backing off sharply when loss indicates it exceeded capacity.forced by · probing upward is the only safe way to discover a limit you cannot query, and backing off hard is required for the system to converge rather than oscillate
Therefore slow start is not conservatism, it is a distributed capacity-discovery algorithm running with no coordination between participants — and it is why the internet remains stable under load without any central controller.
And note what this predicts: short connections never reach full speed, because they finish while still ramping up. That single prediction explains why connection reuse and keep-alive matter so much, why HTTP/2's single multiplexed connection outperforms six parallel HTTP/1.1 connections, and why connection pooling is one of the highest-leverage changes available to a chatty client. It also predicts that on a high-bandwidth, high-latency path, the ramp costs many round trips — which is exactly the regime where TCP performs worst and where protocols like QUIC focused their effort.
Your bytes go into a TCP envelope (ports, sequence numbers), that into an IP envelope (source and destination addresses), that into an Ethernet frame (MAC addresses, hop by hop). Each layer only knows its own addressing scheme and treats the layer above as opaque payload.
DNS sits outside this entirely: it is a lookup that happens before any of it, translating a name into the address the IP layer needs. It is a distributed cache with TTLs, and almost every DNS surprise is a caching surprise.
- IP addresses route between networks; MAC addresses route within one. A packet keeps the same IP endpoints end to end while its Ethernet frame is rewritten at every hop — which is why traceroute works and why NAT is a violation worth understanding.
- The three-way handshake costs one full round trip before any data flows, and TLS costs one or two more. On a 100 ms path that is 200–300 ms before the first byte of your request is sent, which is why connection reuse dominates latency for small requests.
- DNS TTLs are advisory and honoured inconsistently by resolvers, OS caches, JVMs and libraries. Planning a cutover around a TTL means planning around the least compliant cache in the chain, so always keep the old endpoint alive well past the TTL.
- Bandwidth and latency are independent, and you cannot fix latency with money. Throughput on a single TCP stream is bounded by window size divided by round-trip time — which is why a fat, long link stays slow until window scaling is right, and why cross-region transfers need parallel streams.
Fire this model when you see: "the network is slow" with no measurement · a service reachable by IP but not by name · a deploy that half-worked after a DNS change · sockets stuck in CLOSE-WAIT · a transfer that will not exceed a suspiciously round throughput number.
How long should a client wait before deciding a request failed, and what should it do next?
Always set an explicit timeout — the default in most HTTP clients is infinite or minutes long, which is never what you want. Then bound retries by a budget rather than a count, add full jitter, and put a circuit breaker in front of dependencies you retry against.
The insight that matters: a retry is a load-multiplying decision made at the worst possible moment, when the dependency is already unhealthy. Treat retry capacity as a shared resource with a cap — a common approach is to allow retries only while they remain a small fraction of total outbound requests, so a healthy system retries freely and a struggling one automatically stops amplifying its own failure.
(c) Hands-on · 25 min
Build a minimal TCP echo server + client in pure Python (no frameworks), then use it to see the byte-stream problem with your own eyes. Save as echo.py and run in two terminals.
#!/usr/bin/env python3
"""echo.py — minimal TCP echo server + client.
Run server: python3 echo.py server
Run client: python3 echo.py client hello world
Framed mode: python3 echo.py framed-client hello world
"""
import socket
import struct
import sys
import threading
HOST, PORT = "127.0.0.1", 9999
# ---------- server ----------
def server() -> None:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# SO_REUSEADDR: reclaim the port even if a previous socket is in TIME_WAIT.
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(5)
print(f"[server] listening on {HOST}:{PORT}")
while True:
conn, addr = s.accept()
print(f"[server] connected by {addr}")
threading.Thread(target=handle, args=(conn,), daemon=True).start()
def handle(conn: socket.socket) -> None:
"""Naive handler — proves the byte-stream problem exists."""
with conn:
while True:
data = conn.recv(1024) # blocks until at least 1 byte arrives
if not data: # empty = peer sent FIN
print("[server] client hung up")
return
print(f"[server] recv {len(data):4d} bytes: {data!r}")
conn.sendall(b"echo: " + data)
# ---------- naive client ----------
def client(msg: str) -> None:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
# Two sends back-to-back — watch what the server sees.
s.sendall(msg.encode())
s.sendall(b" [second-write]")
reply = s.recv(4096)
print(f"[client] got {reply!r}")
# ---------- framed client (the fix) ----------
def framed_client(msg: str) -> None:
"""Length-prefix each message so the receiver knows where it ends."""
payload = msg.encode()
header = struct.pack("!I", len(payload)) # 4-byte big-endian length
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(header + payload)
# Read the length header first, then exactly that many bytes.
raw_len = recv_exact(s, 4)
(msg_len,) = struct.unpack("!I", raw_len)
body = recv_exact(s, msg_len - len(b"echo: ") + len(b"echo: "))
print(f"[framed-client] got {body!r}")
def recv_exact(s: socket.socket, n: int) -> bytes:
"""Read exactly n bytes or die trying. This is the pattern real protocols use."""
buf = bytearray()
while len(buf) < n:
chunk = s.recv(n - len(buf))
if not chunk:
raise ConnectionError(f"peer closed after {len(buf)}/{n} bytes")
buf.extend(chunk)
return bytes(buf)
# ---------- entrypoint ----------
if __name__ == "__main__":
mode = sys.argv[1] if len(sys.argv) > 1 else "server"
if mode == "server":
server()
elif mode == "client":
client(" ".join(sys.argv[2:]) or "hello")
elif mode == "framed-client":
framed_client(" ".join(sys.argv[2:]) or "hello")
else:
sys.exit(f"unknown mode: {mode}")Anatomy of the script — what to notice, line by line
The lines worth reading twice
Run it and observe
# terminal 1
python3 echo.py server
# terminal 2 — naive client (two back-to-back sends)
python3 echo.py client "hello world"
# what the server prints will often look like:
# [server] recv 25 bytes: b'hello world [second-write]'
# NOT two separate recvs — that's the byte stream merging your two sends.Now inspect the socket state from a third terminal while a connection is alive:
# Linux: list all TCP sockets with process info
ss -tnp | grep 9999
# You'll see two rows:
# ESTAB 0 0 127.0.0.1:9999 127.0.0.1:54321 users:(("python3",pid=1234,fd=4))
# LISTEN 0 5 127.0.0.1:9999 0.0.0.0:* users:(("python3",pid=1234,fd=3))
# And see the actual packets on the loopback interface:
sudo tcpdump -i lo -X port 9999
# You'll literally see SYN, SYN-ACK, ACK, PSH, FIN in the output.- Start the server, then run the naive
clientthree times in a row. Watch how the server sometimes prints one bigrecvand sometimes two smaller ones — TCP is deciding for you. - Now run
python3 echo.py framed-client "hello world". The framing header (\x00\x00\x00\x0b= length 11) is visible in the server'srecvoutput. - Stretch: modify the server to read 4-byte length headers with
recv_exact(4), thenrecv_exact(length). You've just implemented the same wire format Kafka uses.
(d) Production reality · 15 min
Every senior backend engineer has been paged for one of these three failure modes. All three are real, all three took large chunks of the internet down, and all three are worth studying in detail because the same failure modes still happen today at smaller scale.
On 24 June 2019, a small Pennsylvania ISP called DQE Communications ran a "BGP optimiser" that reannounced several thousand of Cloudflare's, Amazon's, and other providers' IP prefixes as if DQE were the origin — but more specific (longer prefix) than the real routes.
DQE's upstream, Allegheny Technologies, passed the leak to Verizon, who accepted the routes and re-advertised them to the rest of the global BGP table. Because BGP prefers more-specific prefixes, huge chunks of Cloudflare-bound traffic got routed to a tiny steel-mill ISP in Pennsylvania. Websites went dark globally for ~2 hours.
Cloudflare called Verizon and NOCs around the world to withdraw the routes manually. The permanent fix is the industry-wide push for RPKI (Resource Public Key Infrastructure) — cryptographically signed route origin authorisations so upstreams can validate what they're being asked to route.
Cloudflare's Nick Coghlan wrote the definitive blog post the same day. Read it — it's the clearest incident writeup you'll ever see.
On 4 October 2021, Facebook engineers ran a routine backbone capacity-audit command that — due to a bug in the audit tool — withdrew every BGP route to Facebook's authoritative DNS servers.
From the internet's point of view, Facebook's DNS ceased to exist. That meant facebook.com, instagram.com, whatsapp.com, and every internal service that resolved through Facebook DNS all became unreachable — including the tools engineers needed to fix the problem. Employee badges stopped working because they auth'd through the same broken DNS.
Engineers had to physically enter data centres and manually reset routers. Cloudflare (whose 1.1.1.1 resolvers were hammered with retries) wrote a beautiful post-mortem showing the exact moment Facebook's BGP announcements disappeared from the global table.
Meta's own postmortem admitted: their audit tool's guardrail against exactly this failure had a bug, and their DNS servers were configured to withdraw themselves if they couldn't reach the backbone — a "safety" that turned into a suicide pact.
On 7 December 2021, an automated capacity-scaling event in AWS's internal network triggered unexpected behaviour in a large number of clients on the internal AWS backbone. Those clients started connection-storming the internal DNS service, which then couldn't scale up fast enough.
Because AWS's own control plane (EC2, Lambda, SQS APIs) uses that same internal DNS, control-plane APIs started timing out. Customers couldn't launch new instances, autoscalers failed, and the many services that treat "us-east-1 is always up" as a design assumption cascaded down.
AWS engineers had to manually throttle the internal traffic pattern and add capacity. The postmortem highlighted that the automation itself lacked good throttling: when the "clients" retried at once, they made the recovery harder.
AWS later hardened internal DNS with better rate limits and pushed harder on multi-region design guidance for customers.
The everyday gotchas (not headline-grabbing, but they'll page you)
Where this shows up next in the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
If you can't teach these three to a friend without notes, redo the session:
- What happens between typing
curl example.comand getting HTML back? (in one minute, naming DNS, TCP handshake, HTTP) - Why is TCP a byte stream, and what's one concrete bug that creates? (with a fix)
- Name one real production outage where DNS or BGP took down a household-name company — and the one-line lesson.
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 60 minutes.