S062 · Networking II — Load Balancers L4 vs L7, Reverse Proxies
How one hostname fans out to a hundred servers without dropping a packet — the L4 vs L7 decision, health checks, sticky sessions, and the reverse-proxy patterns that run every real web system.
🎯 Choose L4 vs L7, run a local Nginx reverse proxy in front of two backends, and explain sticky sessions, health checks, and connection draining without notes.
Why this session exists
Every production system you'll ever touch sits behind a load balancer — even if the "load balancer" is a single 20-line Nginx config on a VM. Get the choice between L4 and L7 wrong and you either burn CPU parsing HTTP you don't need, or lose the ability to route on paths, terminate TLS, and see per-request latency. This session gives you the mental model and a working reverse proxy on your laptop in under an hour.
- Pick L4 vs L7 in 10 seconds given the requirements (throughput vs routing needs).
- Explain reverse proxy vs forward proxy, round-robin vs least-connections, and sticky sessions to a friend.
- Run Nginx locally in front of two Python backends, kill one, and watch traffic drain.
- Diagnose the top-3 LB pitfalls: half-open connections, thundering herd on failover, and unhealthy-instance flapping.
Prerequisites
- S060 · Linux fundamentals — you know what a socket and a port are.
- S061 · Networking I — TCP handshake, ports, DNS.
- S015 · HTTP basics — headers, methods, status codes.
(a) Intuition · 5 min
A busy restaurant has one host at the door and 20 tables. When you walk in, the host looks at your party ("4 people, high chair, window seat please") and picks the right table. They can also refuse you if the kitchen is on fire.
Down the street, the post office has a mail sorter. It doesn't open your letter — it looks at the ZIP code on the envelope and drops it into one of a hundred bins. Fast, dumb, and it doesn't care what's inside.
An L7 load balancer is the restaurant host. It reads the HTTP request — path, headers, cookies, JWT claims — and routes intelligently ("/api/* to service-A, /images/* to service-B, mobile clients to the beta pool"). It can rewrite headers, terminate TLS, and reject malformed requests before they touch your app.
An L4 load balancer is the mail sorter. It sees only (src_ip, src_port, dst_ip, dst_port). It picks a backend, opens a TCP tunnel, and copies bytes both ways. It has no idea whether it's HTTP, gRPC, MySQL, or SSH — and that's its superpower: it can push 10× the packets per second because it does 10× less work per packet.
The four jobs every load balancer does
- Distribute — pick which backend gets the next connection (round-robin, least-connections, hash on client IP, weighted).
- Health-check — poll each backend every few seconds and pull it out of rotation the moment it fails N checks in a row.
- Terminate — end the client TCP + TLS connection at the LB, open a fresh (often HTTP/2, keep-alive) connection to the backend. Fewer handshakes for the origin.
- Observe — every request flows through one place, so this is your single best spot to log latency, status codes, and per-backend error rates.
A quick history so the ecosystem makes sense
- 1996DNS round-robinThe original ‘poor man's load balancer’. Return multiple A records; clients pick one. Still used today for GSLB.
- 1997Cisco LocalDirectorFirst commercial hardware load balancer. F5's BIG-IP follows in 1997 too.
- 2004Nginx v0.1Igor Sysoev at Rambler.ru releases Nginx to solve the C10K problem. Reverse proxy for the masses.
- 2009HAProxy 1.4 · keep-aliveHAProxy becomes the reference L4/L7 open-source LB. Runs GitHub, Stack Overflow, Reddit.
- 2016Envoy · Lyft open-sourcesModern L7 proxy designed for microservices. xDS API, gRPC-native, becomes the data plane for Istio.
- 2020eBPF load balancingCilium, Katran (Meta), and MagLev (Google) push L4 LB into the kernel. Millions of PPS on commodity hardware.
(b) Visual walkthrough · 15 min
The two layers of the OSI stack a load balancer can live at
How a single request flows through an L7 reverse proxy
example.com resolves to the LB's public IP (often an anycast VIP).
Client opens TCP to :443. LB terminates TLS using its cert. Client never talks to the origin.
Reads Host, path, cookies. Applies routing rules and WAF filters (rate limit, block SQLi).
Consult health-check state + LB algorithm (round-robin, least-conn, hash). Skip DOWN backends.
Reuse an existing keep-alive connection if the pool has one; else open a new TCP.
LB pipes response bytes back to the client. Logs status, latency, backend id.
L4 vs L7 — pick one before you pick a product
Fast, dumb, protocol-agnostic
- AWS NLB · GCP TCP LB · Nginx stream · HAProxy mode tcp
- Millions of packets/sec on modest hardware
- Can load-balance MySQL, Redis, SSH, gRPC-with-TLS-passthrough
- Cannot route on HTTP path, cannot terminate TLS without seeing plaintext
- Health checks are TCP-connect or a custom TCP probe
Smart, expensive, HTTP-only
- AWS ALB · GCP HTTPS LB · Nginx · HAProxy mode http · Envoy · Traefik
- Thousands of req/sec per core (parsing HTTP costs CPU)
- Routes on Host, path, header, cookie, JWT claim
- Terminates TLS, injects X-Forwarded-*, does WAF and rate limiting
- Health checks are real HTTP GETs (\/health returning 200)
The five distribution algorithms you'll actually see
Algorithms, ordered by how often you'll pick each
"Round-robin distributes load evenly. Every backend gets the same number of requests, so every backend does the same amount of work."
Round-robin equalises request count, not work. If request cost varies — and it always does — equal counts produce wildly unequal load. A backend that receives one expensive query while others receive cheap ones is overloaded despite a perfectly fair share of requests. Worse, a degraded backend that fails fast receives requests faster, because it finishes sooner and returns to the rotation — the black hole failure mode.
The myth is sticky because it is true when requests are homogeneous, which describes the static-file serving that load balancers were originally built for. It is also the default in every load balancer, so it is what you meet first and what appears to work in staging where every request is the same synthetic call. It fails in production exactly where request cost has a long tail — which is to say, in every real API.
Compare distributions, and notice the second one is what actually matters:
# requests per backend - round-robin makes this flat by construction
sum(rate(requests_total[5m])) by (backend)
# in-flight requests per backend - this is the real load signal
sum(backend_active_requests) by (backend)
# p99 latency per backend - a hot backend shows here first
histogram_quantile(0.99, sum(rate(latency_bucket[5m])) by (le, backend))Flat request counts with a spread in active requests and p99 is the signature that round-robin is the wrong algorithm for your traffic.
Why does a load balancer need health checks at all? Backends could simply report their own health when they fail.
- 1A failing backend is, by definition, in an unreliable state — the failure may be exactly what prevents it from reporting.forced by · self-reporting requires the reporting path to work, which the failure may have broken
- 2The worst failures are silent: a process alive but deadlocked, a full disk, an exhausted connection pool, a JVM in continuous garbage collection. It accepts connections and never responds.forced by · the OS keeps accepting on a listening socket regardless of whether the application is making progress
- 3Therefore health must be assessed by an external observer that treats the backend as a black box and measures what it actually does.forced by · only an outside party can distinguish "not responding" from "reports itself healthy but is not"
- 4But the observer only ever gets evidence about the path between itself and the backend, so it can never distinguish "backend is down" from "the network between us is down".forced by · a timeout is indistinguishable from a partition — this is the fundamental limitation of failure detection in distributed systems
- 5So health checking is inherently a heuristic with two error modes, and the thresholds are a direct tradeoff: check aggressively and you evict healthy backends during transient blips; check leniently and you route to dead ones for longer.forced by · you are estimating an unobservable state from a noisy channel, and no threshold eliminates both errors
Therefore health checks are a tuned failure detector, not a fact. The right settings are derived from how long your backend takes to genuinely recover versus how long you can tolerate routing to a dead one.
And note what this predicts: a health check that is too aggressive is actively dangerous, because during a load spike backends slow down, fail checks, get evicted, and their traffic lands on the remaining backends — which then slow down and get evicted too. The health checker becomes the mechanism of a cascading failure. That is precisely why serious implementations include a minimum healthy fraction (panic threshold): below some proportion of healthy backends, the balancer ignores health entirely and sends to everyone, on the reasoning that degraded service beats no service. That safeguard is not a hack — the derivation demands it.
The load balancer sits in front of a pool and decides where each request goes. It is doing three separate jobs: choosing a backend (the algorithm), knowing which backends are usable (health checking), and terminating connections (TLS, HTTP parsing, connection reuse). Every load balancer question is one of those three.
Because everything flows through it, it is on the critical path for availability. Whatever redundancy your backends have, the balancer needs too — otherwise you built a highly available system behind a single point of failure.
- L4 balances connections by inspecting IP and port; L7 balances requests by parsing HTTP. L7 enables path routing, retries, header manipulation and per-request balancing — and it costs CPU and adds a hop. With HTTP/2 or gRPC the choice is forced: L4 pins every stream on one long-lived connection to a single backend, so you must use L7 or you have no balancing at all.
- Prefer least-connections or peak-EWMA over round-robin whenever request cost varies. They approximate work rather than count, and they degrade gracefully when one backend slows because a slow backend accumulates in-flight requests and is naturally sent less.
- Draining is a distinct state from healthy or unhealthy: stop sending new requests, let in-flight ones finish, then remove. Skipping it turns every deploy into a burst of user-visible errors, and it is the single most common cause of "we see 502s during every rollout".
- Sticky sessions are a constraint you should have to justify. They defeat even distribution, make draining harder, and convert a stateless tier into a stateful one — usually to work around session state that should have been externalised.
Fire this model when you see: errors during every deploy · one backend with double the CPU of its peers · gRPC traffic that ignores newly added pods · a service that gets slower as you add instances · a cascading failure that began with a health check.
Where does load balancing happen: a dedicated proxy tier, or in the client library making the call?
Start with a proxy tier. It is the simplest thing that works, has no client requirements, and the extra hop is usually a fraction of a millisecond inside a datacenter — almost never your actual latency problem.
Move to a mesh when the driver is policy uniformity across many languages, not when the driver is latency. Adopting a mesh to save a network hop is a bad trade: you spend a large operational budget to recover a cost you probably cannot measure at the p50. And whichever you pick, verify the behaviour that actually bites — that draining works during a deploy, and that HTTP/2 traffic is balanced per request rather than per connection.
(c) Hands-on · 25 min
Two tiny Python HTTP servers, an Nginx reverse proxy in front, and a curl loop that shows the round-robin working — then we kill a backend and watch traffic drain. All local, no cloud.
What each block does
Anatomy of the config + script
Uncomment least_conn; in nginx.conf, then:
docker restart s062-nginx
# In one terminal:
python3 -c "
import urllib.request, threading
def hit(i):
r = urllib.request.urlopen('http://127.0.0.1:8080/?slow=' + ('1' if i%2==0 else '0'))
print(i, r.headers['X-Backend'], r.read().decode().strip())
[threading.Thread(target=hit, args=(i,)).start() for i in range(20)]
"You should see the slow requests spread across both backends instead of piling up on whichever one drew the short straw first. Round-robin doesn't know a request is slow — least-connections effectively does.
One more knob you'll meet: sticky sessions
If your app stores state in memory (a shopping cart, a socket.io connection), you need every request from the same user to hit the same backend. Two ways:
- IP hash —
hash $remote_addr consistent;— same client IP always routes to the same backend. Breaks behind corporate NAT (many users share one IP). - Cookie-based — Nginx Plus and every cloud LB inject a
AWSALB/AWSALBCORS/routecookie on the first response. Subsequent requests carry the cookie, and the LB reads it and routes to that backend.
Prefer stateless backends (session in Redis or JWT) so you don't need sticky sessions at all. Session S063 (Caching) shows how.
(d) Production reality · 15 min
A 43-second network partition caused GitHub's east-coast MySQL cluster to fail over. Their L4 load balancers happily kept sending write traffic to a promoted follower that then lost the race — and 22 GB of writes ended up on the wrong side of the split.
Restoring consistency took 24 hours of manual reconciliation, with public status page updates every hour.
hard-stop-after 30m and use -x socket transfer so a reload keeps the listening socket AND the existing connections alive. New connections go to the new worker, old connections keep flowing to the old worker until they close naturally.Common failure modes you'll debug on your own systems
- Half-open connections — client and LB think the TCP is alive, backend crashed 3 minutes ago. Fix: enable TCP keep-alive on the LB with a short interval (e.g. 30 s).
- Thundering herd on failover — 100 clients had connections to backend A. A dies. All 100 reconnect simultaneously to B. B falls over. Fix: exponential backoff in clients + LB "surge queue" limits.
- Health-check flapping — backend is right at the edge of healthy/unhealthy, LB adds/removes it every 5 s, traffic thrashes. Fix: hysteresis — require N=3 consecutive successes to mark UP after a failure, but only M=2 failures to mark DOWN.
- TLS certificate expiry on the LB — the origin cert is fine, but the LB's cert expired last night. Every request is now a browser cert warning. Fix: automated renewal (cert-manager, Let's Encrypt) + expiry alerts 30 days out.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- When would you pick L4 over L7? (one crisp rule)
- Draw the packet path from client browser to backend for an HTTPS request through an L7 LB.
- What is connection draining, and why does every zero-downtime deploy depend on it?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.