S062 · Networking II — Load Balancers L4 vs L7, Reverse Proxies

Module M07: Systems & Infrastructure · Session 62 of 130 · Track: SYS ⚖️

What you'll be able to do after this session

• Explain Networking II to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S061 — Networking I — TCP/IP, DNS, Sockets

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Load Balancer explained (ByteByteGo) — 6-minute visual comparing L4 vs L7 in one whiteboard.
• 🎥 Deep dive (30–60 min) · System Design — Load Balancers Deep Dive (Gaurav Sen) — algorithms, health checks, sticky sessions, real trade-offs.
• 🎥 Hands-on demo (10–20 min) · NGINX Reverse Proxy in 15 min (TechWorld with Nana) — spin up NGINX in front of two backends in real time.
• 📖 Canonical article · NGINX — What is a Reverse Proxy? — the reference definition + diagrams.
• 📖 Book chapter / tutorial · HAProxy Configuration Manual — Introduction — the canonical L4/L7 LB manual, still the gold standard.
• 📖 Engineering blog · Envoy at Lyft — the universal data plane — the story behind the LB/proxy that powers Istio, AWS App Mesh, and half of modern service meshes.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Distributing traffic without dropping it.

Picture a busy restaurant with one host at the door and ten identical chefs in the kitchen. A guest walks in; the host doesn't cook — they just point the guest at whichever chef has the shortest queue. That host is a load balancer. Without them, everyone would pile up at chef #1, chefs #2–10 would sit idle, and the restaurant would look overloaded while actually running at 10% capacity.

On the internet, a load balancer sits in front of a pool of identical backend servers and spreads incoming requests across them. This does three big jobs at once: (1) scale — you can add more chefs without changing the front door, (2) resilience — if one chef quits, the host just stops sending guests there (health checks), and (3) flexibility — you can do rolling upgrades, blue/green deploys, or A/B tests by shifting traffic weights.

The two big families are Layer 4 (L4) and Layer 7 (L7), named after the OSI layer they inspect. An L4 load balancer works at the TCP/UDP level — it sees "IP + port" and forwards packets or connections without understanding what's inside. It's fast, cheap, and dumb — perfect for anything that isn't HTTP. An L7 load balancer actually reads the HTTP request: URL path, headers, cookies, host name. That lets it do smart routing (/api/* → backend A, /images/* → backend B), rewrite headers, terminate TLS, and enforce WAF rules — at the cost of extra latency and CPU.

Forever-gotcha: L4 balancers preserve the connection; L7 balancers terminate it and open a new one to the backend. This means the backend sees the LB's IP (not the client's) unless the LB adds an X-Forwarded-For header. Forget this and your rate-limit-by-IP starts limiting your own load balancer.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

L4 vs L7 at a glance:

Load-balancing algorithms — worked example:

Say you have 3 backends and 6 incoming requests.

1. Round-robin: R1→A, R2→B, R3→C, R4→A, R5→B, R6→C. Fair only if all requests are equal weight.
2. Least-connections: if backend A is stuck on a slow request, new requests go to B and C. Best default for mixed workloads.
3. Weighted round-robin: A weight 5, B weight 1 → 5 out of 6 requests go to A. Useful when hardware isn't identical or for canary rollouts (99% stable, 1% new build).
4. Consistent hashing: hash(user_id) → backend. Same user always hits the same backend (useful for local caches). Adding/removing a backend only reshuffles ~1/N of keys — this is the trick behind Memcached clients and CDNs.

Worked example — health checks. LB pings each backend every 5 s at /health. Rules: "3 failed in a row = mark dead". Backend B's disk fills, /health starts returning 500. After 15 s the LB stops sending traffic to B; ops fires a page; when disk is fixed and 3 checks pass, B is back in rotation automatically.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Let's stand up NGINX as an L7 reverse proxy in front of two tiny Python backends.

# 1) Two backend "apps" — each just prints which port answered
mkdir -p /tmp/lb-demo && cd /tmp/lb-demo
cat > app.py <<'EOF'
import sys, http.server, socketserver
PORT = int(sys.argv[1])
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200); self.end_headers()
        self.wfile.write(f"hello from backend on port {PORT} · path={self.path}
".encode())
    def log_message(self, *a): pass
socketserver.TCPServer(("127.0.0.1", PORT), H).serve_forever()
EOF
 
python3 app.py 8001 &
python3 app.py 8002 &
 
# 2) NGINX config — needs `nginx` installed (apt install nginx / brew install nginx)
cat > nginx.conf <<'EOF'
events {}
http {
  upstream backend {
    least_conn;                       # try round_robin / ip_hash too
    server 127.0.0.1:8001 max_fails=2 fail_timeout=10s;
    server 127.0.0.1:8002 max_fails=2 fail_timeout=10s;
  }
  server {
    listen 8080;
    location /api/ {
      proxy_pass http://backend;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Real-IP       $remote_addr;
    }
    location /health { return 200 "ok
"; }
  }
}
EOF
 
nginx -c "$PWD/nginx.conf" -p "$PWD" -g "daemon off;" &
NGINX_PID=$!
 
# 3) Hit it a bunch and watch it alternate
for i in $(seq 1 10); do curl -s http://127.0.0.1:8080/api/hi; done
 
# 4) Kill backend 8001 and re-run — NGINX should route everything to 8002
kill %1
for i in $(seq 1 5); do curl -s http://127.0.0.1:8080/api/hi; done
 
# 5) Cleanup
kill $NGINX_PID %2 2>/dev/null; true

Checklist — what to observe:

1. Requests alternate between ports 8001 / 8002 — that's least_conn at work.
2. After killing 8001, NGINX takes up to fail_timeout (10 s) to fully mark it dead. Watch the retries in nginx error log at /tmp/lb-demo/logs/error.log.
3. /api/* is proxied; /health is answered by NGINX directly (no backend hit).
4. In the backend, if you log self.headers, you'll see X-Forwarded-For and X-Real-IP — proof that the LB is telling the backend who the real client is.
5. Try switching to ip_hash — every curl from the same host now sticks to the same backend (session affinity).

Try this modification: add a third backend on 8003 that always returns HTTP 500. Configure NGINX with proxy_next_upstream error http_500; so failed responses transparently retry on the next backend. Watch it hide the failure from the client.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1 — the cookie stampede. A social app used cookie-based session stickiness so users always hit the same backend (for in-memory session state). When one backend died, 20 k users were all redirected simultaneously to backend B, which promptly died from the load spike. Lesson: stickiness is a smell. Push session state into Redis so any backend can serve any user.

War story #2 — the invisible 4xx. An ALB health check hit / on the backend, which returned 200 even when the DB was down. The app happily served error pages to real users while the LB said "all healthy". Fix: health endpoints must exercise real dependencies (DB ping, cache ping) — but not so hard that a slow dep makes all instances flap.

War story #3 — TLS in the middle. A team terminated TLS at the ALB and used HTTP inside the VPC. Fine, until a compliance audit demanded end-to-end encryption. Retrofitting mTLS across 200 services took months. Modern default: service mesh (Envoy/Istio/Linkerd) so mTLS is free and transparent.

Common gotchas at production scale:

• Thundering herd on scale-out. New backend joins the pool; LB sends it a burst of traffic before its JIT/cache is warm; it falls over. Use slow_start (NGINX Plus / Envoy) to ramp weight over 30 s.
• DNS-based LB caching. Route 53 returns different IPs per query but Java's default networkaddress.cache.ttl = -1 caches the first one forever. Countless outages from this single line.
• Cross-AZ traffic cost. In AWS, sending traffic between availability zones costs money. Bad load-balancing configs can 3× your bill. Prefer zone-aware routing (ALB → same-AZ target if healthy).
• X-Forwarded-For spoofing. If your app trusts XFF from any upstream, an attacker can forge their client IP. Only trust XFF from your LB's IP.
• Connection draining. During deploys, tell the LB "stop sending new requests to this backend" but let in-flight ones finish. AWS calls it "deregistration delay"; K8s calls it preStop + terminationGracePeriodSeconds. Forgetting = user-visible 502s during every deploy.

Envoy at Lyft, ATS at Yahoo, and GLB at GitHub are the gold-standard reads for how the biggest shops solve this.

(e) Quiz + exercise · 10 min

1. Give the sharpest one-sentence difference between an L4 and L7 load balancer.
2. Why does least_connections usually beat round_robin in production?
3. A backend's /health returns 200 but real requests return 500. Name two ways to make the health check catch this.
4. Consistent hashing: when you add one backend to a pool of 10, what fraction of keys get remapped, and why does that matter?
5. What does "connection draining" achieve during a deploy?

Stretch problem (connects to S061 — TCP/IP): an L4 load balancer forwards TCP connections; an L7 load balancer terminates the client TCP connection and opens a new one to the backend. Draw the two SYN/SYN-ACK/ACK sequences side by side and explain (a) which model preserves the client's source IP naturally, and (b) which one enables features like HTTP/2 multiplexing between LB and backend. Answer in ~8 lines.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Networking II? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S063): Caching — Cache-Aside, Write-Through, TTLs, Invalidation
• Previous (S061): Networking I — TCP/IP, DNS, Sockets
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.