Networking II — Load Balancers L4 vs L7, Reverse Proxies

Why this session exists

A load balancer chooses where work goes, but equal request counts do not imply equal work. A health check observes a path, not an infallible fact about a machine. A retry may duplicate an operation. Design these together instead of selecting a product from an L4/L7 slogan. This lesson preserves the reverse-proxy configuration exercise and adds a deterministic offline model of selection, health hysteresis and draining. No server, Docker container or live traffic generator is run here.

Prerequisites

Know TCP connections, HTTP request boundaries and file descriptors. Understand why a timeout does not prove an operation failed before committing. The reference uses NGINX open-source HTTP upstream behavior; active-health or commercial features are not silently assumed.

Intuition: mail sorter, restaurant host and proxy direction

A mail sorter can route an envelope without reading the letter. An L4 design uses flow information such as protocol and endpoint addresses/ports. It may forward/NAT packets, use direct server return or terminate/proxy transport connections; opening a second TCP tunnel is not inherent to every L4 balancer.

A restaurant host uses richer information about a party. An HTTP-aware L7 proxy can route by Host, path or configured header/cookie rules. JWT-claim routing requires actual validation/configuration, not blind trust in a decoded payload. TLS termination is a separate product capability: some L4 products terminate TLS without offering HTTP path routing. Neither a tenfold L4 speedup nor a fixed L7 requests-per-core number follows from the layer label.

A forward proxy acts for clients reaching destinations; a reverse proxy acts for servers receiving clients. Many load balancers are reverse proxies, but DNS distribution and direct-routing packet balancers are counterexamples to “all load balancers are reverse proxies.” A CDN combines proxy/cache/routing capabilities at multiple locations; a single reverse proxy need not balance at all.

Four responsibilities, not four universal features

1. Selection: choose eligible backends and apply the policy at a flow/request boundary.
2. Health: combine passive failures and, where configured/supported, active probes. Not every product polls a /health endpoint.
3. Connection/security handling: decide passthrough versus termination, upstream TLS identity, keepalive and timeouts. Traffic is not automatically encrypted on the second hop just because the browser used HTTPS.
4. Observation: collect attempts, status, latency, queueing, bytes and backend identity at the relevant layer. An L4 path cannot automatically report every HTTP status.

Ecosystem evolution

DNS-based distribution, appliance/packet balancing, HTTP reverse proxies and dynamically configured data planes coexist. NGINX upstream directives document weighted round robin, least connections and hashing; newer methods have their own introduction/version notes. Avoid the old unsupported chronology claiming that all kernel balancers use eBPF or that one algorithm is every vendor's default. Choose from the documented release and workload, not an adoption list.

L4 versus L7 and the request path

For an HTTPS L7 path, DNS identifies the proxy endpoint; client transport/TLS terminates there; the proxy parses HTTP, chooses a route, then selects an eligible backend. It reuses or creates an upstream connection and streams/buffers the response according to configuration. Logs must distinguish total request time from connect and upstream time. A WAF or rate limiter is optional policy on this path, not a guarantee from installing any reverse proxy.

L4 passthrough is useful for non-HTTP protocols and end-to-end TLS to the backend. HTTP path/cookie routing requires an HTTP-aware layer. HTTP/2 multiplexes many requests on one connection: L4 flow affinity can pin them to one backend, but still balances across multiple connections. You are not forced to use L7 in every gRPC system; client-side channel/subchannel balancing is another design. Choose the unit of balancing explicitly.

Five selection policies and a worked skew example

Round robin cycles over eligible peers; weighted round robin apportions selections according to weights over time. It approximates equal work when requests and capacities are comparable. A fast-failing peer does not automatically get more requests under strict round robin merely because it finishes sooner; that feedback problem is more relevant to load-aware policies that mistake fast failure for capacity.

Least connections chooses fewer active upstream connections, accounting for weights under the product's rules. It can help when long-lived work tracks connection occupancy. It can be fooled by multiplexing, idle connections, heterogeneous capacity or a fast-error backend. Connection count is a proxy for work.

Weights express relative capacity or controlled traffic shares, not exact percentages for a tiny sample. In NGINX use supported down/configuration changes for exclusion rather than assume weight=0 is valid. Canary safety also needs error/latency checks, not just gradually increasing traffic.

IP hash/consistent hash provides affinity; clients behind a NAT can concentrate on one backend. Consistent hashing reduces remapping on membership change under the scheme's assumptions; NGINX ip_hash and hash ... consistent are different directives. Affinity is not durability.

Random two choices samples candidates and compares load. Its attractive theoretical results depend on the queue/workload model; it is not universally optimal or a default in every tier. Independently acting clients also have incomplete load information.

Worked example: four sequential assignments have costs 10, 1, 10, 1. Strict two-peer round robin gives A both cost-10 jobs and B both cost-1 jobs: counts 2/2, total work 20/2. A shortest-queued-work policy can improve this case if it can estimate cost or observe backlog. Least connections cannot know a future request's cost. Compare request count, inflight/queue depth, error rate and per-backend latency before claiming a policy fixes the workload.

Keep request counts, occupancy and latency separate

The original metric examples remain useful if your instrumentation actually exports these names and labels. These are PromQL references, not queries executed here:

sum(rate(requests_total[5m])) by (backend)
sum(backend_active_requests) by (backend)
histogram_quantile(0.99, sum(rate(latency_bucket[5m])) by (le, backend))

The first query estimates per-backend request rate from a counter. The second sums a gauge of inflight requests: do not take a counter rate of it. The third follows the Prometheus histogram_quantile contract and estimates p99 from classic histogram bucket rates, preserving the le label; all aggregated series must measure the same latency unit and compatible bucket layout. If requests are counted at both proxy and backend, scope the selectors to avoid double counting. A missing series is not automatically zero. Equal request counts with unequal occupancy/latency motivate a skew investigation, not proof that changing algorithms alone fixes it. Error rates and request mix still matter.

Health is a failure detector, not an oracle

A deadlocked process may accept a TCP handshake while failing an application probe. A probe timeout may reflect the backend, a network partition, probe overload or its dependency. Detection thresholds trade false removal against time spent routing to a bad target. NGINX open-source passive max_fails=2 fail_timeout=5s records qualifying unsuccessful attempts in that interval and excludes the server temporarily; the precise qualifying conditions are governed by proxy_next_upstream. This is not a periodic active HTTP check.

Separate liveness, readiness, overload/admission, and draining. Do not make every instance unready merely because a shared optional dependency is slow: redistribution can overload the survivors. Conversely a database read replica needs more than an open TCP port: test query capability, intended role and a meaningful lag/freshness requirement. SELECT 1 proves query execution, not correct replication position. Probes should be bounded, read-only and cheap enough not to become load.

Hysteresis—such as two consecutive failures to mark down and three successes to restore—is a scenario policy, not the exact algorithm of every product. Some balancers support panic/fail-open behavior when few endpoints remain. That is a deliberate availability/safety tradeoff, not a universal requirement to ignore health. For a fenced database writer, routing to a known wrong leader is not an acceptable “degraded” fallback.

Draining, retries and trusted forwarding

Draining stops new assignments while allowing existing work a bounded grace interval. Killing a backend demonstrates failure, not graceful draining. Long-lived WebSockets or streams may need reconnect/resume protocols, and HTTP/2 may need graceful connection shutdown. A proxy often stays on a WebSocket's data path for the full connection; it does not necessarily disappear after upgrade.

NGINX proxy_next_upstream controls when trying another server is allowed. It cannot recover a response already partially sent to the client. Non-idempotent requests are not freely replayable after being sent upstream; the explicit non_idempotent option changes behavior and is not enabled in this lesson. Bound tries and time, and use application deduplication for uncertain writes. The retry timeout setting is not automatically a total client end-to-end deadline, and read timeout measures inactivity between reads, not total response duration.

For NGINX upstream HTTPS, encryption alone is insufficient: the documented proxy_ssl_verify default is off. A production design must explicitly configure certificate verification, trust roots and the intended upstream name/SNI as required. This loopback HTTP reference does not demonstrate that setup.

At a trusted edge, replace or construct forwarding headers deliberately. An attacker-supplied X-Forwarded-For is not proof of origin. Trust only the known proxy chain and configure the backend to reject direct bypass if it relies on forwarded identity/scheme. Correct Host, scheme and TLS verification matter for redirects and security decisions.

Queue time and retry budgets belong to the same request

Work an explicit budget rather than adding independent timeouts. Assume the caller allows 250 ms total; 70 ms has already been spent resolving, connecting and queueing; reserve 20 ms to return a response. Only 160 ms remains for backend attempts and intervening backoff. If the next attempt may consume 100 ms and a following backoff costs 20 ms, the remaining retry can receive at most 40 ms. Giving both attempts fresh 250 ms budgets violates the caller's deadline.

At three nested layers, allowing three attempts per layer permits up to 3 * 3 * 3 = 27 deepest-layer attempts for one logical request if every outer attempt triggers all inner attempts. A shared remaining deadline, attempt cap and retry-traffic budget can stop this amplification; jitter changes timing, not this worst-case count. An ambiguous write needs a durable idempotency-key/result contract before replay. A locally expired deadline does not itself cancel backend work already committed or executing.

Queue admission is separate from health and selection. With two backends each allowing two active requests and a single waiting queue capped at three, an instantaneous burst of eight requests admits at most seven and rejects one under this scenario policy. The three waiting requests have not started service: their wait still consumes their own deadlines. Expired waiters should leave the queue without being sent upstream. A healthy endpoint can reject excess work without declaring itself dead and moving an unbounded backlog to a neighbor.

# Pure arithmetic/admission fixture; not an NGINX queue implementation.
def admission_counts(arrivals, backends, slots_each, waiting_limit):
    values = (arrivals, backends, slots_each, waiting_limit)
    if any(type(v) is not int or v < 0 for v in values):
        raise ValueError('nonnegative integer counts required')
    active = min(arrivals, backends * slots_each)
    waiting = min(arrivals - active, waiting_limit)
    return active, waiting, arrivals - active - waiting
 
assert admission_counts(8, 2, 2, 3) == (4, 3, 1)
assert admission_counts(2, 2, 2, 3) == (2, 0, 0)
assert admission_counts(8, 0, 2, 0) == (0, 0, 8)
remaining_ms = 250 - 70 - 20
assert remaining_ms == 160
assert remaining_ms - 100 - 20 == 40
assert 3 ** 3 == 27

The offline fixture establishes conservation and arithmetic, not queue fairness, service duration or distributed overload protection. The NGINX reference below explicitly bounds next-upstream tries/time, but does not implement this custom queue or a total caller deadline. proxy_read_timeout remains an inactivity interval. For a real release, verify where queueing occurs, whether cancellation propagates, how body replay is buffered, and whether the application enforces idempotency.

Hands-on: deterministic selection, hysteresis and draining

The following pure model records an active-request count, not a real TCP connection count. It makes the distinction visible rather than passing off a simulation as NGINX. Work finishes only through finish; a draining backend remains alive until existing work ends.

from dataclasses import dataclass
 
@dataclass
class Backend:
    name: str
    active: int = 0
    healthy: bool = True
    draining: bool = False
    failures: int = 0
    successes: int = 0
 
    def probe(self, ok):
        if ok:
            self.failures = 0
            self.successes += 1
            if self.successes >= 3:
                self.healthy = True
        else:
            self.successes = 0
            self.failures += 1
            if self.failures >= 2:
                self.healthy = False
 
class Pool:
    def __init__(self, backends):
        self.backends = backends
        self.cursor = 0
 
    def assign(self, policy='round-robin'):
        eligible = [b for b in self.backends if b.healthy and not b.draining]
        if not eligible:
            raise RuntimeError('no eligible backend')
        if policy == 'round-robin':
            chosen = eligible[self.cursor % len(eligible)]
            self.cursor += 1
        elif policy == 'least-active':
            chosen = min(eligible, key=lambda b: (b.active, b.name))
        else:
            raise ValueError('unknown policy')
        chosen.active += 1
        return chosen
 
    def finish(self, backend):
        if backend.active <= 0:
            raise ValueError('unbalanced completion')
        backend.active -= 1
 
a, b = Backend('alpha'), Backend('beta')
pool = Pool([a, b])
assert [pool.assign().name for _ in range(4)] == ['alpha','beta','alpha','beta']
assert (a.active, b.active) == (2, 2)
a.draining = True
assert pool.assign('least-active') is b
assert a.active == 2  # drain does not kill existing work
pool.finish(a); pool.finish(a)
assert a.active == 0
b.probe(False); assert b.healthy
b.probe(False); assert not b.healthy
try:
    pool.assign()
except RuntimeError:
    pass
else:
    raise AssertionError('routed to unavailable/draining peer')
b.probe(True); b.probe(True); assert not b.healthy
b.probe(True); assert b.healthy
assert pool.assign() is b
costs = [10, 1, 10, 1]
assert (sum(costs[::2]), sum(costs[1::2])) == (20, 2)
print('selection, hysteresis, draining and work skew: PASS')

Anatomy and independent checks

Eligibility precedes selection; an algorithm cannot rescue an empty safe pool. A drain changes only new assignment eligibility, not inflight work. The hysteresis counter resets the opposite streak, so alternating results do not accidentally accumulate consecutive successes. Tests cover removal, recovery, zero eligible peers and the count-versus-work counterexample. An independent oracle enumerates short probe-result sequences and checks the state transitions.

NGINX and two Python backends: complete reference

This replaces the old Docker-host gateway mismatch with three native loopback processes in one disposable working directory. Use an already installed, supported NGINX release and review its configuration; do not pull images or install dependencies merely for this audit. These files are reference material only. The Python handler includes a real slow=1 branch, HTTP/1.1 framing and a health route; the old experiment suggested a slow parameter its handler ignored.

# backend.py — reference, not started during review.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlsplit, parse_qs
import sys
import time
 
class Handler(BaseHTTPRequestHandler):
    protocol_version = 'HTTP/1.1'
 
    def do_GET(self):
        url = urlsplit(self.path)
        if url.path == '/health':
            body = b'OK\n'
        else:
            slow = parse_qs(url.query).get('slow') == ['1']
            time.sleep(0.1 if slow else 0.001)
            body = ('served-by=' + self.server.label + '\n').encode()
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.send_header('Content-Length', str(len(body)))
        self.send_header('X-Backend', self.server.label)
        self.end_headers()
        self.wfile.write(body)
 
    def log_message(self, *args):
        pass
 
if __name__ == '__main__':
    port, label = int(sys.argv[1]), sys.argv[2]
    with ThreadingHTTPServer(('127.0.0.1', port), Handler) as server:
        server.label = label
        server.serve_forever()

# nginx.conf — reference for a disposable native loopback exercise.
pid nginx.pid;
error_log error.log;
worker_processes 1;
events { worker_connections 128; }
http {
    access_log access.log;
    upstream app_pool {
        server 127.0.0.1:8081 max_fails=2 fail_timeout=5s;
        server 127.0.0.1:8082 max_fails=2 fail_timeout=5s;
        # least_conn;  # alternative to default weighted round robin
        keepalive 8;
    }
    server {
        listen 127.0.0.1:8080;
        location = /proxy-health { return 200 "proxy-alive\n"; }
        location / {
            proxy_pass http://app_pool;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 1s;
            proxy_read_timeout 2s;
            proxy_next_upstream error timeout http_502 http_503;
            proxy_next_upstream_tries 2;
            proxy_next_upstream_timeout 3s;
        }
    }
}

keepalive 8 limits idle cached upstream connections per worker for the upstream group, not eight per backend or the total active-connection count. Explicit HTTP/1.1 and an empty Connection header make the intended upstream reuse clear across release defaults. /proxy-health proves only that the proxy responds. HTTP upstream configuration does not establish an active backend health-check service.

For a separately authorized exercise, start backend.py 8081 alpha and backend.py 8082 beta, retain exact PIDs, run nginx -t against the disposable prefix/config, then launch that instance in the foreground or with an exact managed PID. Send bounded GETs to http://127.0.0.1:8080/, inspect X-Backend and attempts, and stop only those owned processes in finally/a shell trap. Do not use broad pkill -f or remove someone else's named Docker container.

To test failure, stop one owned backend and observe passive detection/retry; do not promise the very next request and every later request will be error-free. To test draining, remove new eligibility using the chosen implementation's supported mechanism while a known request is inflight, verify it completes, then stop the process. Graceful NGINX reload can leave old workers serving existing connections; it is not equivalent to instantly cancelling all old upstream assignments. Record that behavior rather than calling a kill a drain.

For the slow-request comparison, hold workload/concurrency constant, collect per-peer inflight and latency distributions, then compare round robin and least connections. A single twenty-request run cannot prove universal improvement. Authentication, TLS upstream verification, production resource limits and shutdown handling are intentionally outside this reference server.

Affinity and deployment choices

IP hashing can map ten users behind one corporate NAT to one peer. Cookie affinity distinguishes clients better but has product-specific names and availability; not every cloud LB injects the same cookies. A dead affinity target still loses unreplicated session state. Store durable/session state in an appropriate shared service or explicitly design its failover; replacing every session with JWT does not solve all state.

A dedicated proxy centralizes policy and works with heterogeneous clients but adds a critical dependency and traffic hop. Client-side balancing removes that shared data-plane hop but needs discovery, policy rollout and compatible implementations in callers. A sidecar/service mesh standardizes many policies near workloads, while adding CPU/memory, proxy hops and control-plane complexity. It is not automatically equivalent to zero-hop client-side latency. Measure the actual cost before using a mesh as a performance fix.

Production scenarios and operational checklist

A database failover with two apparent leaders requires fencing and cluster-aware routing, not just TCP health. A pathological WAF/extension can exhaust the proxy hot path; stage changes, constrain resource use and preserve rollback. A proxy reload can disrupt long-lived clients; verify continuity and reconnect behavior rather than asserting a particular Discord configuration incident. These are explicit generic scenarios replacing unsupported incident details.

Watch for half-open connections and middlebox idle timeouts; use application deadlines and appropriate keepalive, not a universal 30-second magic number. Bound reconnect bursts with jitter/admission and retry budgets. Use hysteresis without hiding sustained failure. Monitor certificate expiry at each terminating hop and test renewal. Redundancy of the backend tier is insufficient when a single proxy or a shared bad configuration can fail the whole path.

Recall and worked answers

1. Choose L4 or L7

Use the information required: HTTP path/header routing needs HTTP awareness; opaque flow balancing can use L4. TLS termination alone does not force L7. Throughput is a measured product/workload property, not a universal layer ratio.

2. Forward versus reverse proxy

A forward proxy represents clients to destinations; a reverse proxy represents servers to callers. Many balancers are reverse proxies, but DNS/direct-routing designs disprove the word “always.”

3. Least connections versus round robin

Least connections can reduce assignments to peers with long-lived occupied connections. It cannot see every request inside one multiplexed connection and may favor fast failures. Equal request count does not imply equal CPU or queueing; inspect all relevant signals.

4. What does proxy_next_upstream guarantee?

It permits selected next-peer attempts under documented conditions, subject to replayability, response progress and configured budgets. It does not make crashes mid-response invisible or safely replay every write. A client-visible error can remain even with a healthy alternative.

5. Ten clients behind NAT

IP-based affinity may concentrate them on one peer. Cookie affinity or an application key can distinguish sessions; stateless/shared-state designs reduce dependence on stickiness. None guarantees durable recovery from a dead backend.

6. Forwarded headers

X-Forwarded-For and X-Forwarded-Proto communicate client address/scheme under a trusted-proxy contract. Overwrite untrusted values at the edge or parse only a validated chain; prevent direct backend bypass where necessary. Headers are assertions, not authentication by themselves.

7. One hot backend passes health

A shallow /health may test liveness rather than useful progress. Inspect route skew, affinity, queue depth, CPU and request costs. Do not blindly eject every overloaded node at once; use bounded admission/load-aware routing and dependency-aware readiness appropriate to the failure mode.

8. Draining

Stop new assignments, wait for current work within a defined grace budget, then terminate or force-close according to policy. Long streams need a reconnect/resume story. There is no universal 30-second/5-minute setting or absolute zero-error guarantee.

9. Firewall idle timeout

A five-minute silent middlebox expiry can make a pooled connection stale. Pool idle expiry below the known boundary, suitable keepalive, deadlines and safe reconnect/retry can help. Keepalive does not prove application responsiveness or repair an already committed duplicate write.

10. Read-replica pool health

Check query capability, intended read-only role, required replication/freshness position and dependency reachability under bounded read-only probes. A disconnected replica may still meet a permitted stale-read contract; a “healthy” SELECT 1 may violate a freshness requirement. Document fail-open/fail-closed behavior and never substitute a wrong writer merely to keep traffic flowing.

Explain-out-loud test

Draw client/proxy/backend connections, explain which hop terminates TLS, and distinguish passive failure detection, active probing, admission, draining and retry. A complete answer identifies the unexecuted integration test that would prove behavior for a chosen product release.

Connections and primary sources

Load balancing feeds caching, CDN, Kubernetes service/ingress and multi-region design, while per-attempt metrics feed SLOs. Those links explain the role, not a claim that all named products implement identical algorithms.

NGINX documentation captured on 2026-10-08 supplies the exact upstream/keepalive/retry contracts; the 2026-10-10 review replayed those bytes and read the certificate-verification default. A new Prometheus documentation capture qualifies the restored metric expressions; queue/retry arithmetic is an authored scenario, not measured service behavior. The pure model executes; Python syntax and config directives are reviewed. No NGINX parser/runtime, TLS proxy, container, WAF, live failover or benchmark was executed.

• NGINX HTTP load balancing
• NGINX upstream module
• NGINX proxy module

Original resource trail and reading purpose

The original session allocated 90 minutes for intuition, visuals, practice, production reasoning and recall. Treat that as optional pacing, not a completion or diagnostic-speed guarantee. Read the mechanisms here before attempting separately authorized service work.

These original companion destinations are retained for discovery, not used as reviewed claim evidence. Video identity, attribution, duration, playback and external reachability have not been verified in this pass; search links are searches, not specific videos. Historical promotional claims and inconsistent labels are not repeated as facts.

• Original video/search destination 1 — balancing overview.
• Original video/search destination 2 — flow versus HTTP routing.
• Original video/search destination 3 — reverse-proxy configuration study.
• Reverse Proxy vs. Load Balancer — proxy role definitions.
• Introducing the AWS Application Load Balancer — product-specific application-routing context, not proof of every NLB capability.
• Introduction to Modern Network Load Balancing and Proxying — balancing architecture tradeoffs.

For reviewed propositions use the primary sources above and the source-qualified mechanisms, not the optional resource labels. Prerequisites: S060 OS basics and S061 networking. Next: S063 caching. The original label “S015 HTTP basics” was incorrect: S015 is Big-O; HTTP familiarity remains a prerequisite, not a mislabelled link. Hub: The 6-Month Learning Plan. Navigation indicates a route, not acceptance of another lesson.