Search Tech Journey

Find topics, journeys and posts

6-month learning plan64 / 130
back to blog
systemsintermediate 50m read

S064 · CDN — Edge, Cache Hierarchies, Cache-Control

How your static asset travels 40 ms to Sydney instead of 400 ms — edges, origins, cache hierarchies, and the four Cache-Control directives you'll set every day.

⚙️SystemsM07 · Systems & Infrastructure· Session 064 of 130 90 min

🎯 Set correct Cache-Control on every response, understand origin shield + cache hierarchies, and configure a Cloudflare-style CDN in front of an origin — with purge working.

Why this session exists

A CDN is the highest-leverage infrastructure decision you'll make: turn it on and your global P95 drops from 400 ms to 40 ms overnight, your origin's egress bill drops 90%, and you gain DDoS protection almost for free. Turn it on incorrectly and you cache personal data at the edge, or your users see yesterday's home page for a month. This session teaches both sides — the mental model AND the exact Cache-Control incantations.

You will be able to
  • Explain edge · shield · origin hierarchy, and what problems each layer solves.
  • Set Cache-Control, ETag, and Vary correctly on static assets and API responses.
  • Purge a CDN cache surgically without nuking every object.
  • Diagnose the top-4 CDN bugs: caching authenticated responses, missing Vary, over-long TTLs, and cache-key mismatches.

Prerequisites

  • S015 · HTTP basics — headers, methods, status codes.
  • S062 · Load balancers — reverse-proxy mental model.
  • S063 · Caching strategies — cache-aside, TTL, invalidation.


(a) Intuition · 5 min

Bookstores vs the Library of Congress
🌍 Real world

There is one Library of Congress in Washington DC that owns every book. If everyone on Earth had to fly there to read one page, reading would collapse.

Instead, thousands of local bookstores keep copies of the popular titles. You walk in, grab a copy, done. If the store doesn't have what you want, they order it from the central library — usually once — and now every subsequent local reader gets a copy from the store too.

Bookstores stock the top 1% of titles because that 1% accounts for 90% of demand. The long tail still exists at the central library; you just wait longer.

💻 Code world

Your origin is the Library of Congress — one place (or two, one per region) that has every asset. Your CDN edges are the local bookstores — hundreds or thousands of PoPs (points of presence) around the world, each holding whatever's been recently requested nearby.

The first request from Sydney for /logo.png travels to your origin in Virginia (~200 ms), gets cached at the Sydney PoP, and every subsequent Sydney user gets it in 15 ms. This is why static-heavy sites feel identical everywhere on Earth: the bytes never leave the reader's continent.

What a CDN actually gives you

Five things you get by enabling one, in order of value
  • Latency reduction — bytes come from ~50 km away, not ~5000 km. Global P95 drops 10×.
  • Origin offload — 95%+ of requests never touch your server. Egress bandwidth bill drops accordingly.
  • DDoS absorption — the edge fleet has 200+ Tbps aggregate. Attacks that would flatten your origin bounce off.
  • TLS termination at edge — one place to rotate certs; clients get sub-100 ms TLS handshake.
  • Programmable edge — modern CDNs run your code (Cloudflare Workers, Lambda@Edge) 30 ms from the user.

A quick history so the products make sense

  1. 1998
    Akamai founded · MIT spin-off
    First commercial CDN. Solves ‘the internet is slow’ for early web publishers.
  2. 2008
    CloudFront launches
    AWS makes CDN a self-service commodity. Prices collapse.
  3. 2010
    Cloudflare launches
    Free tier + DDoS protection. Onboards millions of sites in a few years.
  4. 2017
    Cloudflare Workers · edge compute
    V8 isolates at the edge. Your code runs 30 ms from any user, no cold starts.
  5. 2020
    QUIC + HTTP/3 rollout
    UDP-based, 0-RTT resumption. CDNs deploy globally before origins do — another edge win.

(b) Visual walkthrough · 15 min

The cache hierarchy — three tiers

Why the middle tier exists. Without a shield, every one of the 200 edge PoPs misses independently and hits your origin. On a cold cache after purge, that's 200 simultaneous fetches for one asset. The shield collapses those 200 misses into 1 fetch to origin. Cloudflare calls this "tiered caching"; Fastly calls it "origin shield"; CloudFront calls it just "origin shield" too. Turn it on — always.

Anatomy of a single cached response

The four headers that decide EVERYTHING about caching

Cache-Control
The primary directive. `public, max-age=31536000, immutable` caches for a year at every layer. `no-store` skips caching entirely. Learn the vocabulary — see next section.
primary
ETag / Last-Modified
Server fingerprints of the current version. When TTL expires, cache does a conditional request (`If-None-Match`), origin responds 304 Not Modified if unchanged — bytes stay put, only metadata refreshed.
validator
Vary
Tells the cache ‘this response depends on request header X, cache separate versions per value’. `Vary: Accept-Encoding` = separate cache entries for gzip vs brotli.
keying
Age
Response header showing how many seconds this entry has been in the cache. Useful for debugging (‘why is this stale? — Age says 3600 s and max-age is 60 s’).
debug

The Cache-Control directives you need memorised

Static asset

Hashed filename · cache forever

  • Cache-Control: public, max-age=31536000, immutable
  • Filename includes content hash: /app-a3f9c2.js
  • New deploy = new hash = new URL — no invalidation needed
  • Standard for JS/CSS bundles, fonts, hashed images
HTML page

Short TTL · validators

  • Cache-Control: public, max-age=60, must-revalidate
  • Or: no-cache (fetch, but 304 is allowed)
  • ETag with content hash so 304 responses are cheap
  • Standard for the shell that loads your hashed assets
API response · public

Short TTL · Vary carefully

  • Cache-Control: public, max-age=30, s-maxage=300
  • s-maxage overrides max-age at the CDN specifically
  • Vary: Accept-Encoding, Accept-Language
  • Good fit: product catalogues, public feeds
Authenticated / personal

Never cache at CDN

  • Cache-Control: private, no-store
  • ‘private’ = browser MAY cache; ‘no-store’ = don't even
  • Skip this and you'll serve user A's data to user B
  • Set explicitly on every /api/me, /account, /admin route

The 90/9/1 rule you can hold in your head

1
90% edge hit

Static assets + long-TTL API responses. Served in 20 ms from the nearest PoP.

2
9% shield hit

Warm asset that expired at edge but is still fresh at the regional shield. ~60 ms.

3
1% origin hit

Cold or purged content. ~150 ms. This is where your origin bandwidth bill lives.

4
Origin protection

During a cache purge, tiered caching collapses N edge misses → 1 shield fetch → 1 origin fetch.


Common misconception
✗ What most people think

"A CDN is for static assets — images, CSS, JavaScript. Dynamic and personalised content can't be cached, so a CDN can't help there."

✓ What is actually true

Static caching is the smallest benefit. A CDN terminates TLS close to the user, which removes multiple round trips from every request including uncacheable ones; it keeps warm, congestion-window-mature connections to your origin so your dynamic content rides an already-fast pipe; it absorbs volumetric attacks; and with edge compute it can assemble personalised responses from cached fragments. Even a pure pass-through for a 100% dynamic API is usually meaningfully faster.

Why the myth is so sticky

The myth is sticky because caching is the feature CDNs are sold on, and it is the one you configure explicitly, so it becomes the mental summary of the whole product. The connection-level benefits are invisible: nothing in a dashboard says "you saved two round trips on TLS negotiation", so the largest win for dynamic traffic is the one nobody attributes.

Prove it to yourself

Measure where the time actually goes on an uncacheable request, with and without the edge:

curl -w 'dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' \
  -o /dev/null -s https://your-api.example.com/dynamic

# compare the gap between time_appconnect and time_starttransfer
# (origin processing) with everything before it (connection setup).
# On a distant origin, setup often dominates - and that is exactly
# the part an edge POP removes, cacheable or not.
From first principles
Start with the question

Why does physical distance to a server matter so much, when a request is only a few kilobytes and links are gigabit?

  1. 1
    Signals in fibre travel at roughly two-thirds the speed of light, so distance imposes a hard minimum latency that no engineering can reduce.
    forced by · it is a physical constant, not an implementation detail — London to Sydney is about 80 ms one way at best, and real routes are longer than great-circle distance
  2. 2
    Establishing a connection is not one exchange. TCP needs a handshake, TLS needs its own, and only then does the request go out.
    forced by · each layer must confirm the peer before trusting it, and confirmation requires a message and a reply
  3. 3
    Therefore the fixed cost of a request is several round trips, and each round trip pays the full distance penalty twice.
    forced by · latency multiplies by the number of sequential exchanges, and none of them can be skipped or parallelised
  4. 4
    TCP slow start then compounds this: a new connection cannot use available bandwidth immediately, so a large response takes several further round trips to ramp up.
    forced by · the sender must probe capacity incrementally, as derived in the networking session
  5. 5
    So the only way to reduce this is to shorten the distance over which those round trips occur — terminate the connection near the user and let the long-haul segment run over a pre-established, already-warmed connection.
    forced by · you cannot reduce the number of round trips below the protocol's minimum, so you must reduce the cost of each one
⇒ Therefore

Therefore a CDN's primary mechanism is moving the connection setup close to the user, not storing files. Caching is a further optimisation layered on top of that.

And note what this predicts: any protocol change that removes a round trip should produce a benefit comparable to moving the server closer. That is exactly what TLS 1.3 (one round trip instead of two) and QUIC/HTTP-3 (combining transport and crypto handshakes, with 0-RTT resumption) deliver. It also predicts that the benefit of an edge is largest for chatty, small-payload, high-latency workloads — mobile users on distant networks — and smallest for a single bulk transfer, where the ramp cost is amortised across many megabytes.

Mental modelMove the front door, not the building

Your origin stays where it is. The CDN puts a front door in every city: the user's expensive handshake happens metres away, and the long journey to the origin happens over a connection the CDN already established and warmed. If the answer is already at the front door, the journey never happens at all.

Cache keys are the whole configuration problem. A cache key that includes something varying per user gives you a 0% hit rate; a key that omits something important serves one user's content to another.

  • The origin controls caching via headers, and the two axes are separate: Cache-Control: max-age for browsers, s-maxage for shared caches, and stale-while-revalidate to decouple user latency from revalidation. Setting only max-age means you cannot purge from users' browsers — one reason long browser TTLs are dangerous and long edge TTLs are safe.
  • Normalise the cache key aggressively: strip tracking query parameters, restrict Vary to headers that genuinely change the response. Every additional dimension in the key multiplies the number of cached variants and divides your hit rate.
  • Cache-busting belongs in the URL. Immutable, content-hashed filenames with a very long TTL are strictly better than short TTLs plus purging, because they make the cache key change when the content does — no invalidation required.
  • Edge compute changes the shape of what is cacheable: cache the page, personalise a fragment at the edge. This converts "this page is personalised so nothing is cacheable" into "99% of this page is cacheable", which is usually the difference that matters.
🔔 Fires when you see

Fire this model when you see: a hit rate near zero on assets that should never change · users in one region reporting slowness · a deploy where old assets are still being served · one user seeing another's personalised content · an origin taking traffic it should never see.

The tradeoff

How long should the edge TTL be for content that changes unpredictably?

Short TTL
+ you gain changes propagate quickly with no operational action, and the maximum staleness is bounded and obvious. Nobody needs to remember to purge, which means nobody can forget to.
− you pay hit rate falls and origin load rises proportionally, because every expiry sends a request home. At scale this is a real capacity cost, and it is paid continuously for content that may not have changed in weeks.
pick when when content changes often and unpredictably, and origin capacity is not the constraint
Long TTL with active purging
+ you gain a very high hit rate and minimal origin load, while changes still propagate quickly because a purge is issued on write. Best of both when it works.
− you pay correctness now depends on the purge firing every time content changes — a code path that is easy to forget in a new feature and easy to break silently. Purges also take time to propagate globally, so there is a window regardless.
pick when when writes are relatively infrequent and pass through a small number of well-understood code paths where the purge can be reliably attached
Immutable URLs with an effectively infinite TTL
+ you gain the cache can never be wrong, because a change produces a different URL. Hit rates approach 100%, no purge infrastructure is needed, and rollback is instant since the old URL still resolves.
− you pay requires a build step that hashes content and rewrites every reference, and the referencing document itself must have a short TTL — you have moved the problem to exactly one place rather than eliminated it.
pick when all build-time assets without exception: JavaScript, CSS, images, fonts
What a senior engineer actually does

Use immutable URLs for anything produced by a build, and long TTL plus purge for content managed through a CMS or admin interface where the write path is small and known. Reserve short TTLs for content that changes continuously and where purging every change would be its own volume problem.

The mistake worth avoiding is a uniform TTL policy across all content. Different content has genuinely different change rates and different staleness costs, and one number cannot be right for all of it. Set the policy per content class, and — critically — keep browser TTLs short even where edge TTLs are long: you can purge an edge, but you can never reach into a million browsers to correct a mistake.


(c) Hands-on · 25 min

We'll build a tiny origin server that sets Cache-Control correctly for each route type, run a local Varnish (a widely-used HTTP cache) in front of it to simulate a CDN edge, and watch the behaviour with curl. Then we'll do a targeted purge and see it work.

#!/usr/bin/env bash# s064-cdn-demo.sh local origin + Varnish acting as an edge cache.# Requires: python3, docker.set -euo pipefail DIR="$HOME/projects/learning/s064"mkdir -p "$DIR" && cd "$DIR" log() { printf "\033[1;36m %s\033[0m\n" "$*"; } log "1/4 Writing origin.py sets Cache-Control per route"cat > origin.py <<'PY'"""origin.py a tiny origin with correct per-route caching headers."""

What each block does

Anatomy of the origin + VCL

origin.py · 4 route types
Each route sets Cache-Control appropriate to its class: immutable for hashed assets, short-TTL+ETag for HTML, public with s-maxage for public APIs, private+no-store for /me. This is the shape you'll set in every real app.
origin
ETag + 304 handling
The HTML route hashes its body and returns 304 Not Modified if the client sends the same ETag. This is the bandwidth-saver even when the edge TTL is short — origin returns 43 bytes instead of the full page.
validator
s-maxage vs max-age
The API route says ‘browser caches 30 s, CDN caches 300 s’. s-maxage is CDN-specific. This lets you refresh eagerly per-user while still absorbing traffic at the edge.
control
VCL vcl_recv PURGE
Allows out-of-band cache invalidation. In production this endpoint must be firewalled OR require auth (every CDN provides this via its API). Never leave PURGE open to the internet.
purge
VCL uncacheable on no-store
Explicitly tells Varnish not to cache when the origin says private/no-store. Without this, if Varnish's own TTL rules disagree with the origin's headers, you cache authenticated pages. Enemy of the state.
safety
X-Cache HIT/MISS
Debugging discipline: every CDN response gets an X-Cache header from your edge. Grep for MISS in production logs to find hot origin-punching endpoints.
debug
Try itBreak the cache with a missing Vary header and see what goes wrong

Add gzip encoding when the client requests it, but forget the Vary: Accept-Encoding header. Then:

# Client 1 asks for gzip:
curl -sH "Accept-Encoding: gzip" http://127.0.0.1:8080/api/catalogue | file -
# Cache now stores the gzipped body under key `/api/catalogue`.
 
# Client 2 does NOT ask for gzip:
curl -s http://127.0.0.1:8080/api/catalogue
# You'll see raw gzip bytes — the browser would show garbage.

Add Vary: Accept-Encoding and the cache keys become (/api/catalogue, gzip) vs (/api/catalogue, identity) — two separate entries, correct content to each client. This is one of the top-3 CDN bugs in the wild.

💡 Hint · Remove the Vary handling and serve gzip to one client, uncompressed to another — the second client will get gzip bytes it can't decode.

(d) Production reality · 15 min

War story Steam (Valve)· 201534,000 users saw other users' account pages · Christmas Day
🔥 What broke

On December 25 2015 Steam was under DDoS. Their CDN (a bespoke setup) had cache rules tuned for the attack. A misconfiguration made the edge cache authenticated pages — including account details, phone numbers, and last four digits of credit cards.

Users refreshing the store page were served the last cached response, regardless of who they were logged in as.

🧯 The fix
Emergency: Steam disabled the store entirely for hours. Long-term: hard rule that any response with Set-Cookie, Authorization, or a session cookie is never cacheable at the edge — enforced in VCL, not left to origin discipline.
🎓 Lesson to steal
‘Cache-Control set by origin’ is a promise the origin might break under stress. Belt-and-braces: your CDN layer should ALSO refuse to cache anything with auth-shaped headers. Two independent barriers.
Post-mortem
War story Cloudflare· 2017months of leaked customer data (‘Cloudbleed’)
🔥 What broke
A buffer-overflow bug in Cloudflare's HTML parser at the edge leaked snippets of memory — including other customers' HTTPS request/response bodies — into cached responses. Search engines then crawled and cached those pages themselves.
🧯 The fix
Cloudflare pushed a hotfix in hours, contacted major search engines to purge affected snapshots, and rewrote the parser in a memory-safe language. Post-mortem is textbook. But the underlying lesson: anything that sits in the request path at edge-scale is a systemic risk.
🎓 Lesson to steal
The edge sees everything. Code that runs there — WAF rules, edge functions, header transforms — is on the hot path for the entire internet. Treat it like kernel code, review like release engineering.
Post-mortem
War story Common failure mode · every marketing site· 2024widespread
🔥 What broke
Marketing pushes a new home page at 09:00. The CDN caches the old page for 24 hours because the origin returned Cache-Control: public, max-age=86400 and no purge was issued. Users see the old page all day. Support inbox explodes.
🧯 The fix
Add a deploy-time step that hits the CDN's purge API (Cloudflare: POST /purge_cache) with the list of changed URLs. Or use tag-based purging: Cache-Tag: home-v2 on the response, purge by tag on deploy. Fastly and Cloudflare Enterprise both support this natively.
🎓 Lesson to steal
Long TTL + no purge = stuck data. Either integrate purge into the deploy pipeline OR use versioned URLs so ‘new content’ = ‘new URL’ (never purge again). The Netflix/Airbnb pattern: every static asset has a content hash in its name.

Common failure modes to internalise

  • Caching set-cookie responses — the cache serves user A's session cookie to user B. Every LB/CDN default rule should refuse to cache responses with Set-Cookie.
  • Missing Vary — same URL returns different bodies based on Accept-Encoding, Accept-Language, or Authorization. Cache stores one, serves it to everyone. Fix: Vary header lists which request headers partition the cache.
  • Cache key too coarse — a URL like /api/products?category=x&sort=y — different query strings should be different cache entries. Most CDNs cache by full URL including query string by default; verify yours.
  • Cache-Control set from framework defaults — Django/Express/etc. often ship Cache-Control: no-cache on every response until you configure otherwise. You'll get 0% cache hit ratio and wonder why the CDN "isn't working."

Where this shows up in the rest of the plan

CDN patterns propagate through the plan
S062 · Load balancers
The origin sits behind an LB. CDN → LB → app fleet.
S063 · Caching strategies
The same cache-aside primitives, applied at the HTTP/edge layer.
S068 · Azure Cloud
Azure Front Door + CDN products; how the CSPs package the same primitives.
S077 · Web performance
Core Web Vitals live and die on CDN hit ratio + smart Cache-Control.
S085 · Security · DDoS
The CDN is the DDoS shield. What it absorbs vs what it forwards.
S095 · Edge compute
Cloudflare Workers, Lambda@Edge — your code on the CDN itself.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

  1. How does a CDN reduce global P95 latency by 10×? (edge PoPs + hierarchy)
  2. Which single header decides whether a response gets cached at the edge, and what are the four values you'll set most often?
  3. What are two ways to invalidate a CDN cache, and when do you prefer versioned URLs over purge?

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.