6-month learning plan · · session 82 / 13082 / 130

systemsintermediate 55m read

S082 · TLS 1.3, PKI & Cert Lifecycle

The green padlock, demystified — key exchange, certificate chains, revocation, and why 60 % of production outages the past decade were expired certs.

🔒SecurityM10 · Security· Session 082 of 130 90 min

🎯 Understand how TLS 1.3 negotiates a session, how PKI proves identity, and how to keep certificates alive without paging yourself at 3 a.m.

Why this session exists

Every request your app makes over HTTPS goes through a dance older than half the engineers reading this — a dance that combines Diffie–Hellman key exchange, X.509 certificates, chain-of-trust validation, and clock synchronisation, all in under 100 ms. When it works, you never think about it. When it breaks, a single expired cert takes down Slack, Microsoft Teams, or the entire Fastly edge for three hours. This session is the layer under the padlock, told in the order you'd actually need to debug it.

You will be able to

Explain the TLS 1.3 handshake — client hello, key share, cert, finished — in under a minute without notes.
Read an X.509 certificate with openssl and identify subject, issuer, SAN, validity, and signature algorithm.
Explain why cert expiry causes outages, and design a rotation strategy that survives DNS + humans.
Diagnose the top-4 TLS errors (expired, self-signed, hostname mismatch, unknown CA) in under 60 seconds.
Choose between Let's Encrypt, an internal CA, and a paid CA for a given system with real trade-offs.

Prerequisites

S055 · HTTP fundamentals — you know how a request/response works over TCP.
S060 · Linux fundamentals — comfortable with openssl on the command line.
S081 · Authentication vs authorization — you understand identity vs permission.

Pre-read · skim before the session (30 min total)

💡 Intuition 13 min

How does HTTPS work? What's a CA? What's a self-signed Certificate?

Hussein Nasser

The clearest ‘chain of trust’ walkthrough on YouTube. Watch once, ideas stick forever.

🔬 Deep dive 16 min

TLS Handshake Explained - Computerphile

Computerphile

Mike Pound walks through the handshake with an actual whiteboard. TLS 1.2, but the mental model transfers.

🛠️ Hands-on 20 min

Wireshark TLS 1.3 handshake — packet-by-packet

YouTube search · pick top result

Seeing the four flights of packets in Wireshark makes the abstract concrete.

📖 Reading 10 min

What happens in a TLS handshake?

Cloudflare Learning Center

Cloudflare terminates more TLS than almost anyone. Their tutorial is the canonical primer.

📖 Reading 12 min

How Let's Encrypt Works

Let's Encrypt · official

The ACME protocol in eight paragraphs. Read this and you'll never manually renew a cert again.

📖 Reading 25 min

Everything you should know about certificates and PKI

smallstep · Mike Malone

If you read one long-form PKI post in your life, this is it. Bookmark it — you will re-read it during your first incident.

(a) Intuition · 5 min

A passport control checkpoint you can't see

🌍 Real world

You land in a foreign country. At immigration, the officer inspects your passport — a document signed by your government. The officer doesn't personally know your government, but they trust it because their government (via the UN, treaties, and shared cryptographic keys) tells them to. If the passport is expired, or the photo doesn't match, or the country isn't recognised, you don't get in.

The whole exchange takes 30 seconds and involves cryptographic verification, an implicit chain of trust between governments, and a mutually-agreed protocol. Nobody at the counter explains it to you.

💻 Code world

TLS is the same, in milliseconds. Your browser lands at bank.com. The server hands over its certificate — a document signed by a Certificate Authority (CA). Your browser doesn't personally trust bank.com, but it ships with a bundle of ~140 root CA public keys, and if the certificate chain climbs to one of them, the CA vouches for the site's identity.

Then browser and server agree on a shared session key using elliptic-curve Diffie–Hellman — a maths trick that lets two parties compute the same secret in public without ever transmitting it. Every byte after that is encrypted with AES.

The three problems TLS solves — remember this trio

TLS solves exactly three problems, and nothing else

Confidentiality — nobody between you and the server can read the payload (AES-GCM or ChaCha20-Poly1305 in TLS 1.3).
Integrity — nobody between you can modify the payload without detection (built into AEAD ciphers).
Authenticity — the server you're talking to is actually the one whose domain is in the URL bar (this is the certificate part, and it's the only part that involves PKI).

The history of an accidental empire

1994
SSL 1.0 · Netscape
Never released — cryptographers laughed it off the whiteboard. SSL 2.0 (1995) shipped and was also broken by 1996.
1996
SSL 3.0
The version everyone actually used. Deprecated 2015 after the POODLE attack; still surprises us in log files.
1999
TLS 1.0 · IETF
SSL 3.1 renamed. Political move to escape Netscape's brand. Cryptographically similar.
2008
TLS 1.2
SHA-256, AES-GCM, authenticated encryption. Ran the internet for a decade.
2015
Let's Encrypt
Free, automated CA. Kills the $100/year certificate market. HTTPS adoption jumps from 25 % to 90 %+ in five years.
2018
TLS 1.3 · RFC 8446
1-RTT handshake, forward secrecy mandatory, all old ciphers removed. The version we teach today.
2024
Post-quantum hybrids
Chrome + Cloudflare ship X25519-Kyber768. Preparing for the day large quantum computers break ECDH.

(b) Visual walkthrough · 15 min

The TLS 1.3 handshake — four messages, one round trip

Compare to TLS 1.2 which took two round trips — TLS 1.3 halved the handshake latency for every HTTPS request on the internet.

The chain of trust — how your browser decides to trust a stranger

Client sends ClientHello

Includes SNI (Server Name Indication — which site on this IP?), supported cipher suites, and a fresh ECDHE public key.

Server picks cipher + sends key_share

Server picks strongest mutually-supported cipher, sends its own ECDHE public key. Both sides now derive the same session key.

Server sends certificate (encrypted)

The leaf cert + intermediates. Only the client can read it — everything after msg 2 is encrypted.

Client validates chain

Walks up: leaf → intermediate → root. Checks signatures, validity dates, hostname match against SAN.

Both send Finished

MAC over the entire handshake — proves nobody tampered with any earlier message. Handshake complete.

Encrypted application data flows

Every subsequent byte is AES-GCM or ChaCha20-Poly1305 with the shared session key. Attacker sees ciphertext only.

Certificate anatomy — a leaf cert dissected

What's actually inside a certificate (X.509 v3)

Subject

Who this cert is for. Legacy CN=bank.com; modern browsers ignore CN and use SAN instead.

identity

Subject Alternative Name (SAN)

The actual list of domains this cert covers: bank.com, www.bank.com, api.bank.com. Multiple entries allowed.

identity

Issuer

The CA that signed this cert. E.g. ‘R3 · Let's Encrypt Authority’.

trust

Validity (notBefore / notAfter)

The two dates that cause 60 % of outages. Let's Encrypt = 90 days. Public CAs cap at 398 days (was 825, dropping to 47 by 2029).

lifecycle

Public Key

The key the client encrypts to. RSA 2048/4096 or ECDSA P-256/P-384. ECDSA is smaller + faster.

crypto

Signature

The issuer's signature over everything above. Verify with issuer's public key → chain climb continues.

crypto

Extensions

Key usage (signing? encryption?), extended key usage (server auth? client auth?), CRL distribution points, OCSP URLs, SCTs (Certificate Transparency).

Choosing a CA — the three options

Let's Encrypt

Free · automated · 90 days

$0. Sponsored by ISRG (nonprofit).
90-day cert forces automation (a feature, not a bug).
ACME protocol — certbot, acme.sh, cert-manager on k8s.
Trusted by every modern browser + OS.
Not suitable if you can't run automated challenges (HTTP-01 or DNS-01).

Internal CA (private PKI)

For service-to-service · not browser-facing

You run your own root CA (smallstep, HashiCorp Vault PKI, AWS PCA).
Every service gets its own cert, signed by your CA.
Root distributed only to your fleet — browsers never see it.
Perfect for mTLS between microservices.
You now own key management, rotation, revocation. Non-trivial.

Paid CA (DigiCert / Sectigo / GlobalSign)

Enterprise · EV certs · human support

$100 – $2000/year per domain (wildcards more).
Extended Validation (EV) shows company name in some UIs (mostly deprecated now).
Human support during incidents.
Legal contracts + insurance around cert misissuance.
Chosen mostly for compliance/procurement reasons, not technical ones.

Where certs die — the failure modes

Common misconception

✗ What most people think

"The padlock means the site is safe. HTTPS encrypts the connection, so if I see the lock, the data is protected and the site is legitimate."

✓ What is actually true

TLS gives you exactly three properties on the transport: confidentiality, integrity, and authentication of the server's domain name. It proves you are talking to whoever controls that hostname — nothing about whether they are honest, whether the app is secure, or what happens to your data after it arrives. A phishing site with a free 90-day certificate has a perfectly valid padlock, because it genuinely does control its domain.

Why the myth is so sticky

The myth is sticky because it was nearly true in the era when certificates were expensive and issued only after manual organisational vetting. A padlock in 2005 did weakly imply "someone paid and was checked". Automated domain-validated issuance made certificates free and instant, which was an enormous net win for the web — and simultaneously severed the link between "encrypted" and "trustworthy". Browsers reinforced the old intuition for years with green bars and reassuring language before quietly removing them.

Prove it to yourself

Inspect what the certificate actually asserts. Look at the subject and the issuer, and notice how little identity information a DV cert contains:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

# subject is usually just: CN = example.com
# no company, no address, no vetting -- only 'controls this DNS name'
# and the SAN list, not CN, is what browsers actually match on

From first principles

Start with the question

Why does TLS need certificates at all? Diffie–Hellman already lets two strangers agree on a shared secret over a public wire without ever transmitting it. If we can get encryption for free, why drag in a whole global hierarchy of authorities?

1
Raw Diffie–Hellman gives two parties a shared secret that a passive eavesdropper cannot compute, even having seen every byte exchanged.
forced by · the discrete-log problem makes deriving the secret from the public values computationally infeasible
2
But it gives no evidence about who is at the other end. An active attacker in the path simply runs DH twice — once with you, once with the real server — and sits in the middle decrypting, reading, and re-encrypting. Both sides see a flawless handshake.
forced by · nothing in the DH exchange is bound to an identity; it authenticates nobody
3
So the server must prove identity by signing something only the real server could sign — a private key whose public half you can verify. Now confidentiality plus authentication is achievable.
forced by · a signature over the handshake transcript binds the key exchange to the key holder
4
That merely relocates the problem: how do you know that public key belongs to the domain you typed? Fetching it over the network is circular — an attacker who can intercept the connection can also swap the key.
forced by · you cannot bootstrap trust from a channel that is itself untrusted
5
Therefore trust must be pre-installed out of band. Your OS and browser ship with a set of root CA public keys, delivered with the software itself. A CA signs a statement binding a domain to a public key, and you verify that signature with a key you already had before the connection existed.
forced by · at least one key must arrive by a path the network attacker does not control
6
Roots are then kept offline and used only to sign intermediates, which do the day-to-day issuing — so a compromised issuing key can be revoked without invalidating the root that is baked into a billion devices.
forced by · a key you cannot rotate must never be exposed to routine operations

⇒ Therefore

Therefore PKI exists to solve the one thing cryptography cannot solve by itself: key distribution. Encryption is the easy half; knowing whose key you hold is the hard half, and it is unavoidably a trust-anchor problem, not a math problem.

And note what this predicts: the entire system's security is bounded by its weakest root, because any trusted CA may issue for any domain. That is precisely why Certificate Transparency logs and CAA records exist — they do not prevent mis-issuance, they make it publicly detectable and let a domain owner declare which CAs are allowed to issue for it. Both are patches for a structural weakness the derivation predicts must exist.

Mental modelSealed envelope from a vouched-for sender

Picture two separable jobs. The envelope is sealed so nobody can read or alter the contents in transit — that is the symmetric session key, negotiated fresh per connection, doing the actual bulk encryption. The signature and letterhead prove who sealed it — that is the certificate chain, and it is used only during the handshake.

Public-key crypto is slow, so it is used only to establish and authenticate a fast symmetric key; everything after the handshake is symmetric. The chain of trust walks upward — leaf signed by intermediate, intermediate signed by root, root already on your machine — and you stop the moment you reach a key you had beforehand.

Handshake = asymmetric (identity + key agreement). Session = symmetric (bulk data). The expensive part happens once.
Validation is a chain walk to a pre-installed root, plus hostname matching against the SAN list, plus expiry, plus revocation status. All four must pass; browsers are weakest on the last one.
Forward secrecy comes from ephemeral key exchange: the session key is never derivable from the long-term private key, so stealing the server key tomorrow does not decrypt traffic captured today.
TLS protects data in transit only. It ends at the terminating proxy or load balancer — beyond that point, unless you re-encrypt, it is plaintext on your own network.

🔔 Fires when you see

Fire this the moment you see: certificate errors being clicked through or verification disabled to "make it work" · a cert expiring at 3am and taking a service down · TLS terminated at the load balancer with plaintext behind it · pinning that will break at renewal · a self-signed cert in production · an internal service where nobody can name what the trust anchor is.

The tradeoff

For service-to-service traffic inside your own network, do you terminate TLS at the edge, run mutual TLS everywhere, or terminate at the edge and re-encrypt internally with a private CA?

Terminate at the edge, plaintext internally

+ you gain simplest to operate and debug — you can tcpdump internal traffic, and there is exactly one certificate to renew; lowest CPU cost and lowest latency on internal hops

− you pay a single compromised host, misconfigured mirror port, or hostile tenant on shared infrastructure sees everything in the clear; you have made an implicit bet that the network perimeter holds, which is the assumption every lateral-movement breach violates

pick when a genuinely isolated network segment with a low-sensitivity workload and no compliance requirement for encryption in transit — a shrinking set

Full mTLS (both sides present certificates)

+ you gain every hop is encrypted and both ends are authenticated, so service identity becomes cryptographic rather than "whatever IP called us". This is what makes zero-trust and per-service authorization policies actually enforceable.

− you pay you now run a private CA and must issue, rotate, and revoke certificates for every workload — at scale that is only tractable with automation (short-lived certs issued by a mesh or workload-identity system). Debugging gets materially harder, and clock skew or a rotation bug becomes a total outage.

pick when multi-tenant infrastructure, regulated data, or any environment where you must assume the internal network is hostile — and you have the automation to rotate certs without humans

Edge termination + internal re-encryption to a private CA

+ you gain traffic is encrypted end to end without requiring every client to hold a certificate; far less operational surface than full mTLS while removing the plaintext-on-the-wire exposure

− you pay you get confidentiality but not client authentication — any host that can reach the service still looks like a legitimate caller. It is half the security of mTLS for perhaps a third of the operational cost.

pick when you need encryption in transit for compliance but service identity is already handled at another layer (tokens, network policy)

What a senior engineer actually does

The honest driver of this decision is automation, not security preference. mTLS everywhere is strictly better on security and strictly worse on operations, and its failure mode is an outage caused by your own security system — expired certificates take down more services than attackers do. If you have a mesh or workload-identity system that issues short-lived certificates without human involvement, take mTLS. If rotation would be a human running a script, you will eventually page yourself at 3am for a self-inflicted outage.

Whatever you choose, the non-negotiable parts are: automate renewal and alert on expiry well before it happens, never disable verification to unblock a deploy, and be able to name your trust anchor for every connection. Most production TLS incidents are not cryptographic failures — they are expiry, hostname mismatch, and an incomplete chain that worked in curl and failed in a stricter client.

(c) Hands-on · 25 min

We're going to inspect a real certificate, generate a self-signed one, and simulate a Let's Encrypt renewal — all with openssl, which ships on every Unix system.

What each block does

Anatomy of the script

Block 1 · openssl s_client

The Swiss army knife for TLS. Connects to any host:port, prints the negotiated handshake, and dumps the cert chain. Every TLS debugging session starts here.

debug

Block 2 · SAN extraction

The SAN list is the ONLY thing modern browsers check for hostname validation. CN is ignored. If your cert doesn't cover the exact hostname, users get a big red warning.

identity

Block 3 · self-signed with ECDSA P-256

Generates a fresh EC keypair AND signs a cert with itself. -addext puts the SAN in — this trips up 90 % of first-time cert generators.

generate

Block 4 · verify locally

Always inspect what you just made. Wrong dates or missing SAN caught in 5 seconds instead of after deploy.

verify

Block 5 · Python TLS 1.3 server

The stdlib ssl module speaks TLS 1.3 by default on Python 3.10+. Twelve lines to a real HTTPS-ish endpoint.

runtime

Block 6 · three curls

The unencrypted (rejected), the -k (skip verify), the --cacert (trust the specific cert). This is exactly the workflow for internal-CA services.

client

Try itProve hostname validation to yourself in 60 seconds

Modify the self-signed cert to only include DNS:dev.local in the SAN (drop the localhost + 127.0.0.1 entries), keep the CN as localhost. Now run:

curl --cacert server.crt https://127.0.0.1:8443/

You'll get SSL: no alternative certificate subject name matches target host name '127.0.0.1'. Trusting the cert is not enough — the hostname must match a SAN entry. This is the single most common cause of "but I added it to the trust store!" confusion.

💡 Hint · Then re-generate the cert with -addext 'subjectAltName=DNS:dev.local' and see it work. The SAN is doing the work, not the CN.

(d) Production reality · 15 min

War story Microsoft Teams · 2020-02-03· 20203-hour global outage · millions of users

🔥 What broke

The authentication certificate for Microsoft Teams expired at 00:00 UTC. Every client — desktop, mobile, web — instantly failed to authenticate. Teams was down worldwide.

Root cause: the cert was tracked manually in a spreadsheet, and the engineer who owned it had left the team. The renewal reminder went to an unmonitored mailbox.

🧯 The fix

Emergency: issued a new cert and pushed it globally in about three hours. Long-term: automated cert rotation with monitoring + alerts at T-30 days, T-14 days, T-7 days, T-1 day, and expiry.

🎓 Lesson to steal

Any cert with a human name attached to it is a future outage. The only safe cert is one that renews itself and alerts before expiry. Every. Single. Time.

Post-mortem

War story Let's Encrypt · 2020-05-27 · CAA rechecking bug· 20203 million certs revoked in 24 hours

🔥 What broke

Let's Encrypt discovered their Boulder CA software had a bug in CAA (Certificate Authority Authorization) rechecking — it wasn't re-verifying CAA records within the required 8-hour window before signing. Per baseline requirements, they had to revoke every affected cert within 5 days.

🧯 The fix

Emergency-notified every affected subscriber, extended the deadline by request from browser vendors, and re-issued clean certs. Anyone with fully-automated renewal (via certbot) barely noticed. Anyone doing manual renewal had a very bad week.

🎓 Lesson to steal

Baseline Requirements (from the CA/Browser Forum) can force emergency revocation at any time. Design for revocation: your automation must handle "renew now" without a human, not just "renew before day 60."

Post-mortem

War story Fastly · 2021-06-08· 20211-hour global outage · Reddit, NYT, GOV.UK, Amazon down

🔥 What broke

Not TLS itself, but adjacent — a config change at Fastly triggered a latent bug that took most of their edge offline. During the incident, cascading TLS handshake failures showed up in every downstream observability tool. The lesson: when a CDN goes down, everyone else's TLS metrics also go red because handshakes to the origin timeout.

🧯 The fix

Fastly reverted the config in 49 minutes. Downstream teams learned that "TLS handshake timeout" often means "the thing terminating TLS for you is having a bad day," not "our cert is broken."

🎓 Lesson to steal

Correlate before you rotate. A spike in TLS errors that coincides with a CDN outage is a CDN problem, not your cert. Don't emergency-renew during a CDN incident — you'll just create a second incident.

Post-mortem

The rotation playbook — what mature teams actually do

Where this shows up in the rest of the plan

TLS + PKI touch every networked system you'll ever build

S083 · OWASP Top 10

TLS misconfiguration is A02 in the current list. Weak ciphers, missing HSTS, expired certs.

S070 · Load balancing

L7 LBs terminate TLS. Cert lives on the LB, backend gets plain HTTP over private network.

S057 · gRPC + protobuf

Uses mTLS heavily for service-to-service auth. Same PKI concepts, different transport.

S077 · Observability

Cert expiry monitoring belongs in your dashboards next to latency and error rate.

S105 · Kubernetes

cert-manager is the standard way to issue Let's Encrypt or internal CA certs to any Ingress.

S123 · System design · payment systems

PCI-DSS mandates TLS 1.2+ for cardholder data in transit. You'll design for mTLS + HSM key storage.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal

★ = stretch question

Explain-out-loud test

If you can't teach these three without notes, redo the session:

What are the four flights of a TLS 1.3 handshake? (name each in order)
How does your browser decide to trust a stranger's certificate? (walk the chain)
What's the difference between a cert being expired vs revoked vs self-signed? (one sentence each)

What comes next

🔒SecurityNext →

S083 · OWASP Top 10 & Threat Modelling

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.