S022 · Statistics — CLT, Hypothesis Testing, Confidence Intervals

Module M02: Math Foundations · Session 22 of 130 · Track: Math 📊

What you'll be able to do after this session

• Explain Statistics to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S021 — Probability — Random Variables, Distributions, Expectation

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · But what is the Central Limit Theorem, really? — 3Blue1Brown — the visual explanation you'll never forget.
• 🎥 Deep dive (30–60 min) · Hypothesis Testing — full walkthrough (StatQuest) — p-values, nulls, alternatives, all of it.
• 🎥 Hands-on demo (10–20 min) · Confidence Intervals explained in code (StatQuest) — build a CI in Python from scratch.
• 📖 Canonical article · Wikipedia — Central Limit Theorem — the survey.
• 📖 Book chapter / tutorial · NIST Engineering Statistics — Hypothesis testing — rigorous but readable reference.
• 📖 Engineering blog · Netflix — Interpreting A/B test results — CLT and hypothesis testing in production.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How to make claims from noisy data.

Suppose you want to know the average height of every adult in India. Measuring all 900 million people is absurd. So you sample 1000, take the average, and ... how do you know that 1000-person average is close to the true 900-million-person average? That's the entire game of statistics: making trustworthy statements about a population from a sample.

The Central Limit Theorem (CLT) is the magic trick that makes it work. It says: no matter how weird the underlying population looks (skewed, bimodal, chaotic), if you take enough samples and average them, the distribution of those averages is bell-shaped (Normal). This means we can attach a confidence interval like "the true mean is 170 cm ± 2 cm, 95% of the time" without knowing anything about the population's true shape.

Hypothesis testing is the courtroom version: you assume nothing interesting is happening (the "null hypothesis"), then ask "how surprising would my observed data be if the null were true?" That surprise level is the p-value. If it's very surprising (typically p < 0.05), you reject the null and declare something is going on.

One-sentence gotcha to carry forever: A 95% confidence interval does not mean "there's a 95% chance the true mean is in this interval" — the true mean is fixed; the interval is what's random. It means "if I repeated this whole experiment many times, 95% of the intervals I'd build would contain the true mean."

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Let's watch the CLT actually happen. Take a wildly non-normal population: exponential (looks like a slide, very skewed).

Worked example — A/B test decision:

You run version A and version B of a landing page. After 1000 visitors in each bucket:

• A: 100 conversions (10.0% conversion rate)
• B: 130 conversions (13.0% conversion rate)

Is B really better, or did we get lucky? Set up hypothesis test:

• Null (H₀): the two versions have the same true conversion rate.
• Alternative (H₁): the rates differ.

Under H₀, the difference in observed rates would be roughly Normal (thanks to CLT) with mean 0 and standard error SE ≈ sqrt(p̂(1-p̂)(1/nA + 1/nB)) where p̂ = combined rate = 230/2000 = 0.115.

SE ≈ sqrt(0.115 · 0.885 · (1/1000 + 1/1000)) ≈ 0.0142

Observed difference: 0.03. Z-score = 0.03 / 0.0142 ≈ 2.11. Two-sided p-value ≈ 0.035. Under the usual 0.05 threshold, you'd reject the null and ship B.

Confidence interval for the true difference: 0.03 ± 1.96 · 0.0142 = [0.002, 0.058]. The interval is entirely positive — supports the reject decision.

Rule of thumb table for interpreting p-values:

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# clt_and_ab_test.py — python3 clt_and_ab_test.py
import random, math, statistics
 
random.seed(7)
 
# ---- 1. Watch the CLT emerge from an exponential population ----
def exp_sample():
    # simulate exponential(rate=1) via inverse CDF
    return -math.log(1 - random.random())
 
n = 30                    # sample size
num_samples = 5000
means = []
for _ in range(num_samples):
    sample = [exp_sample() for _ in range(n)]
    means.append(sum(sample) / n)
 
mu = statistics.mean(means)
sigma = statistics.stdev(means)
print(f"CLT check: distribution of sample means")
print(f"  mean of means = {mu:.4f}   (population mean = 1.0)")
print(f"  stdev of means = {sigma:.4f}  (theory: 1/sqrt(30) = {1/math.sqrt(30):.4f})")
 
# ---- 2. Two-proportion A/B test ----
def two_prop_ztest(succ_a, n_a, succ_b, n_b):
    p_a = succ_a / n_a
    p_b = succ_b / n_b
    p_pool = (succ_a + succ_b) / (n_a + n_b)
    se = math.sqrt(p_pool * (1 - p_pool) * (1/n_a + 1/n_b))
    z = (p_b - p_a) / se
    p_two_sided = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
    return p_a, p_b, z, p_two_sided
 
p_a, p_b, z, pval = two_prop_ztest(100, 1000, 130, 1000)
print(f"
A/B test: A={p_a:.3f}, B={p_b:.3f}, z={z:.3f}, p={pval:.4f}")
 
# ---- 3. Confidence interval for the difference (B - A) ----
diff = p_b - p_a
se_diff = math.sqrt(p_a*(1-p_a)/1000 + p_b*(1-p_b)/1000)
ci_low, ci_high = diff - 1.96*se_diff, diff + 1.96*se_diff
print(f"95% CI for (B - A): [{ci_low:.4f}, {ci_high:.4f}]")
verdict = "SHIP B" if ci_low > 0 else "INCONCLUSIVE / KEEP A"
print(f"Decision: {verdict}")

Checklist:

1. The empirical stdev of sample means is close to 1/sqrt(30) ≈ 0.183.
2. The distribution of means is bell-shaped even though the underlying is skewed.
3. The A/B test p-value is around 0.035 — below the 0.05 threshold.
4. The 95% CI for the difference lies entirely above 0.
5. Change n from 30 to 5 in step 1 — the CLT breaks down; the distribution is still visibly skewed.

Try this modification: re-run the A/B test with only 200 visitors per arm (20 conversions vs 26). Same 10% vs 13% rate, but much less data. Does the CI still exclude zero? What does that tell you about the power of small samples?

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

In real product teams, statistical mistakes are more common than statistical wins. Three recurring ones:

1. Peeking / early stopping. Analysts check the dashboard hourly and stop the test the first time p < 0.05. This inflates false-positive rate from a nominal 5% to somewhere between 25% and 50%. Netflix, Uber, and Booking.com all publish on this — the fix is sequential testing (mSPRT, group-sequential designs) built into internal platforms like Netflix's XP or Booking's ExP. Never eyeball a p-value against a fixed threshold on running data.

2. Multiple comparisons. You test 20 metrics in one experiment; even with no real effect, roughly 1 will cross p < 0.05 by pure chance. Solutions: pre-declare a single primary metric, apply Bonferroni or Benjamini-Hochberg corrections for exploratory metrics, and never cherry-pick "the metric that moved."

3. SUTVA violations (interference). The Stable Unit Treatment Value Assumption assumes user A's outcome doesn't depend on user B's bucket. This breaks in social networks (if I see my friend's new feature, I'm indirectly treated), marketplaces (if bidders in test bucket win auctions, control bucket loses them), and rate-limited systems. Facebook, Uber, and LinkedIn publish extensively on network-aware A/B testing — the practical fix is often cluster-randomized designs.

Latency / SLO angle: in SRE work, "average latency" is meaningless — the distribution is heavy-tailed and users experience the tail. Google's SRE book and Gil Tene's "How NOT to measure latency" talk both make the case: always report p50/p95/p99 (and often p999). CLT-based confidence intervals apply to those quantiles too, but the math is trickier — use bootstrap.

The universal habit: whenever a stakeholder says "the metric moved by X%," ask (a) what's the confidence interval around X, (b) did we pre-register this metric, and (c) how many metrics did we look at? Those three questions filter out ~80% of bogus wins.

(e) Quiz + exercise · 10 min

1. State the Central Limit Theorem in one sentence a non-technical PM could follow.
2. Why does the standard error of the mean shrink as 1/sqrt(n) and not 1/n?
3. A 95% confidence interval for conversion lift is [−0.5%, +2.1%]. Can you claim the feature helped?
4. What is the "peeking" problem in A/B testing, and how does it inflate false positives?
5. Why is average latency a misleading SLO for a heavy-tailed distribution?

Stretch (connects to S021 — Probability): Given a random variable X with mean μ and variance σ², derive why the sample mean of n i.i.d. draws has variance σ²/n. (Hint: use Var(aX + bY) = a²Var(X) + b²Var(Y) for independent X, Y.)

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Statistics? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S023): Arrays & Strings — Indexing, Slicing, Two-Pointer
• Previous (S021): Probability — Random Variables, Distributions, Expectation
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.