Statistics & A/B Testing
A deep-dive on Statistics & A/B Testing — part of a 36-topic evergreen learning series.
Why this session matters
Part of a 36-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.
Session 04 · ML track 📊
Every AI/ML engineer needs to reason about uncertainty. This session covers the statistical foundation: descriptive stats, distributions (normal, binomial, Poisson), hypothesis testing (t-test, chi-square, Mann-Whitney), confidence intervals, and — critically — designing and analyzing A/B tests including power analysis, sample size, MDE, and CUPED variance reduction.
Pre-read (30 min before session)
Watch 1–2 of these before the deep-dive:
- StatQuest Statistics Playlist — Josh Starmer
- A/B Testing Deep Dive — Emma Ding
- Hypothesis Testing — 3Blue1Brown Bayes
Then skim chapter 1 of Trustworthy Online Controlled Experiments (Kohavi et al).
Deep-dive (90 min)
1. Probability + distributions (25 min)
Random variables, expectation, variance, covariance. Common distributions and when each shows up: Normal (CLT), Binomial (conversion), Poisson (arrivals), Exponential (time between events), Beta (Bayesian priors). Central Limit Theorem — the reason we can do any parametric inference.
2. Hypothesis testing (25 min)
Null vs alternative. p-values (what they actually mean vs the popular misinterpretation). Type I vs Type II errors. Two-sample t-test, Welch's t-test, Mann-Whitney U (non-parametric), chi-square (categorical), Fisher's exact (small counts). Multiple testing (Bonferroni, BH-FDR).
3. A/B testing at scale (30 min)
Designing an experiment: primary metric, guardrail metrics, MDE, power analysis (target 80% power at α=0.05). Sample size formulas. Sequential testing traps (peeking inflates false positives). CUPED for variance reduction (30-50% smaller samples). Stratified sampling and heterogeneous treatment effects.
4. Analysis + gotchas (10 min)
SRM (sample ratio mismatch) as the first check. Novelty and primacy effects. Interference between experiments. Simpson's paradox in slicing. When to use bootstrap CIs.
Reading list
- Trustworthy Online Controlled Experiments — Kohavi, Tang, Xu
- Statistical Rethinking — Richard McElreath (Bayesian counterpoint)
- Evan Miller's A/B testing calculators — https://www.evanmiller.org/ab-testing/
Research links
Hands-on drill
You launch a new checkout flow to 5% of users. After 3 days, conversion rate is 4.8% (new) vs 4.5% (control), n=20k per arm. Compute the z-test statistic + p-value. Would you ship? Now — you check daily and see it swing from p=0.03 → 0.06 → 0.02 → 0.05. Explain the peeking problem.
Post-session checklist
- Can you explain what a p-value actually means in 60 seconds?
- Can you explain why sequential peeking inflates false positives in 60 seconds?
- Can you explain how CUPED reduces variance in 60 seconds?
- Did you complete the hands-on drill above?
- Did you write 3 flashcards for tomorrow's recall?
- What's the one thing you'd want to revisit in the next revision session?
What's next
Session 05 continues the series. See the hub page for the full sequence and revision pattern.