Statistics — CLT, Hypothesis Testing, Confidence Intervals

Review boundary and corrected contracts

A p-value conditions on the null and sampling model, not the probability the null is true. Confidence level describes repeated-interval coverage. CLT requires assumptions and no fixed n=30 cutoff; peeking changes fixed-horizon error rates. The numerical exercises use independent synthetic samples and optional NumPy/SciPy dependencies; they do not establish an effect in a real product experiment.

The examples below teach the stated input contracts; fragments and deliberately incorrect counterexamples are exercises, not production implementations. Complexity counts elementary operations unless a bit-cost or allocation caveat is stated. Practice times are suggestions, not measured performance or hiring guarantees.

Learning objective: Read a p-value, a confidence interval, and an A/B test dashboard without lying to yourself — and simulate all three from scratch in Python.

Why this session exists

Every A/B test result you'll ever ship, every ML model comparison you'll ever defend, every "is this metric change real?" Slack thread — they all rest on three ideas: the Central Limit Theorem (why averages of noisy stuff behave), hypothesis testing (how you compare a statistic with a null model), and confidence intervals (how you communicate uncertainty). Skip this session and you'll spend the next decade being technically wrong in high-stakes meetings — the kind of wrong that ships a bad checkout flow to 100M users.

You will be able to

• Explain the Central Limit Theorem in one sentence and demonstrate it with 20 lines of Python.

• State the null and alternative hypothesis for any A/B test you're asked to evaluate.

• Compute a p-value and a 95% confidence interval for a difference in means — and interpret both without the four classic mistakes.

• Spot p-hacking, peeking, and multiple-comparison abuse in a colleague's analysis before it ships.

• Know when to use a t-test, when to use a proportion z-test, and when to just bootstrap.

Prerequisites

• S021 · Probability — random variables, expectation, variance.
• S019 · Linear Algebra Basics — vector operations (used to summarise a sample).
• Comfort with numpy from S005–S009 Python foundations.

Pre-read · skim before the session (30 min total)

Further reading: An Introduction to Statistical Learning · Ch. 5 (Resampling)

Further reading: How Not To Run An A/B Test

Further reading: xkcd · Significant

(a) Intuition · 5 min

Statistics is how noisy measurements become confident claims

Real: You're a coffee roaster. You claim your new bean batch is stronger than the old one. You brew one cup of each, taste them — the new one feels stronger. Are you sure? Maybe you were more caffeinated when you tasted the new one. Maybe the water temperature drifted 2 °C. So you brew 100 cups of each, shuffle them, taste blind. Randomization and blinding reduce some sources of bias, and averaging reduces sampling variability; the observed difference still needs uncertainty analysis and does not prove a real effect. That's statistics: the mathematics of separating signal from noise when every single measurement is unreliable.

 
    You changed the checkout button from blue to green. Conversion went from 3.1% to 3.4% over one day. Real improvement or lucky Tuesday? A hypothesis test answers exactly that: how surprised should I be by this difference if the true conversion rate didn't change at all?
    A p-value measures tail probability under the specified null and sampling model. A 95% confidence procedure aims to cover the fixed true lift in 95% of repeated experiments under its assumptions; it does not assign a 95% probability to this particular interval. The CLT supports some large-sample approximations, not arbitrary distributions or every testing method.
  

The three big ideas, one sentence each

Three ideas that unlock all of applied statistics

The elementary CLT assumes iid observations with finite nonzero variance. The standardized mean approaches normality asymptotically; skewness and tail behavior affect how good a finite-sample approximation is.

• Hypothesis testing — specify a null and sampling model, choose a statistic and tail rule, then compute the probability of a statistic at least as extreme under that model. Reject according to a pre-specified error-control rule, not a posterior probability of truth.

• Confidence interval — a range that would contain the true value in 95% of hypothetical repeats of the experiment. NOT ‘a 95% chance the true value is in this range for this one experiment’. A Bayesian credible interval can have a posterior probability interpretation under its specified prior/model; it is a different construction.

The most-misquoted definition in all of science

A p-value is NOT "the probability the null hypothesis is true." It is "the probability of seeing data at least this extreme, assuming the null hypothesis is true." The direction of the conditional matters. Get this backwards in a PM meeting and the senior data scientist will politely correct you; get it backwards in a research paper and it gets retracted.

A quick history so you know why the world looks like this

A concrete modern milestone is the ASA's 7 March 2016 statement announcement, which quotes six principles warning against treating a threshold as a substitute for scientific reasoning. The lesson is not that significance testing disappeared: current SciPy still implements Student/Welch tests and several bootstrap interval methods. These tools answer different questions under different assumptions; computational resampling did not make inference assumption-free.

(b) From sample means to defensible test results · 15 min

The checkout-button question is not whether two observed rates differ—they almost always will—but whether the difference is large compared with ordinary sampling variation. To judge that, we first need to understand how an average varies across repeated samples.

The Central Limit Theorem in one picture

Read each branch as a fresh sample from the same population, not as another measurement inside one sample. Each branch contributes one mean to the distribution at the bottom; it is those means, rather than the raw observations, whose shape the CLT describes.

The raw population need not be bell-shaped. For independent, identically distributed observations with finite variance, the distribution of sample means becomes approximately normal as the sample size grows. The diagram's sample size of 30 is illustrative, not a universal cutoff: strongly skewed populations can need much larger samples. The narrowing spread of the means is what makes an error bar around an average useful.

The anatomy of a hypothesis test

A hypothesis test turns that sampling variation into a reference point: how unusual would the observed difference be if the button colour had no effect? The order below matters because the hypothesis and decision threshold must be chosen before the result is known.

State H₀ and H₁

Desc: Null: ‘the button colour has no effect on conversion’. Alternative: ‘the two conversion rates differ’ (two-sided) or ‘green > blue’ (one-sided). Pick one BEFORE looking at data.

Pick a significance level α

At a fixed pre-specified valid test, α bounds the rejection probability when the null and its model hold. It is not the chance that a particular shipped decision is wrong.

Compute the test statistic

Desc: Choose a statistic whose exact or approximate null distribution matches the design: t for suitable independent mean comparisons, z for sufficiently large independent proportion samples, or contingency/distribution tests for their stated inputs. Randomization unit, dependence and nuisance-parameter estimation matter.

Compute the p-value

Desc: The probability of seeing a statistic at least this extreme if H₀ were true.

Compare p to α

Desc: p < α → reject H₀ (‘the effect is statistically significant’). p ≥ α → fail to reject (‘we don't have enough evidence’). Note: this is NOT ‘there is no effect’.

Also report the effect size + CI

Desc: A statistically significant effect can be practically small. Report the estimate and uncertainty with units: a change from 3.1% to 3.4% is +0.3 percentage points (about +9.7% relative), not +0.3% relative lift.

When to use which test

Name: Two-sample t-test

Tagline: Compare means of two groups

• e.g. average session length A vs B

• Independent observations/groups; normal-population assumptions matter especially for small samples

• Welch accommodates unequal variances, but tiny skewed/heavy-tailed samples are not automatically safe

• scipy: ttest_ind(a, b, equal_var=False) for Welch; the default assumes equal population variances

Name: Two-proportion z-test

Tagline: Compare rates / conversions

• e.g. conversion rate control vs treatment

• Normal-approximation checks require adequate expected successes/failures in both groups; a threshold such as 10 is a heuristic, not a universal validity theorem

• Standard for A/B testing

• statsmodels: proportions_ztest

Name: Chi-square test

Tagline: Compare categorical distributions

• e.g. is device-type breakdown the same?

• Works for k > 2 groups

• Check independent counts and expected-cell counts; sparse tables may need an exact or simulation-based alternative rather than a chi-square approximation

• scipy: chi2_contingency

Name: Bootstrap

Tagline: Approximate uncertainty by resampling a suitable sampling unit

• Useful for many statistics, including means and suitably regular quantiles, but not universally consistent for every statistic

• Ordinary independent resampling assumes representative iid observations within each group; paired, clustered or time-series data need a matching scheme

• Small, degenerate or poorly representative samples can give misleading intervals; more resamples reduce Monte Carlo error, not sampling bias

• Choose and validate an interval method. The simple percentile interval below is a teaching example, not an automatic remedy for skewness or low sample size

The four interpretations that will get you fired

What p and CI do NOT mean

Name: p = 0.03 does NOT mean ‘97% chance the effect is real’

Desc: Under the null and its sampling model, a statistic at least this extreme has probability 0.03. This is not a probability assigned to the effect being real.

Name: p = 0.20 does NOT mean ‘no effect’

Desc: It means ‘we don't have enough evidence to reject H₀’. Absence of evidence isn't evidence of absence — you might just need more samples.

Name: ‘95% CI = [1.2, 4.8]’ does NOT mean ‘95% chance the true value is in [1.2, 4.8]’

Desc: The fixed parameter either is or is not in the realized interval. ‘95%’ refers to repeated-procedure coverage under the model; approximate methods aim at that nominal level and can under- or over-cover in finite samples.

Name: ‘p < 0.05, so we ship’ ignores effect size

Desc: Large samples can make small effects detectable, depending on baseline rate and variance. A small effect is neither guaranteed significant nor automatically worthless; assess its interval, units and expected value against costs and guardrails.

A useful reporting checklist

When someone reports a p-value with no confidence interval, no effect size, and no sample size, treat their claim as unverified. When they report all three, you can actually reason about whether to care.

Even a correctly computed result can be misread or selected from too many attempts. The next examples separate those failures: reversing the p-value's conditional, then repeatedly testing until a reassuring answer appears.

Myth: "p < 0.05 means there's a 95% chance the effect is real, and p = 0.20 means there's probably no effect. The p-value tells me how likely my hypothesis is."

Truth: A p-value is P(data at least this extreme | the null is true). It is a statement about the data given a hypothesis, not about the hypothesis given the data. It cannot tell you the probability your effect is real — that requires a prior, which the p-value never uses. And a large p-value means "insufficient evidence", never "no effect".

Why: Because the conditional is reversed in exactly the direction that makes the convenient interpretation feel natural, and because the threshold ritual — significant or not — encourages treating 0.05 as a boundary between true and false. The wrong reading is not justified even by high power. The fraction of discoveries that are false depends on the base rate of true effects, power, analysis selection and model validity. When true effects are rare, false positives can make up a large fraction of selected significant findings, as in a low-prevalence diagnostic test. And an underpowered test that finds nothing has told you almost nothing — with low power, most real effects would have been missed, so "not significant" is not evidence of absence.

Test: Run twenty tests on data with no effect at all, and watch significance appear:import numpy as np from scipy import stats rng = np.random.default_rng(0)

hits = 0 for _ in range(20): # 20 'metrics', all pure noise a, b = rng.normal(size=100), rng.normal(size=100) if stats.ttest_ind(a, b).pvalue < 0.05: hits += 1 print(hits) # ~1 significant result, from nothing. That is what 5% means.

Question: Why does peeking at an A/B test and stopping when it hits significance inflate the false positive rate far above 5%? Each individual test is still a valid 5% test — derive why the sequence is not.

Premise: Under a null model of independent zero-mean treatment-minus-control increments, their partial sum is a zero-drift random walk. The mean difference is that sum divided by sample count; it is not itself an unscaled random walk.

Because: with no true effect, each new observation nudges the estimate in a random direction

Premise: A usual fixed-horizon threshold for the mean difference scales approximately as 1/√n. Later tests share earlier observations, so rejection events at different sample sizes are correlated.

Because: the same accumulated sample contributes to multiple test statistics

Premise: A fixed-horizon test controls the probability of being outside the boundary at one predetermined moment. That is the guarantee it makes and the only one.

Because: the 5% was computed for a single evaluation of a single statistic

Premise: Stopping at the first crossing instead evaluates the boundary at every point along the walk, so the relevant probability becomes "does the walk ever cross?" rather than "is it outside at time T?"

Because: an optional stopping rule turns one question into a maximum over many correlated questions

Premise: If Eₖ is rejection at look k, stopping on any rejection uses the event E₁ ∪ ... ∪ Eₘ. Its probability is at least the largest individual rejection probability and at most the sum of them. It can be larger than α; the amount depends on the model and look schedule. Correlation prevents using the independent-test formula blindly.

Because: event inclusion gives the lower bound and the union bound gives the upper bound; neither requires independence

Conclusion: A collection of individually valid 5% tests is not automatically a 5% stopping rule. No universal finite-horizon percentage or unconditional eventual-crossing guarantee follows. For m planned looks, allocating α/m to each gives familywise error at most α by the union bound (often conservatively); specialized group-sequential or always-valid methods can use the dependence structure more efficiently. Alternatively, keep a pre-specified fixed horizon and do not select a significance claim using interim results. Multiple metrics need their own multiplicity plan; the same union-event logic applies, though the appropriate procedure can differ.

Name: Signal, noise, and how many times you looked

Picture: An observed difference includes sampling variation and may also include bias. A p-value is a tail probability for a chosen statistic under a specified null model, not the probability that noise caused the data. Report the estimate, confidence interval and test result together, with their assumptions and units: each answers a different question. The effect size: how big, and does anyone care? The confidence interval: what range is consistent with this data? The p-value: how extreme is the chosen statistic under the specified null model? Reporting only the third — the one that answers the least useful question — is how a statistically significant 0.1% improvement ends up justifying a quarter of engineering work.

• Compute the required sample size before running the test, from the minimum effect worth detecting. If you can't name that minimum, you aren't ready to run the experiment.

• Report the confidence interval, not just the p-value. An interval such as "+2.1 percentage points [−0.3, +4.5]" conveys magnitude and uncertainty. A confidence interval and hypothesis test have equivalent rejection decisions only when constructed as corresponding procedures.

• Multiple comparisons need correction, whether across metrics, segments, variants, or time. For twenty independent valid null tests with rejection probability exactly 0.05, the chance of at least one rejection is 1 − 0.95²⁰ ≈ 64.2%. Interim looks at accumulating data are not independent.

• Statistical significance is not practical significance. Under a suitable consistent test, increasing sample size improves detection of a fixed nonzero effect. Model validity, bias, guardrails and decision costs still matter even after statistical significance.

Trigger: Fire this model the moment you see: a test stopped early because it "hit significance" · a dashboard of twenty metrics with one starred · "no significant difference" used as proof of equivalence · a segment analysis discovered after the fact · a p-value reported with no effect size · any experiment with no pre-registered sample size.

Decision: You need to decide on a feature. Fixed-horizon A/B test, sequential test with early stopping, or Bayesian decision analysis?

Name: Fixed-horizon frequentist test

Gain: rigorous and universally understood, with a clearly controlled error rate; the sample-size calculation forces an explicit conversation about what effect size actually matters

Cost: significance-based stopping or selecting the report using interim results breaks the fixed-horizon calibration. Merely viewing a dashboard does not alter the mathematics if the pre-specified sampling and analysis rule remains unchanged; safety monitoring still needs an explicit policy

Pick: the decision is high-stakes and one-shot, traffic is sufficient to hit the required n in reasonable time, and the result will be scrutinised by others

Name: Sequential / always-valid testing

Gain: a properly designed sequential procedure permits its stated interim decisions while controlling the specified error criterion. It may stop earlier under some effects; neither shorter duration nor immediate detection of harm is guaranteed

Cost: repeated-look control changes thresholds and the power/sample-size tradeoff. Compare the actual design, horizon and stopping rules; not every sequential method has the same worst-case cost

Pick: you run many tests and average duration is the bottleneck, or a bad variant causes real user harm that you cannot leave running to a fixed horizon

Name: Bayesian decision analysis

Gain: gives you the quantity you actually wanted — P(B is better than A) and the expected loss of choosing wrongly — and incorporates prior knowledge from previous experiments instead of discarding it

Cost: the prior is a real choice that must be defended, and a badly chosen one biases the conclusion; error-rate guarantees are not the frequentist ones stakeholders may expect; requires more statistical literacy in the room

Pick: you have genuine prior information from comparable past experiments, or the decision needs an expected-cost framing rather than a binary significant/not verdict

Verdict: For most product experimentation the binding constraint is not the statistical framework — it is discipline. Pre-register the metric, the sample size, and the decision rule before the test starts, and most of the classic failures disappear regardless of which method you chose.Where the framework does matter is organisational: a fixed-horizon test that everyone peeks at is worse than a sequential test designed for peeking, because at least the second one is honest about what people are actually going to do. Pick the method that matches the behaviour you will really get, not the behaviour you wish you had.

(c) Hands-on · 25 min

Run this file top to bottom with NumPy and SciPy available. It simulates exponential sample means, an independent-groups Welch test, a percentile bootstrap, independent null metrics and repeated looks under a normal null. The seed makes the entire sequence reproducible: restarting the file is not a new experiment. The bootstrap expects nonempty finite one-dimensional iid samples from independent groups, a positive resample count and 0 < alpha < 1. The peeking demonstration uses positive counts with n_final divisible by n_peeks and at least two observations per look.

"""stats_playground.py — CLT, t-test, bootstrap, p-hacking demo.
 
Run with:  uv run --with numpy --with scipy stats_playground.py
Deps:     numpy, scipy, matplotlib (only for the CLT figure — comment out if headless)
"""
from __future__ import annotations
import numpy as np
from scipy import stats
 
rng = np.random.default_rng(42)
 
 
# ------------------------------------------------------------------
# 1) Central Limit Theorem — averages of an ugly distribution
# ------------------------------------------------------------------
def demo_clt(n_per_sample: int = 30, n_samples: int = 10_000) -> None:
    """Sample from an exponential (very skewed) and take the mean 10,000 times."""
    means = rng.exponential(scale=2.0, size=(n_samples, n_per_sample)).mean(axis=1)
    print(f"[CLT] population mean=2.0  sample-of-means mean={means.mean():.3f}")
    print(f"[CLT] theoretical SE=σ/√n={2.0/np.sqrt(n_per_sample):.3f}  observed={means.std():.3f}")
    # If you have matplotlib, uncomment to see the bell curve:
    # import matplotlib.pyplot as plt
    # plt.hist(means, bins=60); plt.title("Distribution of sample means"); plt.show()
 
 
# ------------------------------------------------------------------
# 2) Two-sample t-test — is treatment better than control?
# ------------------------------------------------------------------
def demo_ttest() -> None:
    control = rng.normal(loc=100, scale=15, size=200)      # e.g. session length in seconds
    treatment = rng.normal(loc=104, scale=15, size=200)    # small true lift of 4s
    t, p = stats.ttest_ind(treatment, control, equal_var=False)   # Welch's t-test
    diff = treatment.mean() - control.mean()
    print(f"[t-test] observed diff={diff:+.2f}s  t={t:.2f}  p={p:.4f}  -> {'REJECT H0' if p<0.05 else 'fail to reject'}")
 
 
# ------------------------------------------------------------------
# 3) Bootstrap 95% CI for the difference in means
# ------------------------------------------------------------------
def bootstrap_ci(a: np.ndarray, b: np.ndarray, n_boot: int = 10_000, alpha: float = 0.05) -> tuple[float, float, float]:
    diffs = np.empty(n_boot)
    n_a, n_b = len(a), len(b)
    for i in range(n_boot):
        ra = rng.choice(a, size=n_a, replace=True)
        rb = rng.choice(b, size=n_b, replace=True)
        diffs[i] = rb.mean() - ra.mean()
    lo, hi = np.quantile(diffs, [alpha/2, 1 - alpha/2])
    return diffs.mean(), lo, hi
 
 
def demo_bootstrap() -> None:
    control = rng.normal(loc=100, scale=15, size=200)
    treatment = rng.normal(loc=104, scale=15, size=200)
    est, lo, hi = bootstrap_ci(control, treatment)
    print(f"[bootstrap] est diff={est:+.2f}s  95% CI=[{lo:+.2f}, {hi:+.2f}]  contains 0? {lo <= 0 <= hi}")
 
 
# ------------------------------------------------------------------
# 4) p-hacking demo — test 20 metrics with no real effect
# ------------------------------------------------------------------
def demo_phacking(n_metrics: int = 20, n_per_group: int = 100) -> None:
    """Run 20 A/B tests where the truth is that NOTHING changed. Count false positives."""
    false_positives = 0
    for i in range(n_metrics):
        a = rng.normal(size=n_per_group)
        b = rng.normal(size=n_per_group)     # same distribution — truth: no effect
        _, p = stats.ttest_ind(a, b)
        if p < 0.05:
            false_positives += 1
            print(f"  metric #{i:02d}: p={p:.3f}  <-- statistically significant false positive")
    print(f"[p-hacking] {false_positives}/{n_metrics} tests came out ‘significant’ despite zero true effect.")
    print("            Expected number of false positives at α=0.05: n·α =", n_metrics * 0.05)
 
 
# ------------------------------------------------------------------
# 5) Peeking — early-stopping inflates false positives
# ------------------------------------------------------------------
def demo_peeking(n_final: int = 2000, n_peeks: int = 20, n_experiments: int = 1000) -> None:
    hits = 0
    for _ in range(n_experiments):
        a = rng.normal(size=n_final)
        b = rng.normal(size=n_final)          # again, truth: no effect
        step = n_final // n_peeks
        for k in range(step, n_final + 1, step):
            _, p = stats.ttest_ind(a[:k], b[:k])
            if p < 0.05:
                hits += 1
                break                          # ‘we won! ship it!’
    rate = hits / n_experiments
    print(f"[peeking] any-rejection rate over {n_peeks} looks: {rate:.1%}  (single fixed-look reference: 5%)")
 
 
if __name__ == "__main__":
    demo_clt()
    demo_ttest()
    demo_bootstrap()
    demo_phacking()
    demo_peeking()

Anatomy of the script

What each block teaches

Name: demo_clt

Desc: Simulates the distribution of sample means, which is only approximately normal here. For iid finite-variance observations, SE = σ/√n is exact; the normal approximation is asymptotic. Exact normal-theory t results and asymptotic robustness are different justifications.

Name: demo_ttest

Desc: Welch's t-test — the safer default when the two groups may have different variances. Returns t and p; reject H₀ if p < 0.05.

Name: bootstrap_ci

Desc: Resamples independently within each group and takes the 2.5th/97.5th percentiles of mean differences. This percentile method relies on a suitable iid sampling model and approximation quality. The retained helper returns the average bootstrap difference as est; it approximates the observed b.mean() - a.mean() with Monte Carlo error, rather than returning that point estimate exactly. More resamples cannot fix an unrepresentative sample. For a single Bernoulli observation per group, every resample is identical and the zero-width interval can miss the population difference entirely.

Name: demo_phacking

Desc: 20 tests with no real effect → we expect ≈1 false positive by chance (20·0.05). This is the ‘green jelly beans’ problem.

Name: demo_peeking

Desc: Runs up to 20 correlated tests per simulated null experiment and counts any rejection. Compare the observed rate with a single fixed-look control; it is specific to this model, seed and look schedule, not a universal 15–25% rule.

What to observe when you run it

ClassName: ml-4 list-decimal space-y-1

CLT observed SE ≈ theoretical σ/√n. That equation isn't magic — it's a fact you can measure. The Welch p-value and bootstrap endpoints depend on the generated samples and call order. Compare each result with its rule; a simulated true 4-second lift can still be missed. The two demonstrations generate different samples and do not form a matched test/CI pair. Across repeated independent sets of 20 exact null tests, the expected rejection count is 1, not a guarantee for a single run. The proportion of experiment sets with any rejection is a different quantity, about 64.2% under independence. The peeking demo estimates the any-rejection rate for this schedule. A Monte Carlo proportion based on M experiments has approximate standard error sqrt(rate·(1−rate)/M); report simulation uncertainty rather than a cross-company or universal rate claim.

Goal: Feel what ‘underpowered’ means — see how sample size changes p

Hint: Compare rejection frequencies over many fresh simulated samples at each size, keeping the true effect and variance fixed. Individual p-values need not rise monotonically as sample size falls. Power is a probability over repeated samples, not a diagnosis from one large p-value.

Change the sample sizes in demo_ttest:

control = rng.normal(loc=100, scale=15, size=25)
treatment = rng.normal(loc=104, scale=15, size=25)

Call demo_ttest() repeatedly in the same interpreter after changing its sizes, so the RNG advances; do not restart with seed 42 for every repetition. For a more stable comparison, simulate hundreds or thousands of trials at n=25 and n=200 and count p < 0.05. A fixed seed can make that whole experiment reproducible without making its trials identical.

(d) Production reality · 15 min

Where this shows up in the rest of the plan

Root: Statistics feeds every ‘is this real?’ moment

Label: S087 · Model Evaluation

Desc: Held-out losses can support uncertainty estimates under an appropriate sampling model. Cross-validation folds overlap in training data, so their errors are not simply independent observations.

Label: S089 · Bias & Fairness

Desc: Fairness analysis can compare rates and uncertainty across groups, but also requires metric choices, representativeness and domain judgment; it is not reducible to a p-value.

Label: S097 · A/B Testing for ML

Desc: How to ship models with the same rigour as UI changes.

Label: S110 · Distributed Systems Consistency

Desc: SLO ‘p99 latency <200ms’ is a statistical claim about a distribution.

Label: S123 · Metric Design

Desc: Why picking the RIGHT metric matters more than the p-value you compute on it.

Label: S128 · Postmortems & Incident Analysis

Desc: Historical variation can inform anomaly analysis; time dependence, changing systems and causal investigation require more than a generic hypothesis test.

(e) Recall + stretch · 10 min

Self-check (answer follows each question).

Recall — click to reveal

Q: State the Central Limit Theorem in one sentence.

A: For iid observations with mean μ and finite nonzero variance σ², sqrt(n)(sample mean − μ)/σ converges in distribution to standard normal; the sample mean has standard error σ/√n. Finite-sample normality is approximate, with no universal n=30 threshold.

Q: What exactly does p = 0.03 mean?

A: If the null hypothesis were true (no effect), we'd see data at least this extreme in 3% of hypothetical repeats of the experiment. It is not ‘a 97% chance the effect is real’.

Q: You run 20 A/B tests with no real effect at α = 0.05. How many false positives should you expect?

A: If each null test rejects with probability 0.05, the expected count is 1 (linearity of expectation does not need independence). With independent tests the probability of at least one is about 64.2%, not certainty. Bonferroni controls familywise error using α/20; Benjamini–Hochberg targets false discovery rate under its dependence assumptions, a different guarantee.

Q: Why does peeking at A/B test results inflate the false-positive rate?

A: If stopping or reporting depends on any interim rejection, the error event is the union of the look-specific events. Its probability depends on their dependence and schedule and is not generally 5% or a universal 15–25%. Use a fixed pre-specified analysis horizon, a planned multiplicity adjustment, or a sequential procedure with the required error guarantee.

Q: When would you prefer a bootstrap CI over a t-test CI?

A: Consider bootstrap intervals when a suitable resampling scheme can approximate the statistic's sampling distribution and an analytic interval is inconvenient. It can help for medians or quantiles in sufficiently informative samples, but heavy tails, extreme quantiles, dependence and tiny samples can defeat the ordinary percentile bootstrap. Check representative sampling, sampling unit, statistic regularity and interval method rather than treating resampling as assumption-free.

Q: What is a Type I vs Type II error?

A: Type I = false positive = rejecting H₀ when it's true (‘we shipped a change that did nothing’). Type II = false negative = failing to reject H₀ when it's false (‘we missed a real improvement’). α controls Type I; power (1 − β) controls Type II.

Q: You get p = 0.20 from an A/B test with n = 100 per group. Can you conclude ‘no effect’?

A: No. ‘Failed to reject’ ≠ ‘no effect’. With n = 100 you might simply be underpowered — you could easily miss a real 2% lift. Report the interval and the pre-specified analysis result. A corresponding two-sided 95% interval contains zero at p=0.20; an unrelated interval need not. Do not extend a fixed-horizon test solely until significance—plan a new experiment or use a valid sequential design.

Q: In one sentence, what is a confidence interval?

A: A range produced by a procedure with a specified repeated-sampling coverage level under its assumptions. For a nominal 95% procedure, the target is 95% coverage; approximate methods may not attain it exactly. This is not a probability assigned to the fixed parameter after observing one interval.

Q: Why is Welch's t-test usually a safer default than Student's t-test?

A: Student's t assumes both groups have the same variance. Welch's t doesn't — it estimates each group's variance separately. When variances are actually equal, Welch's costs you very little; when variances and group sizes differ, the pooled test can be miscalibrated (liberal or conservative). Welch still requires independent observations and an adequate distributional approximation.

Q: Stretch — you observe a p < 0.001 lift of +0.02% conversion on 50 million users. Do you ship it?

Stretch: true

A: A small p-value is not proof that the effect is real. First check randomization, measurement, the analysis/stopping plan, multiplicity and guardrails. Clarify whether +0.02% means relative percent or percentage points; then use the effect estimate and interval to assess expected incremental value against costs and harms. Declare a minimum practically important effect and plan power before the experiment; an MDE is a design input, not a universal platform feature or automatic shipping rule.

Explain-out-loud test

Teach these three, no notes, in under a minute each:

1. Under which assumptions does the CLT justify a normal approximation for a sample mean?
2. What does a p-value mean — and what does it definitely NOT mean?
3. Give one concrete way peeking at an A/B test breaks it, and one way to fix that.

What comes next

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.

Corrective source notes

• ASA press release, 7 March 2016 — quotes the statement's six principles; significance does not establish truth or practical importance.
• NIST: confidence limits for the mean — confidence describes the interval procedure, not a posterior probability for a fixed interval.
• SciPy bootstrap — percentile construction, alternative methods and degenerate-data limits.
• SciPy independent t-test — Welch requires equal_var=False; independent samples remain part of the contract.

These citations support the named claims, not blanket certification of the lesson. Worked proofs are derived in the text; execution scope and finite-test limits are recorded separately.