6-month learning plan · · session 21 / 13021 / 130

mathbeginner 55m read

S021 · Probability — Random Variables, Distributions, Expectation

Uncertainty, quantified: random variables, the distributions you'll actually meet (Bernoulli, binomial, normal, exponential, Poisson), expectation, variance, and Bayes' rule.

📐MathM02 · Math Foundations· Session 021 of 130 90 min

🎯 Model an uncertain outcome as a random variable, pick the right distribution, compute expectation and variance, and apply Bayes' rule to a real diagnostic-test problem.

Why this session exists

Every modern engineering discipline is drowning in uncertainty. A/B tests, ML model outputs, load forecasts, tail latencies, credit-card fraud detection, LLM sampling — all of them are probability calculations wearing engineering clothes. The difference between "I have a feeling this is a lot" and "the p95 is 340 ms with 95% CI [335, 345]" is fluency in random variables, distributions, and Bayes' rule. This session gets you the vocabulary, and — crucially — the intuition to spot when someone (or you) is using it wrongly.

You will be able to

Define a random variable, its distribution, and its PMF/PDF/CDF — and translate between them.
Recognise Bernoulli / binomial / normal / exponential / Poisson in the wild and pick the right one.
Compute expectation E[X] and variance Var(X) — both from formula and by simulation.
State and apply Bayes' rule to update beliefs given evidence (the medical-test canonical example).
Explain the Central Limit Theorem in one sentence and know why it underlies everything from confidence intervals to A/B tests.

Prerequisites

S016 · Discrete math — counting/combinatorics underlies discrete probability.
S017 · Vectors — you'll compute expected values as dot products with probabilities.
High-school algebra, integration is optional (we'll use NumPy for anything continuous).

Pre-read · skim before the session (35 min total)

💡 Intuition 15 min

Bayes theorem, the geometry of changing beliefs

3Blue1Brown

Bayes' rule via pictures — after this, ‘prior × likelihood’ stops sounding intimidating.

🔬 Deep dive 15 min

Central Limit Theorem, clearly explained

StatQuest

The clearest explanation of CLT on YouTube. Watch before you use any t-test, confidence interval, or A/B test.

🛠️ Hands-on 12 min

The Main Ideas behind Probability Distributions

StatQuest with Josh Starmer

Bernoulli/binomial/Poisson in one video with real examples.

📖 Reading 20 min

Seeing Theory — a visual introduction to probability and statistics

Brown University

Interactive JavaScript visualisations of every concept in this session. Play with each chapter.

📖 Reading 15 min

Arbital · Bayes' rule guide

Arbital

The best written explanation of Bayes' rule for engineers. Walks through the reasoning slowly.

📖 Reading 10 min

numpy.random — the modern Generator API

numpy.org

Every distribution you need, plus fast sampling. `rng = np.random.default_rng()` is the modern idiom.

(a) Intuition · 5 min

A random variable is a labelled roulette wheel

🌍 Real world

You spin a roulette wheel. The wheel has some pockets — some hold big prizes, some small, some nothing. The random variable is the amount you'd win on any given spin. It's not one number; it's a distribution — a description of every possible value and how likely each is.

Two questions matter about any random variable: on average, what do I win? (expectation) And how much does the outcome swing around that average? (variance). Every probability calculation you'll ever do reduces to some version of those two questions.

💻 Code world

In code, a random variable is a function that returns a random number when called. In NumPy: rng.binomial(n=10, p=0.3). Its distribution is the theoretical rule for how those returned numbers are shaped. Its expectation and variance are numbers you can compute either from a formula (E[Bin(n,p)] = np) or by calling it a million times and averaging.

The whole subject exists so you can reason about outputs of things you haven't observed yet. It's the language of prediction.

The five distributions you'll see 100× a year

Meet the family

Bernoulli(p) — one flip of a biased coin. Outcome 0 or 1. Foundation of every yes/no event.
Binomial(n, p) — n independent coin flips; count the heads. Number of clicks in n ad impressions, defects in n parts.
Poisson(λ) — number of independent rare events in a fixed interval. Arrivals per second, errors per hour.
Normal(μ, σ) — the bell curve. Result of many small independent additive effects. Heights, IQ, sum of many things.
Exponential(λ) — waiting time between Poisson events. Time until next request, time until next failure.

A quick history

1654
Pascal & Fermat · gambling correspondence
Two French mathematicians exchange letters about dice games — accidentally invent probability.
1763
Bayes' theorem · posthumous
Reverend Thomas Bayes' work is published two years after his death. Ignored for 150 years, now central to everything.
1810
Laplace · Central Limit Theorem
Pierre-Simon Laplace proves the sum of many random variables tends to normal. The bell curve conquers the world.
1933
Kolmogorov · axioms of probability
Andrey Kolmogorov gives probability its rigorous foundation as measure theory. All modern textbooks descend from this.
1990s
MCMC + Bayesian revival
Cheap computing makes Bayesian inference practical. PyMC/Stan let anyone fit hierarchical models.
2020s
LLM sampling · everything is probabilistic
GPT-4/Claude output token distributions. Temperature, top-k, top-p are all probability operations.

(b) Visual walkthrough · 15 min

PMF vs PDF vs CDF

The five distributions at a glance

Distribution cheat sheet

Bernoulli(p)

P(X=1)=p, P(X=0)=1-p. Mean=p, Var=p(1-p). Example: does this email open?

yes/no

Binomial(n, p)

Sum of n independent Bernoulli(p). Mean=np, Var=np(1-p). Example: heads in 10 flips.

count

Poisson(λ)

Events in a fixed interval when they're independent and rare. Mean=Var=λ. Example: earthquakes per year.

arrivals

Normal(μ, σ)

Continuous bell curve. Mean=μ, Var=σ². Example: measurement noise, height. Emerges via CLT.

bell

Exponential(λ)

Waiting time between Poisson events. Mean=1/λ, memoryless. Example: seconds between HTTP requests.

waiting

Expectation and variance in one paragraph

E[X] = the long-run average value of X

For discrete X: E[X] = Σ x · P(X=x). For continuous X: E[X] = ∫ x · f(x) dx. Just a weighted average with weights = probabilities.

Var(X) = E[(X - E[X])²]

Average squared deviation from the mean. Measures spread. Standard deviation σ = √Var(X).

Linearity: E[aX + bY] = aE[X] + bE[Y]

Always. Works whether X and Y are independent or not. This is a huge trick — most complex problems reduce via linearity.

For independent X, Y: Var(X + Y) = Var(X) + Var(Y)

Variance adds only when independent. Otherwise you need Cov(X, Y). This is where ‘correlation matters’ in portfolio math.

Bayes' rule — the update formula

P(A | B) = P(B | A) · P(A) / P(B)
        =    likelihood × prior     /   evidence

Prior P(A)

What you believed before

Base rate of the hypothesis
Prevalence of the disease
Fraction of spam in your inbox

Likelihood P(B|A)

How well does A explain B?

Sensitivity of the test
Probability of this word given it's spam
Model's assigned probability to the data

Evidence P(B)

How likely is B overall?

Total probability across all hypotheses
P(B) = Σ P(B|Aᵢ) P(Aᵢ) — the normaliser
Often the hardest thing to compute directly

Posterior P(A|B)

Updated belief

What you should now believe
Rerun the formula with new evidence — the posterior becomes the next prior
This is how learning works, formally

The Central Limit Theorem, in one sentence

Common misconception

✗ What most people think

"A test that's 99% accurate means a positive result is 99% likely to be correct. If my classifier flags a transaction as fraud with 99% accuracy, it's almost certainly fraud."

✓ What is actually true

Those are different conditional probabilities. P(positive result | fraud) is a property of the detector; P(fraud | positive result) also depends on the base rate. When fraud is rare, the false positives drawn from the enormous legitimate population swamp the true positives, and a "99% accurate" detector can be wrong on the large majority of its alerts.

Why the myth is so sticky

Because in the balanced examples used to teach classification, the two quantities really are close, so the distinction never surfaces. And English makes it worse — "99% accurate" is a sentence that reads as if it describes the answer rather than the instrument. The sticky part is that reversing a conditional feels harmless in a way that reversing an equation obviously wouldn't. This one error is behind most of the disappointment in deployed classifiers: a model that looked strong on a balanced evaluation set becomes an alert stream nobody trusts, because production has a 0.1% positive rate and nobody recomputed precision at that base rate. It is also why accuracy is the wrong headline metric for any rare event: always predicting "no" on a 0.1% positive class scores 99.9%.

Prove it to yourself

Run the arithmetic on a realistic base rate — the number is genuinely counterintuitive:

base   = 0.001    # 0.1% of transactions are fraud
tpr    = 0.99     # catches 99% of fraud
fpr    = 0.01     # flags 1% of legitimate ones

n = 1_000_000
tp = n * base * tpr           #    990 true alerts
fp = n * (1 - base) * fpr     #  9,990 false alerts
print(tp / (tp + fp))         # ~0.09 -> 9% of alerts are real fraud

From first principles

Start with the question

Why does the Central Limit Theorem hold — why do sample means become normally distributed regardless of the shape of the underlying distribution? This is a strong claim, and it is the reason nearly all of applied statistics works.

1
A sample mean is a sum of independent random variables, divided by n.
forced by · that is the definition of an average, and division by a constant only rescales
2
Summing independent variables averages out their individual idiosyncrasies: an unusually high draw tends to be offset by a typical or low one.
forced by · independence means the deviations are uncorrelated, so they partially cancel rather than accumulate
3
The variance of the sum grows like n, so the standard deviation grows like √n, and the mean's standard deviation therefore shrinks like 1/√n.
forced by · variances of independent variables add, and dividing by n scales variance by 1/n²
4
What remains after that rescaling depends only on the first two moments of the original distribution — its mean and variance. Higher-order features like skew and specific shape are suppressed at a rate governed by n.
forced by · the deviation from normality contributed by skew shrinks faster than the leading term as terms accumulate
5
So the limiting shape is the same for every distribution with finite variance. That shape is the normal distribution — the unique attractor of this summing-and-rescaling operation.
forced by · the normal is stable under summing (a sum of normals is normal) and everything else is drawn toward it

⇒ Therefore

Therefore the normal distribution is not an assumption we impose on data for convenience; it is the shape that averages are forced into by summation. This is why we can build confidence intervals for a mean without knowing the population's distribution at all.

And note exactly what this predicts — including where it fails, which is the part that matters operationally. The 1/√n rate means halving your error bar requires 4× the data, which is why A/B tests on small effects need enormous samples. Finite variance is required, so heavy-tailed data (latencies, incomes, request sizes) converges slowly or not at all — which is precisely why you report p99 latency rather than mean latency, and why a mean over a heavy-tailed distribution is a genuinely unstable statistic. And independence is required, so correlated samples (repeated measurements on the same user, autocorrelated time series) break it, making your confidence intervals far too narrow and your results look significant when they aren't.

Mental modelBase rate first, then evidence

Any probabilistic judgement is two numbers multiplied, never one. Start with the prior — how common is this thing before any evidence? — then apply the likelihood ratio — how much more often does this evidence appear when the thing is true than when it isn't? Evidence updates a prior; it never replaces it.

The practical consequence is that for rare events, evidence must be extraordinarily strong to move you far. A detector that's ten times more likely to fire on fraud than on legitimate traffic still leaves you far from certain when fraud is one in a thousand — it moves the odds by a factor of ten, and you needed a factor of a thousand.

P(A|B) ≠ P(B|A). Write both down explicitly whenever you catch yourself reasoning about a test, a classifier, or a symptom.
Accuracy is meaningless for imbalanced classes. Report precision and recall at your production base rate, and be explicit about which errors you are choosing to make.
Independence is the assumption that quietly fails and it always fails in the direction of overconfidence — correlated failures, shared upstream dependencies, repeated measures on the same subject.
Expected value alone is not a decision. Variance and tail behaviour matter, especially when one tail is catastrophic and unrecoverable rather than merely expensive.

🔔 Fires when you see

Fire this model the moment you see: a rare-event classifier reporting high accuracy · alert fatigue on a monitoring system · "the model is 95% confident" · an A/B test that reached significance in a day · anything described as a false positive rate without a base rate attached.

The tradeoff

Setting the threshold on a fraud/anomaly detector: optimise for precision, for recall, or calibrate the probabilities and decide per case?

High precision (few false alarms)

+ you gain every alert is worth investigating, so reviewers keep trusting the system; investigation cost per alert stays justified and the queue stays manageable

− you pay you miss the majority of true positives, and the misses are invisible — nothing in your metrics shows the fraud you never flagged, so the system looks healthier than it is

pick when each alert triggers expensive manual review or a customer-visible action, and reviewer trust is the binding constraint — the usual case when a human is in the loop

High recall (catch almost everything)

+ you gain few misses, which is essential when a single missed case is catastrophic or legally consequential

− you pay alert volume that overwhelms reviewers; the predictable end state is that alerts get ignored wholesale, at which point effective recall collapses to zero regardless of what the model does

pick when a miss is far more costly than an investigation and you have the capacity — human or automated — to actually process the volume

Calibrated probabilities + cost-based decision

+ you gain separates the model (estimate the probability) from the policy (decide what to do), so thresholds become an explicit business decision per segment, and you can route high-confidence cases to automation and mid-range ones to humans

− you pay requires genuine calibration work — most classifiers are not calibrated out of the box and need Platt scaling or isotonic regression — plus real cost estimates for each error type, which the business often cannot supply

pick when error costs differ across segments, or you need to defend the threshold to a regulator or a stakeholder, or volume forces tiered automated handling

What a senior engineer actually does

Calibrate whenever you can, because a calibrated score lets the threshold be a decision with an owner rather than an artefact of whoever trained the model. An uncalibrated score forces every downstream consumer to guess what 0.8 means, and they will all guess differently.

The failure mode nobody measures: when precision is too low, humans stop reading the alerts, and the system's real-world recall silently drops to near zero while its measured recall stays high. Any threshold decision for a human-facing system is therefore partly a decision about attention, not just about statistics — and that means monitoring the investigation rate alongside the model metrics.

(c) Hands-on · 25 min

You'll simulate five distributions, verify expectation-formula = sample-mean, apply Bayes' rule to a diagnostic-test scenario, and watch CLT emerge. Save as probability_demo.py.

Run it:

uv run --with numpy python probability_demo.py

Expected output (numbers vary slightly per run):

What each block does

Anatomy of the code

demo_distributions

For each family, draw 200k samples and compare sample mean/var to the closed-form theoretical values. Empirical numbers should match to ~2 decimal places.

verify

bayes_medical_test

A famous counterintuitive result: even with a 99%-accurate test, a positive result on a rare disease only means 50% probability of actually having it. Compute both analytically and by simulating 1M people.

bayes

central_limit_theorem_demo

Average n draws from a heavily-skewed Exponential(1). As n grows, kurtosis (peakedness) drops from ~9 (very fat-tailed) toward 3 (normal). CLT in one column.

clt

confidence_interval_demo

1000 fake ‘surveys’ of 500 people each. Compute a 95% CI on the sample proportion; count how often it contains the true p. Should be ~950 out of 1000. This is what ‘95% confidence’ means, operationally.

coverage

Try itRebuild the intuition that ‘rare event, positive test’ is often a false alarm

Change the disease prevalence and see how the posterior changes:

for prev in [0.0001, 0.001, 0.01, 0.05, 0.10, 0.50]:
    p_pos = 0.99 * prev + 0.01 * (1 - prev)
    posterior = (0.99 * prev) / p_pos
    print(f"prevalence {prev*100:>5.2f}%  →  P(disease|+) = {posterior*100:>5.1f}%")

Notice how the posterior rises non-linearly. At 0.01% prevalence a positive test still means only ~1% chance of disease — the ‘base-rate fallacy’ that fools most people (including doctors — see the classic Gigerenzer studies).

💡 Hint · Notice that even at 99.9% specificity + 1% prevalence, the posterior only rises to ~91%. To get a very high posterior after one test on a rare disease you need EXTREMELY specific test OR a follow-up test. This is why doctors always confirm rare-disease positives with a second, independent test.

(d) Production reality · 15 min

War story Netflix · engineering blog· 2019hundreds of A/B experiments per week

🔥 What broke

Product teams ran A/B tests, peeked at the p-value early, and stopped the experiment ‘when it looked significant’. This ‘optional stopping’ dramatically inflates false-positive rates: a genuinely-null experiment tested repeatedly WILL eventually appear significant just by chance.

Result: many ‘winning’ features didn't actually win. Metrics moved randomly, teams celebrated, launched, and didn't move real user behaviour.

🧯 The fix

Adopt sequential-testing methods (mixture sequential probability ratio tests, group sequential designs) that CONTROL false-positive rate under peeking. Or precommit to a fixed sample size before the test starts. Netflix and other big-tech teams publish tooling for this (‘Always-Valid Inference’).

🎓 Lesson to steal

Classical p-values assume you look ONCE, at a predetermined sample size. Peeking is fine if your methodology accounts for it; if not, you're just measuring noise. Understand what your test actually promises.

Post-mortem

War story Common failure · assuming independence in real dataubiquitous

🔥 What broke

A data scientist reports the standard error of a customer-satisfaction score using the ‘simple’ formula (σ/√n), assuming responses are i.i.d. But respondents are clustered by household, region, and demographic — highly correlated. The true standard error is ~3× bigger; the confidence interval is 3× too narrow; ‘significant’ results are noise.

🧯 The fix

Use cluster-robust standard errors, hierarchical (multilevel) models, or bootstrap. Or design the sample to actually BE i.i.d. — random sampling instead of convenience sampling. Learn to recognise the ‘something looks off’ smell when a formula assumes independence and your data clearly aren't.

🎓 Lesson to steal

Independence is a fragile assumption. Any correlated structure (time, location, user, household) breaks the naive formulas. When you're not sure, bootstrap.

War story Common failure · using accuracy on imbalanced dataweekly, every ML team

🔥 What broke

A team builds a fraud detector for a dataset with 0.1% fraud rate. They report ‘99.9% accuracy’. The model predicts ‘not fraud’ for every input. Zero fraud detected. Fraud losses continue. Real problem: they optimised the wrong metric.

🧯 The fix

For imbalanced classes, use precision, recall, F1, AUC-ROC, or PR-AUC. Or reframe as an expected-cost problem: what's the cost of missing a fraud vs blocking a legit transaction? Optimise expected utility, not accuracy. This is applied probability + decision theory in production ML.

🎓 Lesson to steal

‘Accuracy’ is only meaningful when classes are balanced. On rare events, always report per-class metrics AND the base rate. It's the same base-rate lesson as the medical-test example, just wearing ML clothes.

Where this shows up in the rest of the plan

Probability is the language of uncertainty

S022 · Statistics — CLT, hypothesis testing

The applied side of what we set up here.

S079 · Naive Bayes classifier

Literally Bayes' rule applied to text features. Still a strong baseline.

S082 · Logistic regression

Model outputs probabilities via sigmoid; loss = negative log-likelihood.

S097 · Language models sample from a distribution

GPT/Claude output a probability distribution over the next token. Temperature, top-k, top-p all reshape it.

S099 · Reinforcement learning

Expected reward over stochastic environments — the ‘E’ everywhere in the Bellman equation.

S110 · Bayesian A/B testing

Posterior distributions over conversion-rate differences; no p-values in sight.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal

★ = stretch question

Explain-out-loud test

What is expectation, in one sentence, with one example?
State Bayes' rule and name the four pieces.
Why is a positive medical test on a rare disease often not proof of the disease?

What comes next

📐MathNext →

S022 · Statistics — CLT, Hypothesis Testing, Confidence Intervals

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.