S021 · Probability — Random Variables, Distributions, Expectation

Uncertainty, quantified.

Module M02: Math Foundations · Session 21 of 130 · Track: Math 🎲

What you'll be able to do after this session

Explain Probability to a friend in one minute, out loud, no notes.
Recognise it when you see it in real code / systems / papers.
Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

S015 — Big-O Notation — Reasoning About Scale

Pre-read (skim before the session)

🎥 Intuition (5–15 min) · But what is the Central Limit Theorem? (intro to probability framing) — 3Blue1Brown builds probability intuition visually.
🎥 Deep dive (30–60 min) · MIT 6.041 — Lecture 1: Probability Models and Axioms — John Tsitsiklis lays down the whole framework.
🎥 Hands-on demo (10–20 min) · Random variables & distributions in Python (StatQuest) — quick walkthrough with worked examples.
📖 Canonical article · Seeing Theory — Basic Probability — interactive visual textbook from Brown.
📖 Book chapter / tutorial · Wikipedia — Probability distribution — the survey you can skim in 10 min.
📖 Engineering blog · Netflix — Interpreting A/B test results — probability in production.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Uncertainty, quantified.

Imagine you're about to flip a coin. You know it will land heads or tails, but you don't know which. Probability is the language we invented to say sensible things about that "I don't know" — instead of throwing up our hands, we assign a number between 0 and 1 to every possible outcome. A random variable is just a rule that turns each random outcome into a number: "how many heads in 10 flips?" is a random variable. A distribution is the full price list: for every value the random variable could take, what fraction of the time does it happen? Expectation is the long-run average — if you repeated the experiment a million times, what number would you get on average?

Why does this exist? Because almost every real system has uncertainty baked in: users click or don't click, packets arrive or drop, machines fail or don't, models predict correctly or not. Without probability you'd be stuck saying "sometimes it works, sometimes it doesn't." With it you can say "the tail latency is 200ms at p99" or "this A/B test is 95% likely to be a real lift."

One-sentence gotcha to carry forever: Expected value is not the "most likely value" — the expected value of a single die roll is 3.5, which is impossible to actually roll. Expectation is a long-run average, not a prediction of the next event.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Let's walk through a concrete example: rolling one fair six-sided die.

The random variable X = "the face you rolled." The distribution is uniform:

Outcome (x)	1	2	3	4	5	6
P(X = x)	1/6	1/6	1/6	1/6	1/6	1/6

Expected value: E[X] = 1·(1/6) + 2·(1/6) + ... + 6·(1/6) = 21/6 = 3.5. Nobody rolls 3.5, but average a million rolls and you'll be within a rounding error of it.

Now a slightly richer example — number of heads in 3 fair coin flips. Sample space has 8 equally likely outcomes (TTT, TTH, THT, HTT, THH, HTH, HHT, HHH). Random variable Y = "count of heads." The distribution:

Y	0	1	2	3
P(Y = y)	1/8	3/8	3/8	1/8

E[Y] = 0·(1/8) + 1·(3/8) + 2·(3/8) + 3·(1/8) = 12/8 = 1.5. Notice 1.5 is again "impossible" as a single outcome — it's the average.

Here's the mental model as a flow:

Two families to recognise on sight:

Bernoulli / Binomial — discrete, "success or failure, n times." Coin flips, click-through rates, packet drops.
Normal (Gaussian) — continuous, bell curve. Heights, measurement noise, sums of many small effects (that's the Central Limit Theorem, coming next session).

The key move: whenever you see uncertainty in a system, ask "what's the random variable, what's its distribution, what's its expected value?" That reframing alone unlocks A/B tests, risk models, and monitoring.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# probability_playground.py — run with: python3 probability_playground.py
import random
from collections import Counter
 
random.seed(42)  # reproducible
 
# ---- 1. Simulate rolling a fair die 100,000 times ----
rolls = [random.randint(1, 6) for _ in range(100_000)]
avg = sum(rolls) / len(rolls)
print(f"Die: empirical mean = {avg:.4f}  (theoretical = 3.5)")
 
# ---- 2. Empirical distribution of "heads in 3 flips" ----
def three_flips():
    return sum(random.randint(0, 1) for _ in range(3))
 
samples = [three_flips() for _ in range(100_000)]
counts = Counter(samples)
print("
Heads in 3 flips — empirical vs theoretical:")
theoretical = {0: 1/8, 1: 3/8, 2: 3/8, 3: 1/8}
for k in sorted(counts):
    emp = counts[k] / len(samples)
    print(f"  P(Y={k}) empirical={emp:.4f}  theoretical={theoretical[k]:.4f}")
 
mean_y = sum(samples) / len(samples)
print(f"E[Y] empirical = {mean_y:.4f}  (theoretical = 1.5)")
 
# ---- 3. Law of Large Numbers in action ----
running = []
total = 0
for i, r in enumerate(rolls[:5000], 1):
    total += r
    running.append(total / i)
print(f"
Running average at n=10:   {running[9]:.3f}")
print(f"Running average at n=100:  {running[99]:.3f}")
print(f"Running average at n=1000: {running[999]:.3f}")
print(f"Running average at n=5000: {running[4999]:.3f}")

Checklist — what to observe when you run it:

Die empirical mean lands within ~0.01 of 3.5.
The 3-flip empirical probabilities match 1/8, 3/8, 3/8, 1/8 to 3 decimals.
E[Y] hovers around 1.5.
The running average is noisy at n=10 but tightens to 3.5 by n=5000.
Change the seed — numbers wiggle, but the pattern holds.

Try this modification: replace the fair die with a loaded die where 6 comes up twice as often as any other face. Compute the new theoretical expectation by hand, then verify with simulation.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Probability shows up everywhere in production — usually the moment it's ignored, an incident is born. A/B testing is the canonical example: at Netflix, Microsoft, and Booking.com, teams run thousands of experiments a year, and misreading a "significant" result costs real revenue. The classic bug is peeking — checking your test every hour and stopping when the p-value dips below 0.05. This inflates false positives massively; the Netflix post linked above walks through exactly this failure mode.

Common pitfalls that eat teams alive:

Confusing correlation with expectation. "The average request takes 80 ms" doesn't tell you a user waiting at the p99 sees 800 ms. Percentiles beat averages for latency SLOs — Google's SRE book hammers this repeatedly.
Assuming independence when there isn't any. "Machine failures are independent, so we're safe with 3 replicas." Then a whole rack loses power, all 3 replicas were on it, and 3 nines becomes 2 nines overnight. Correlated failure is the silent killer of availability math.
Small-sample overconfidence. Your model has 92% accuracy on a 50-example holdout. That's ±7% at 95% confidence — basically useless. Always compute confidence intervals on eval metrics.
Ignoring the base rate. A fraud model with 99% accuracy sounds great, until you notice 99% of transactions are legitimate and predicting "always legit" also scores 99%.

Top teams handle this by (a) using sequential testing frameworks (Optimizely, Statsig, Eppo) that adjust for peeking, (b) reporting confidence intervals alongside every metric, (c) monitoring p50/p95/p99 not just average, and (d) simulating rare-event failures explicitly rather than trusting "independence." The mental habit to build: whenever someone shows you a single number, ask "what's the uncertainty around it?"

(e) Quiz + exercise · 10 min

What is the difference between a random variable and its distribution?
Compute E[X] where X is the outcome of rolling a fair 4-sided die.
If you flip a fair coin 10 times, what's the expected number of heads? Why is that intuitive?
Give a real-world example where the expected value is not a possible outcome.
Why does the "peeking" problem in A/B tests inflate false positives?

Stretch (connects to S015 — Big-O): Simulating 1 million die rolls to estimate the mean is O(n) in time and space. How would you estimate the mean using O(1) memory? Write the update rule for a running mean.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

What is Probability? (one sentence, no jargon)
When would you use it? (one concrete example)
When would you NOT use it? (one anti-pattern)

What comes next

Next session (S022): Statistics — CLT, Hypothesis Testing, Confidence Intervals
Previous (S020): Calculus II — Gradients & Gradient Descent from Scratch
Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.