S092 · Evaluation Metrics — P/R/F1/ROC/PR/AUC

Module M11: Classical ML · Session 92 of 130 · Track: ML 📏

What you'll be able to do after this session

• Explain Evaluation Metrics to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S086 — Logistic Regression — Sigmoid, Cross-Entropy, from Scratch

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · StatQuest: The Confusion Matrix — TP/FP/TN/FN with no jargon.
• 🎥 Deep dive (30–60 min) · StatQuest: ROC and AUC, Clearly Explained! — the one-shot best explanation of ROC/AUC on the internet.
• 🎥 Hands-on demo (10–20 min) · StatQuest: Precision vs Recall — the 9-minute clip that finally makes P vs R stick.
• 📖 Canonical article · scikit-learn: Metrics and scoring — the authoritative reference; bookmark it.
• 📖 Book chapter / tutorial · Google ML Crash Course: Classification (P/R/ROC/AUC) — interactive threshold sliders that make it click.
• 📖 Engineering blog · Netflix: Interpreting A/B Test Results — false positives and significance — how Netflix reasons about false positives at scale.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Accuracy is almost never the right metric.

A spam filter that flags every email as "not spam" is 99% accurate on a normal inbox — because 99% of your mail isn't spam. Accuracy is a liar when classes are imbalanced. That single realisation is why we need precision, recall, F1, ROC, and AUC.

Think of a smoke detector. Precision asks: "when it beeps, is there actually smoke?" Recall asks: "when there's actually smoke, does it beep?" A hyper-sensitive detector wakes you when you make toast (low precision, high recall). A dead-battery detector never wakes you but also misses the actual fire (terrible recall). You cannot have both maxed for free — you're always trading them off by moving a threshold.

Gotcha to remember forever: every model that outputs a probability has a decision threshold (usually 0.5 by default). Precision, recall, F1 all depend on that threshold. ROC-AUC and PR-AUC don't — they summarise the model's ranking quality across all thresholds. Report both threshold-free (AUC) and threshold-specific (P/R at the operating point you'll actually deploy) numbers.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The confusion matrix — memorise this shape

From these four cells, everything else falls out:

• Precision = TP / (TP + FP) — of what I flagged, how much was right?
• Recall (Sensitivity, TPR) = TP / (TP + FN) — of what was actually positive, how much did I catch?
• F1 = 2·(P·R)/(P+R) — harmonic mean; punishes imbalance between P and R.
• Specificity (TNR) = TN / (TN + FP)
• FPR = 1 − Specificity = FP / (FP + TN)

ROC vs PR — when to use which

Worked example: fraud detection, 1000 transactions, 10 actual fraud

Your model at threshold 0.5 predicts 20 frauds. Of those 20, 8 are real and 12 are legit.

• TP = 8, FP = 12, FN = 2, TN = 978
• Precision = 8/20 = 0.40 → 60% of your fraud alerts are false alarms.
• Recall = 8/10 = 0.80 → you catch 80% of fraud.
• F1 = 2·(0.4·0.8)/(0.4+0.8) = 0.533
• Accuracy = (8+978)/1000 = 0.986 — impressively useless.

Lower the threshold to 0.3 and you might get Recall=0.95 but Precision=0.20 (10 real, 40 false alerts). Whether that's better depends on the cost of a missed fraud vs the cost of a false alarm — a business decision, not a math one.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# pip install scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    confusion_matrix, roc_auc_score, average_precision_score,
    precision_recall_curve, classification_report,
)
 
# Imbalanced binary problem: 5% positives
X, y = make_classification(
    n_samples=10000, n_features=20, weights=[0.95, 0.05], random_state=42,
)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
 
model = LogisticRegression(max_iter=1000, class_weight="balanced").fit(Xtr, ytr)
probs = model.predict_proba(Xte)[:, 1]
 
# Default threshold 0.5
preds_05 = (probs >= 0.5).astype(int)
print("=== threshold = 0.5 ===")
print(confusion_matrix(yte, preds_05))
print(classification_report(yte, preds_05, digits=3))
 
# Sweep thresholds and pick the one that maximises F1
prec, rec, thr = precision_recall_curve(yte, probs)
f1 = 2 * prec * rec / (prec + rec + 1e-12)
best_idx = f1.argmax()
best_thr = thr[best_idx] if best_idx < len(thr) else 0.5
print(f"\nBest F1 = {f1[best_idx]:.3f} at threshold = {best_thr:.3f}")
print(f"  precision = {prec[best_idx]:.3f}, recall = {rec[best_idx]:.3f}")
 
# Threshold-free summaries
print(f"\nROC-AUC = {roc_auc_score(yte, probs):.3f}")
print(f"PR-AUC (Average Precision) = {average_precision_score(yte, probs):.3f}")

What to observe when you run it:

1. Default-threshold classification_report shows precision, recall, F1 for each class — always inspect both, not the "accuracy" line.
2. best_thr will rarely equal 0.5 for imbalanced problems — usually much lower.
3. ROC-AUC will look impressive (often 0.90+) even when precision at 0.5 is embarrassing — ROC-AUC is largely insensitive to imbalance.
4. PR-AUC is a much more honest read-out for the positive class on imbalanced data.
5. class_weight='balanced' doesn't change ranking (AUC) but shifts the operating point.

Try this modification: change weights=[0.99, 0.01] (1% positives). Watch ROC-AUC stay high while PR-AUC and best-F1 collapse. That's the imbalance trap in one experiment.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: the "97% accurate" cancer model. A widely-cited study reported a 97% accurate classifier for a rare disease with ~2% prevalence. Because everyone reported accuracy, a model that predicts "no disease" for every patient would score 98%. The model was, in fact, worse than that constant baseline on the positive class. Always report per-class recall on imbalanced medical / fraud / security problems.

War story #2: AUC that hides horror. A recommender team optimised for ROC-AUC and shipped a model with AUC 0.94. Users hated it. Root cause: the model was excellent at ranking within the top 1000 items but terrible at putting the right item in the top 10 — the only region users see. They switched to NDCG@10 and PR-AUC and the offline metric finally correlated with A/B test wins.

Common bugs:

• sklearn's f1_score default is binary and computes for the positive class only. Multi-class? Pass average='macro' or 'weighted'. They can differ by 30 points.
• roc_auc_score(y_true, y_pred_labels) is nearly always wrong. Pass probabilities, not hard predictions.
• Threshold tuning on the test set. Same crime as early-stopping on test — pick threshold on validation, report on test.
• Class labels flipped. In sklearn, class 1 is positive by convention. If your CSV has "fraud"=0 and "legit"=1, every metric is inverted.

How top teams handle it:

• Google Ads / Meta Ads report calibration curves alongside AUC — a model can rank well but predict wrong probabilities, which breaks bid pricing.
• Netflix reports offline metrics on multiple slices (new users, cold-start content, region) — never one aggregate number.
• Every serious production ML team ships a model card that includes P/R at the deployment threshold, PR-AUC, calibration ECE, and a per-slice breakdown.

(e) Quiz + exercise · 10 min

1. Define precision and recall in your own words, using a real-life analogy that isn't spam detection.
2. Why is accuracy a bad metric on a dataset with 99% negatives, and what should you use instead?
3. What does an ROC-AUC of 0.5 mean about your model? What about 1.0? What about 0.4?
4. When would you prefer the PR curve over the ROC curve, and why?
5. You have a fraud model with ROC-AUC = 0.95 but precision at your operating threshold is 0.15. How is this possible, and what would you do about it?

Stretch (connects to S091): You trained an XGBoost binary classifier for churn (5% positive class). Your CV ROC-AUC is 0.88, but at threshold 0.5 recall is 0.12. Explain what's happening and propose two concrete fixes (one at training time, one at inference time).

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Evaluation Metrics? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S093): Feature Engineering — Encoding, Scaling, Missing
• Previous (S091): Gradient Boosting — XGBoost, LightGBM
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.