S094 · Imbalanced Data — SMOTE, Class Weights, Thresholds

Module M11: Classical ML · Session 94 of 130 · Track: ML 🎭

What you'll be able to do after this session

• Explain Imbalanced Data to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S092 — Evaluation Metrics — P/R/F1/ROC/PR/AUC

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Krish Naik: Handling Imbalanced Datasets (SMOTE etc.) — 10-min overview of the whole toolkit.
• 🎥 Deep dive (30–60 min) · Data School: Handling imbalanced classes with scikit-learn — longer walkthrough of class weights, thresholds, and pitfalls.
• 🎥 Hands-on demo (10–20 min) · Krish Naik: SMOTE with imbalanced-learn in Python — live SMOTE code on a real dataset.
• 📖 Canonical article · imbalanced-learn: User Guide — the library and the theory in one place.
• 📖 Book chapter / tutorial · Google ML: Imbalanced Data — short guide to downsampling and upweighting.
• 📖 Engineering blog · Stripe: How we built Radar for fraud detection — how Stripe handles the extreme (fraud rate <1%) imbalance.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Fraud, churn, disease — all imbalanced.

Fraud, cancer, churn, click-through, machine failure — most of the interesting problems in the world are imbalanced. The positive class you care about might be 1 in 100, 1 in 1000, or 1 in a million. A model trained naively on that data will discover that "always predict negative" is the easiest win, and your accuracy will be 99.9%. Congratulations, your fraud model just approved every credit card thief on the planet.

The mental model is a democracy that gerrymanders. When one class is 99% of the votes, the algorithm learns to please that majority and ignores the minority entirely. There are three main ways to fight back: resample the data (oversample positives with SMOTE, or undersample negatives), reweight the loss (tell the model each positive is worth 20 negatives), or retune the threshold (default 0.5 is arbitrary — pick the threshold that maximises F1 or business value on validation data).

Gotcha to remember forever: never apply SMOTE (or any resampling) to your validation or test set. You want your held-out metrics to reflect reality — the imbalanced world your model will actually face. Resampling belongs strictly inside the training-fold of your cross-validation loop, or before .fit() on train data only.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The three strategies at a glance

How SMOTE works, in one picture

For each minority point, SMOTE picks a random minority neighbour and places a new synthetic point somewhere on the segment between them. Repeat until the minority class is balanced.

Worked example: fraud, 10,000 rows, 100 positives (1%)

• Baseline (no rebalancing) at threshold 0.5: Precision 0.60, Recall 0.10. Business is unhappy.
• Threshold 0.15 (tuned on validation for max F1): Precision 0.35, Recall 0.55.
• class_weight='balanced' + threshold tuned: Precision 0.30, Recall 0.70.
• SMOTE on train + threshold tuned: Precision 0.32, Recall 0.72. Marginal gain, sometimes a loss.

The lesson: try threshold tuning + class weights first. Reach for SMOTE only if it demonstrably helps on your problem, and never trust papers that report on SMOTE-balanced test sets.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# pip install imbalanced-learn scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, precision_recall_curve, average_precision_score
from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
 
X, y = make_classification(
    n_samples=20000, n_features=20, n_informative=5,
    weights=[0.99, 0.01], flip_y=0.01, random_state=42,
)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
print(f"Train pos rate: {ytr.mean():.4f} | Test pos rate: {yte.mean():.4f}")
 
def eval_all(name, probs):
    print(f"\n=== {name} ===")
    preds = (probs >= 0.5).astype(int)
    print("threshold 0.5:")
    print(classification_report(yte, preds, digits=3))
    p, r, thr = precision_recall_curve(yte, probs)
    f1 = 2*p*r/(p+r+1e-12)
    b = f1.argmax()
    best_thr = thr[b] if b < len(thr) else 0.5
    print(f"best F1={f1[b]:.3f} @ thr={best_thr:.3f} P={p[b]:.3f} R={r[b]:.3f}")
    print(f"PR-AUC={average_precision_score(yte, probs):.3f}")
 
m1 = LogisticRegression(max_iter=1000, random_state=42).fit(Xtr, ytr)
eval_all("baseline", m1.predict_proba(Xte)[:, 1])
 
m2 = LogisticRegression(max_iter=1000, class_weight="balanced", random_state=42).fit(Xtr, ytr)
eval_all("class_weight balanced", m2.predict_proba(Xte)[:, 1])
 
pipe = ImbPipeline([
    ("smote", SMOTE(random_state=42)),
    ("clf",   LogisticRegression(max_iter=1000, random_state=42)),
])
pipe.fit(Xtr, ytr)
eval_all("SMOTE + LogReg", pipe.predict_proba(Xte)[:, 1])

What to observe when you run it:

1. Baseline recall at threshold 0.5 will be tiny — often <0.10. Business would fire you.
2. class_weight='balanced' boosts recall and (usually) drops precision.
3. SMOTE performs comparably to class weights on this synthetic dataset. Real results vary.
4. The best-F1 threshold is nearly always well below 0.5 — typically 0.1–0.3 for a 1% positive class.
5. PR-AUC is a much better summary here than ROC-AUC; ROC-AUC will read misleadingly high.
6. Use imblearn.pipeline.Pipeline, not sklearn.pipeline.Pipeline, for SMOTE — the former ensures SMOTE runs only during fit, not predict.

Try this modification: wrap the SMOTE pipeline in cross_val_score(pipe, X, y, cv=StratifiedKFold(5), scoring='average_precision') and compare against the class-weighted model. If SMOTE isn't strictly better in CV, skip it in production.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: SMOTE that leaked. A credit-risk team applied SMOTE to their entire dataset before splitting into train/test. Test accuracy: 0.99. Production accuracy: 0.68. Synthetic minority points in the test set were interpolations of training minority points, so the model was practically memorising. Always resample inside the training fold, using imblearn.pipeline.Pipeline.

War story #2: chasing recall into oblivion. An anti-abuse team at a big social network moved their spam-classifier threshold from 0.5 to 0.05. Spam catch rate went from 60% to 90%. FPR went from 0.1% to 3%. At scale, that meant blocking 30 million legitimate posts per day. They rolled back within hours. Your operating point is a business decision — know the cost of an FP and an FN.

Common bugs:

• SMOTE on categorical features. SMOTE interpolates in numeric space; averaging one-hot vectors is nonsense. Use SMOTENC (mixed) or SMOTEN (all-categorical).
• Class weights forgotten inside a Pipeline. Verify with a tiny sanity-check dataset.
• Reporting metrics on a rebalanced test set. Instant credibility death. Test set is sacred.
• Ignoring "unknown" as a class. Most points are neither confidently positive nor negative — consider a 3-way outcome: positive / negative / send to human review.

How top teams handle it:

• Stripe Radar (fraud), Meta Integrity (spam), YouTube Trust & Safety rely on class weights + threshold calibration + human review queues for ambiguous cases. Pure resampling is rarely the primary lever.
• Netflix, Airbnb use cost-sensitive loss functions — bespoke asymmetric losses that encode the business cost of each error type.

(e) Quiz + exercise · 10 min

1. Why is accuracy a dangerous metric on imbalanced data, and what are two better metrics to report?
2. In your own words, how does SMOTE generate synthetic minority samples? What class of feature does it not work on out of the box?
3. Why must resampling (SMOTE, random over/under) happen inside the training fold and never touch the validation/test set?
4. Given a fraud model with 1% positives, why would the optimal decision threshold on validation typically be well below 0.5?
5. Name one situation where you would prefer class weights over SMOTE, and one where you would try SMOTE first.

Stretch (connects to S092): You inherit a churn model with ROC-AUC 0.86 but at the default 0.5 threshold recall is 0.08. Design a step-by-step diagnostic + fix plan using threshold tuning, class weights, and (only if needed) resampling. What metrics do you report to stakeholders, and on which dataset?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Imbalanced Data? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S095): Model Selection — CV, Hyperparameter Tuning, Optuna
• Previous (S093): Feature Engineering — Encoding, Scaling, Missing
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.