Imbalanced Data — SMOTE, Class Weights, Thresholds

Why this session exists

A classifier that predicts negative for every row is 99% accurate at 1% positive prevalence, yet has zero positive recall. This is a useful baseline, not evidence that accuracy is mathematically wrong or that every learner must fail. Logistic regression minimizes log loss, not accuracy; a rare class with distinctive features can still be learned and detected at 0.5. Separate ranking, probability calibration, and decision policy before changing training data.

You will diagnose imbalance using both positive count and prevalence, implement random over/undersampling and SMOTE safely, derive balanced class weights, select validation thresholds, and compare average precision with operating-point confusion matrices. A validation target is an estimate, not a deployment SLA guarantee.

Prerequisites

Use S092 metrics, S093 pipelines, and S086 logistic regression. Declare the positive label, prediction time and sampling unit. A hundred positives may be sufficient for one simple problem and inadequate for a heterogeneous one; no positive-count cutoff establishes learnability.

(a) Intuition · 5 min

A lifeguard who never raises an alarm can be correct on nearly every observation while missing the event that matters. But an alarm on every observation is not the answer either: it catches everything and overwhelms the responders. The useful question is how many true cases can be caught within a defensible false-alarm cost or review capacity.

The model supplies scores. A threshold turns scores into actions. Lowering a threshold on a fixed ranking cannot improve average precision or ROC-AUC; those metrics use scores, not the selected cutoff. It can improve recall while changing precision and workload. Precision need not change monotonically as tied groups or individual examples enter the alert set.

The three fixes, ranked by "try first"

Start with a representative split, metric and unweighted model. Inspect the validation precision–recall curve. If useful operating points already exist, select a policy without retraining. Next compare weighted learning; then compare resampling if its geometry or computational benefits justify it. This is a debugging order, not a theorem that threshold tuning beats SMOTE.

Threshold selection itself consumes validation information. Weights and resampling can change both ranking and calibration, and neither has zero leakage risk when tuned on the test set. Anomaly detection is an alternative when labeled positives are unavailable or unrepresentative, not an automatic consequence of prevalence below a magic percentage. Unusual events need not be harmful events.

A quick history

SMOTE's named contribution is synthetic interpolation between minority neighbors, rather than simple duplication. The imbalanced-learn guide distinguishes it from ADASYN, which allocates synthesis according to local difficulty, and from random oversampling. That mechanism is what the historical development contributes to this lesson; it does not establish a universal adoption date, citation count or industry success rate.

The three families of fixes

For a weighted binary log loss, an ideal unrestricted population minimizer is

Derive this by setting the derivative of $-w_1p\log q-w_0(1-p)\log(1-q)$ to zero. Balanced weights are $w_c=n/(K n_c)$. With 990 negatives and 10 positives, $w_0=1000/1980$ and $w_1=50$, a ratio of 99. For an uninformative problem with $p=0.01$, the weighted optimum is $q=0.5$, not the deployment probability 0.01. This refutes the idea that class weighting automatically preserves calibration while SMOTE alone breaks it.

Random duplication resembles weighting the empirical loss, but stochastic batches, regularization normalization and solver details can prevent identical fitted parameters. A custom error-cost weight is also not automatically a calibrated probability model or the optimal final action policy.

How SMOTE actually works

Fit scaling on training data, find each minority point's nearest minority neighbors, choose a point–neighbor pair and sample

For $x_i=(1,2)$, $x_j=(3,6)$ and $\lambda=0.25$, the result is $(1.5,3)$. Repeat until the requested sampling ratio is reached. With k_neighbors=5, each resampled class needs at least six training examples in every fitting fold, not merely in the full dataset.

The local segment must be meaningful. If nearby same-class points straddle a gap occupied by another class, interpolation can introduce mislabeled synthetic points. The entire minority class need not be globally convex or unimodal: disconnected clusters can be fine if neighbor choices remain local. High dimension, irrelevant features, inappropriate scaling and label noise can make the neighbor geometry unreliable; none makes failure inevitable on every dataset.

Ordinary SMOTE can produce fractional one-hot indicators. SMOTENC handles mixed categorical/continuous inputs with a different categorical treatment; SMOTEN addresses categorical-only inputs. Neither validates business constraints such as mutually incompatible fields. Check synthetic validity as well as metrics.

When to use which resampler

BorderlineSMOTE concentrates synthesis on minority points judged near a class boundary from their local neighborhood, rather than treating all minority anchors equally. This can focus useful effort but also emphasizes an unreliable boundary when labels or distances are poor. Tomek-link or edited-nearest-neighbor cleaning removes selected boundary/noisy points; it is not the same operation as random undersampling. SMOTETomek and SMOTEENN combine synthesis with those cleaning rules. Focal loss instead reduces the relative loss contribution of easy examples. In binary form, $FL=-\alpha_t(1-p_t)^\gamma\log p_t$, where $p_t$ is the score assigned to the true class. At $\gamma=0$ this becomes weighted cross entropy; for positive gamma, confidently correct examples contribute less. It belongs to the loss branch and still needs independent evaluation.

In sklearn, logistic regression, SVC and random forests support class_weight. XGBoost's binary weighting interface uses scale_pos_weight or sample weights, not the same universal class_weight='balanced' argument.

Calibration, prevalence and the decision rule

Calibration means that cases assigned probability near 0.7 have positive frequency near 70% in the population being assessed. An unweighted model is not guaranteed calibrated either. Fit any calibrator on held-out or cross-fitted predictions at the intended prevalence; fitting it to training predictions can overfit.

Under class-conditional random sampling only, with unchanged $P(x\mid y)$, prior correction relates sampled and deployment odds:

SMOTE changes feature distributions too, so this prior-only correction is not generally sufficient for it. At deployment prevalence $\pi$, TPR $r$ and FPR $f$, precision is $\pi r/[\pi r+(1-\pi)f]$. At $\pi=0.001,r=0.95,f=0.01$, precision is about 8.68%: a small FPR can still create a large queue. ROC remains a valid class-conditional ranking summary; report AP/PR and workload alongside it rather than calling ROC a lie.

For calibrated deployment probabilities and only error costs, predict positive when $C_{FP}(1-p)\le C_{FN}p$, giving $t=C_{FP}/(C_{FP}+C_{FN})$. If the actual constraint is 500 reviews per day, evaluate precision and recall at that capacity, including ties and daily volume variation. Calibration and capacity are different requirements.

(c) Hands-on · 25 min

This standalone CPU lab uses NumPy 2.2.6, sklearn 1.7.2 and imbalanced-learn 0.14.0. It disables synthetic label flipping, compares five training interventions using the same folds, selects one with development CV, then selects its cutoff on a separate validation set. Only the frozen winner is scored on the final test. It is a generated-data experiment, not a fraud benchmark.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (average_precision_score, precision_recall_curve,
                             confusion_matrix, classification_report)
from sklearn.utils.class_weight import compute_class_weight
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
 
X, y = make_classification(n_samples=6000, n_features=12, n_informative=6,
    weights=[.99, .01], flip_y=0, random_state=42)
ids = np.arange(len(y))
dev, test = train_test_split(ids, test_size=.2, stratify=y, random_state=1)
train, val = train_test_split(dev, test_size=.25, stratify=y[dev], random_state=2)
assert not (set(train) & set(val) or set(dev) & set(test))
cv = list(StratifiedKFold(3, shuffle=True, random_state=3).split(X[train], y[train]))
assert min(np.bincount(y[train][a]).min() for a, _ in cv) > 5
 
def pipeline(sampler=None, weighted=False):
    steps = [('scale', StandardScaler())]
    if sampler is not None:
        steps.append(('sample', sampler))
    steps.append(('model', LogisticRegression(max_iter=1000,
        class_weight='balanced' if weighted else None)))
    return Pipeline(steps)
 
models = {
    'plain': pipeline(), 'weighted': pipeline(weighted=True),
    'over': pipeline(RandomOverSampler(random_state=4)),
    'under': pipeline(RandomUnderSampler(sampling_strategy=.2, random_state=4)),
    'smote': pipeline(SMOTE(random_state=4, k_neighbors=5)),
}
cv_scores = {name: cross_val_score(m, X[train], y[train], cv=cv,
    scoring='average_precision', n_jobs=1).mean() for name, m in models.items()}
print('training CV AP', cv_scores)
name = max(cv_scores, key=cv_scores.get)
m = models[name].fit(X[train], y[train])
s = m.predict_proba(X[val])[:, 1]
p, r, t = precision_recall_curve(y[val], s)
assert len(p) == len(r) == len(t) + 1
eligible = np.flatnonzero(r[:-1] >= .9)
assert len(eligible), 'No feasible validation threshold'
j = eligible[np.argmax(p[:-1][eligible])]
threshold = float(t[j])
# This validation target is not asserted on the untouched test population.
score = m.predict_proba(X[test])[:, 1]
print('winner', name, 'validation recall/precision', r[j], p[j], 'cutoff', threshold)
print('test AP', average_precision_score(y[test], score))
for cutoff in [.5, threshold]:
    pred = score >= cutoff
    print('fixed cutoff', cutoff, 'TN FP / FN TP',
          confusion_matrix(y[test], pred, labels=[0, 1]))
    print(classification_report(y[test], pred, zero_division=0))
w = compute_class_weight('balanced', classes=np.array([0, 1]), y=y[train])
assert np.isclose(w[1] / w[0], (y[train] == 0).sum() / y[train].sum())
assert np.allclose(np.array([1., 2.]) + .25 * (np.array([3., 6.]) - [1., 2.]), [1.5, 3.])
print('training weights', w, 'split positive counts', *(int(y[a].sum()) for a in [train,val,test]))

Anatomy of the script

Scaling and resampling live in an imbalanced-learn pipeline so each CV fit learns them only from its training subset. Prediction transforms without resampling: validation and test support stay unchanged. Unlike an ordinary sklearn transformer, a sampler changes both X and y; also note imbalanced-learn's documented fit_transform versus fit().transform() behavior is not identical when sampling is involved.

AP is the noninterpolated sum of precision weighted by recall increments, not every definition of PR area. The last PR point has no threshold, so the code searches p[:-1] and r[:-1]. It uses >= consistently, states a tie policy (first maximizing candidate), and never chooses a threshold from test labels. Report positive support because this small validation set makes recall estimates coarse.

Leakage practice with an answer

Exercise: Why is “scale all rows, SMOTE all rows, then split” invalid? Does the difference from honest CV quantify leakage exactly?

Answer: Fitted scaling includes future validation statistics; SMOTE neighbor selection may use future validation rows; synthetic relatives can cross the split; and balancing evaluation changes prevalence. These are several simultaneous changes. A score increase is not an exact measure of leaked signal, nor guaranteed in every run. Audit indices and sampler placement, not just whether a cheating score happens to rise. A pipeline cannot fix features that already include future outcomes.

(d) Production reality · 15 min

In a hypothetical payments queue, weighting increases alerts but capacity remains fixed. Compare ranking at the actual capacity and calibrate expected loss if probabilities drive action. Do not attribute this mechanism to an unverified company incident.

For recommendation candidates, downsampling negatives can save training work, but a uniform sample and a hard-negative sample encode different distributions. Evaluate against the actual candidate-generation process. A new-user split and a future-interaction split answer different questions.

For screening, report false alarms and missed cases at the deployment prevalence, including confidence intervals and subgroup counts. An AUROC of 0.99 alone cannot establish a useful clinical operating point. Label delays and verification bias can corrupt measured recall; no resampler repairs missing outcome ascertainment.

Where this shows up in the rest of the plan

S095 makes sampling fold-local. Stratification helps preserve support for independent classification rows, but group/time isolation takes priority; it cannot create positives when their count is smaller than the number of folds. Feature availability, rare error evaluation and monitoring all reuse this distinction. Monitor observed prevalence, AP, calibration and capacity with delayed labels; investigate changes instead of retraining at an arbitrary 20% drift threshold.

(e) Recall + stretch · 10 min

1. Why might 0.5 miss most positives? Low conditional probabilities can coexist with useful ranking. It is not a theorem that the optimizer predicts every negative.
2. What does balanced mean? $n/(K n_c)$ per-class sample weight; at 99:1 its positive/negative ratio is 99.
3. When is plain SMOTE inappropriate? When interpolated fields are invalid, neighbor support is insufficient, or local segments cross classes. A tiny ratio alone does not prove any of these.
4. Why the sampler-aware pipeline? Resampling alters labels and rows during fitting, not prediction; ordinary sklearn intermediate transformers preserve the sample correspondence.
5. Recall is 5% at 0.5: next action? Inspect validation operating points and costs. Lowering the cutoff can reach high recall by predicting almost everyone positive, which may be useless.
6. Why reassess calibration? Both weighting and resampling alter the training objective/distribution. Use independent representative predictions for calibration.
7. Is stratification mandatory? No. Choose the deployment split first; use stratification where compatible and count positives in each fold.
8. Why undersample ten million negatives? To reduce fit cost while preserving useful negative modes. Measure information loss; neither a 100-fold speedup nor a 1% score loss follows from row counts.
9. Nothing works: what follows? Revisit features, labels, split, model family, capacity and costs before declaring signal absent. Duplicating rows creates no independent labels.
10. Stretch: one positive in 10,000. Use time-aware, possibly group-aware splits; retain untouched deployment-prevalence evaluation; choose supervised learning when labels support it; compare sampled/weighted baselines; set validation capacity/cost policy; monitor support, delayed outcomes and drift. Set alert bounds from uncertainty and operational risk, not universal PSI/AP percentages.

Explain-out-loud test

Explain why a high-accuracy baseline can be useless, distinguish data/loss/decision interventions, and trace a SMOTE neighbor back to its permitted fitting fold. Then derive the weighted optimum $q$ and explain why a threshold cannot improve score-based AP.

What comes next

Continue to model selection and CV. The next lesson puts every intervention—including the choice to use SMOTE—inside an explicit evaluation boundary.

Sources and execution boundary

Primary library references: sampling mechanisms, resampling leakage, pipeline semantics, balanced weights, AP, PR endpoints. The CPU program tests APIs and split discipline on synthetic data; no fraud, healthcare or deployment performance is inferred.