Evaluation Metrics — P/R/F1/ROC/PR/AUC

Why this session exists

A model that predicts “not fraud” for every transaction gets 99% accuracy when fraud prevalence is 1%. That number is correct, but it answers only how often the decisions match labels. It does not tell us whether fraud was detected, whether analysts can handle the alert queue, or whether probabilities are trustworthy enough for expected-cost calculations.

Metric literacy starts with the decision and its consequences. We will derive confusion-matrix metrics, construct ROC and precision–recall curves, distinguish average precision from trapezoidal area, select a threshold on validation data, and show a concrete ranking with ROC-AUC 0.99 but average precision about 0.295. No single metric is inherently honest while all the others “lie”; they answer different questions.

Prerequisites

Know binary probabilities and thresholds, arrays, and the training/validation/test split. Declare which label is positive before computing anything. Scores can be probabilities or unbounded decision scores; ROC and PR ranking do not require calibrated probabilities. Expected-cost thresholds based directly on probability values do.

(a) Intuition · 5 min

An alerting system can admit more cases by lowering its score threshold. It catches additional true positives but may also create more false alarms. The useful operating point depends on the consequences of each action and available review capacity. A hospital screen, a spam quarantine and a transaction-review queue need not choose the same threshold even with equally accurate rankings.

A perfect ranking can achieve both precision and recall of one, so there is no theorem that both can never be maximized. For an imperfect fixed ranking, the achievable operating points form a curve. Changing the model or features can alter that curve; changing only a threshold chooses a point on it.

The two questions every metric answers

Precision: among the examples called positive, how many were actually positive? Recall: among all actual positives, how many were caught? These denominators differ. “99.9% precision” means one false alarm per thousand alerts, not a false-positive rate of one per thousand actual negatives.

F1 is a symmetric harmonic summary of precision and recall at a selected operating point. ROC-AUC is a pairwise ranking summary. Average precision emphasizes precisions as recall increases and depends on prevalence. Expected cost and precision-at-capacity translate model outputs into a particular operational decision. Report both a ranking summary and relevant operating-point counts when both questions matter.

A quick history so you know why we have five metrics

The historical motivation that matters here is the distinction between detection and retrieval. ROC represents the true-positive/false-positive rates of a thresholded detector, while precision–recall represents how much relevant material is retrieved and how much returned material is relevant. F-measures compress the latter trade-off into one scalar. These are different evaluation perspectives, not a chronology in which the newest metric invalidates the older ones.

The scikit-learn metrics guide documents their definitions, averaging conventions and limitations. The lesson does not rely on unverified stories about a particular leaderboard switching metrics or a company's user-satisfaction decline.

The confusion matrix — the single most important diagram in classification

With rows representing true labels and columns representing predictions, scikit-learn's binary matrix with labels=[0,1] is

For $TP=8,FP=2,FN=4,TN=86$, precision is $8/10=0.8$, recall is $8/12=2/3$, accuracy is $94/100=0.94$, specificity is $86/88\approx0.9773$ and FPR is $2/88\approx0.0227$. F1 can be computed without dividing by precision or recall:

More generally,

Larger beta increases the relative emphasis on recall, but beta is not automatically an exact currency cost ratio. F1 ignores TN and is a nonlinear function of counts; optimizing it is not the same as minimizing $C_{FP}FP+C_{FN}FN$ with fixed costs.

If there are no predicted positives, precision has denominator zero. If there are no actual positives, recall is undefined. A reporting policy such as zero_division=0 makes output deterministic but does not make an undefined fraction empirically meaningful. Always show class support. AUC also requires both classes for its usual binary interpretation.

A confusion matrix is sufficient for these thresholded metrics, not every metric: log loss and Brier score need probabilities, while ranking curves need scores across thresholds. Two models with the same confusion matrix at 0.5 can have very different AUC or calibration.

How ROC and PR curves are built

Sort scores descending and move a threshold through their distinct values. Predict positive at score >= threshold, adding tied scores together. At each point count TP, FP, FN and TN. ROC plots FPR versus TPR (recall); PR plots recall versus precision. Scikit-learn's PR arrays include a terminal (precision=1, recall=0) plotting endpoint with no corresponding threshold. Use precision[:-1], recall[:-1] with the threshold array.

ROC-AUC has the pairwise interpretation

It is 0.5 for an uninformative constant score, 1 for perfect strict separation, and can be below 0.5 for reversed rankings. Supplying hard labels instead of scores produces a coarse ranking statistic, not the full score-based curve you intended.

Average precision uses the non-interpolated step summary

when thresholds are traversed so recall increases. Scikit-learn returns recalls in decreasing order, so an equivalent computation is sum((rec[:-1]-rec[1:])*prec[:-1]). Trapezoidal PR area linearly interpolates between points and generally differs. Name which statistic you report; average_precision_score is not simply another API spelling for every possible “PR-AUC.”

Worked ranking. With descending labels [1,0,1,0] and distinct scores, positive ranks are 1 and 3. AP is $(1+2/3)/2=5/6\approx0.8333$. Three of four positive–negative pairs are correctly ordered, so ROC-AUC is 0.75. A constant score gives AP equal to prevalence in the standard grouped-threshold calculation. A finite random ordering need not give exactly the prevalence; it is a no-skill reference, not a lower bound—an adversarial ranking can score worse.

When to use which curve

ROC remains meaningful with imbalance: it measures discrimination through class-conditional score distributions. If those distributions stay fixed while only prevalence changes, ROC is unchanged in the population. That is a conditional statement; changing the actual sampled cases, subgroups or distribution can change it. A high ROC-AUC need not imply acceptable precision at the only threshold you can operate.

At prevalence $\pi$, TPR $r$ and FPR $f$, Bayes' rule gives

With $r=0.8,f=0.01$, prevalence 0.5 gives precision about 0.9877; prevalence 0.001 gives about 0.0741. The false-positive rate did not “inflate AUC”; the same conditional rates yield a much less useful alert queue in a rarer-positive population. This calculation explains why a PR curve and workload counts are informative alongside ROC.

Use AP/PR when positive retrieval quality matters, but compare on the same population or account for changing prevalence. A fall in AP over time can reflect a different base rate, a worse ranking, or both. Use cost or precision/recall at a fixed review capacity when that is the real decision. An overall area can hide weak performance in a narrow operating region even when it is high.

The threshold–metric trade-off

Lowering a threshold makes the predicted-positive set grow. Recall cannot fall when actual-positive support is nonzero. Precision can rise or fall: admitting one more true positive after a false alarm can improve it. For descending labels [0,1,1], precision moves from 0 to 1/2 to 2/3 as recall increases. A “better than random” AUC does not guarantee local monotonic precision.

For calibrated deployment probabilities with costs only on errors, choose positive when $C_{FP}(1-p)\le C_{FN}p$, giving $t=C_{FP}/(C_{FP}+C_{FN})$. Equal error costs produce 0.5 regardless of class balance; F1 is not the equal-cost rule. If reviewing every alert costs money or interventions only partly prevent loss, write those action costs explicitly instead.

For a target recall, inspect validation thresholds meeting it and choose the one with best precision (or lowest workload under another declared rule). Reaching 95% recall on a small validation sample does not guarantee a 95% deployment SLA. Estimate uncertainty, preserve representative prevalence and evaluate relevant subgroups and time periods. Capacity-constrained precision@K is the number of positives among the top K divided by K; handle score ties with an explicit stable or randomized policy.

Multiclass reporting without hiding the minority class

Per-class precision/recall/F1 usually treat each class as one-versus-rest. Macro averaging gives each class equal weight; support-weighted averaging weights by true class support; micro averaging pools TP/FP/FN counts first. In ordinary single-label multiclass evaluation over all labels, micro precision/recall/F1 equal accuracy. Weighted F1 is not simply F1 computed from weighted precision and recall. Report class support and individual class failures alongside averages when rare classes matter.

(c) Hands-on · 25 min

The complete lab computes exact fixtures and two bounded classifier experiments. Breast-cancer labels are recoded so malignant is positive. The synthetic rare-positive generator disables label flipping; otherwise default label noise can dominate a nominal 0.1% positive rate. Neither experiment is a fraud or medical benchmark.

import numpy as np
from sklearn.datasets import load_breast_cancer, make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
    f1_score, accuracy_score, roc_auc_score, average_precision_score,
    precision_recall_curve, roc_curve, auc, classification_report)
 
 
def report(name, y, score, threshold):
    pred = score >= threshold
    tn, fp, fn, tp = confusion_matrix(y, pred, labels=[0,1]).ravel()
    print(name, 'threshold', threshold, 'prevalence', y.mean(),
          'TN/FP/FN/TP', (tn,fp,fn,tp))
    print('accuracy/P/R/F1', accuracy_score(y,pred),
          precision_score(y,pred,zero_division=0), recall_score(y,pred,zero_division=0),
          f1_score(y,pred,zero_division=0))
    print('ROC-AUC/AP', roc_auc_score(y,score), average_precision_score(y,score))
 
 
def tune_for_recall(y, score, target=.95):
    if not 0 < target <= 1 or np.sum(y==1) == 0:
        raise ValueError('positive support and target in (0,1] required')
    pre, rec, thresholds = precision_recall_curve(y, score)
    indices = np.flatnonzero(rec[:-1] >= target)
    if len(indices) == 0:
        raise ValueError('target not feasible on validation')
    chosen = indices[np.argmax(pre[indices])]
    return float(thresholds[chosen])
 
 
def experiment(name, X, y):
    Xdev,Xtest,ydev,ytest = train_test_split(X,y,test_size=.2,stratify=y,random_state=42)
    Xtr,Xval,ytr,yval = train_test_split(Xdev,ydev,test_size=.25,stratify=ydev,random_state=43)
    model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1500))
    model.fit(Xtr,ytr)
    val = model.predict_proba(Xval)[:,1]
    threshold = tune_for_recall(yval,val)
    assert recall_score(yval, val >= threshold) >= .95
    # Frozen decisions: final test does not choose the threshold.
    score = model.predict_proba(Xtest)[:,1]
    report(name+' default',ytest,score,.5)
    report(name+' validation-tuned',ytest,score,threshold)
    pre,rec,thr = precision_recall_curve(ytest,score)
    fpr,tpr,_ = roc_curve(ytest,score)
    assert len(thr)+1 == len(pre) == len(rec)
    assert np.isclose(np.sum((rec[:-1]-rec[1:])*pre[:-1]), average_precision_score(ytest,score))
    print('trapezoidal PR area (NOT AP)', auc(rec,pre))
    print('ROC first points', list(zip(fpr,tpr))[:5])
    print('PR first points', list(zip(rec,pre))[:5])
    print(classification_report(ytest,score>=threshold,zero_division=0))
 
 
# Exact confusion-matrix fixture: TP8, FP2, FN4, TN86.
y = np.r_[np.ones(12), np.zeros(88)]
pred = np.r_[np.ones(8),np.zeros(4),np.ones(2),np.zeros(86)]
assert tuple(confusion_matrix(y,pred,labels=[0,1]).ravel()) == (86,2,4,8)
assert np.isclose(precision_score(y,pred), .8)
assert np.isclose(recall_score(y,pred), 2/3)
assert np.isclose(f1_score(y,pred), 8/11)
rank_y = np.array([1,0,1,0]); rank_s = np.array([.9,.8,.7,.6])
assert np.isclose(roc_auc_score(rank_y,rank_s), .75)
assert np.isclose(average_precision_score(rank_y,rank_s), 5/6)
 
# High ROC, modest AP: twelve negatives, then ten positives, then 1188 negatives.
rare_y = np.r_[np.zeros(12),np.ones(10),np.zeros(1188)]
rare_score = np.arange(len(rare_y),0,-1)
assert np.isclose(roc_auc_score(rare_y,rare_score), .99)
expected_ap = np.mean(np.arange(1,11)/(12+np.arange(1,11)))
assert np.isclose(average_precision_score(rare_y,rare_score),expected_ap)
print('exact rare ranking ROC/AP', .99, expected_ap)
assert np.isclose(roc_auc_score([0,1],[.5,.5]), .5)
 
X, benign = load_breast_cancer(return_X_y=True)
experiment('bundled breast cancer', X, 1-benign)
X,y = make_classification(n_samples=4000,n_features=12,n_informative=5,
    weights=[.99,.01],flip_y=0,random_state=42)
experiment('synthetic rare positives', X,y)

Anatomy of the script

confusion_matrix(..., labels=[0,1]) fixes a 2×2 orientation even if predictions contain one class. Each experiment retains both true classes through stratification, allowing binary AUC. A real streaming batch with no positives should report that limitation rather than forcing a misleading score.

The scaler and classifier are a pipeline fitted on training only. Threshold selection uses validation scores with >=, matching the PR API convention. The terminal plotting point is excluded. Test recall is measured, not asserted to meet the validation target: that would confuse selection with a population guarantee.

The fixture uses exact counts independent of model training, while the two model experiments print measured results. AP is checked against the recall-increment calculation, and trapezoidal PR area is labeled separately. To draw the curves, plot the returned FPR/TPR pairs for ROC and recall/precision pairs for PR; annotate the chosen test operating point, prevalence and class counts. An area alone loses that operational context.

The rare-ranking fixture has ten positives and 1,200 negatives. Each positive is outranked by exactly twelve negatives, so there are 120 misordered pairs out of 12,000 and ROC-AUC is $1-120/12000=0.99$. The precisions at positive ranks are $i/(12+i)$ for $i=1,\ldots,10$, giving AP $0.2948769$. This is a concrete high-AUC/low-precision example, not the claim that a nearly random classifier can somehow obtain 0.99 AUC solely through imbalance.

(d) Production reality · 15 min

Choose the metric from the decision

For a review team processing 100 transactions daily, precision@100 estimates how many reviews find positives, while recall@100 estimates what fraction of observed positives are captured. Add costs and delayed-label availability. If scores tie at the capacity boundary, a scalar threshold may admit more than 100 cases; ranking plus a tie policy implements the actual capacity rule.

For a recall requirement, report the number of positives behind the estimate and the cost in false alarms. Binomial uncertainty calculations assume an appropriate sampling model; correlated cases may need grouped resampling. A observed 19/20 recall of 95% is not persuasive evidence that the true rate exceeds 95%. Validation selection further complicates interval interpretation, so use independent evaluation.

Do not rebalance a final test set and interpret its precision as deployment precision without reweighting or an explicit prevalence adjustment. Class-conditional metrics such as ROC may be useful under case-control sampling, but real population shifts can affect the conditional score distributions too. A threshold and model should be monitored together with prevalence, volume, missing labels and subgroup behavior.

Probability calibration is a separate axis. A strictly increasing score transform preserves rank-based AUC/AP while changing probability values. Log loss and Brier score assess probability quality but also reflect discrimination and outcome uncertainty; a lower Brier score alone is not a pure calibration certificate. Use reliability information when downstream decisions multiply probabilities by costs.

Rare-event exercise with controlled prevalence

Changing weights in a synthetic generator and refitting changes both the sample and fitted model, so it is not a controlled proof that ROC stays fixed while AP changes. A cleaner prevalence thought experiment holds class-conditional rates fixed and uses the displayed Bayes formula. For an empirical comparison, sample from fixed class-conditional score pools, report finite-sample variability, and keep the positive-label definition unchanged. Disable or explicitly model label noise; otherwise an intended 0.1% event can be overwhelmed by random class flips.

Where this shows up in the rest of the plan

Class weighting and resampling in imbalanced learning are evaluated through these same distinctions. Model-selection CV aggregates a declared scorer, but may average fold metrics differently from computing one metric on all pooled out-of-fold predictions. Feature engineering is useful only if it improves a relevant metric under the correct information boundary. Monitoring thresholds must account for label delay and distribution changes rather than using an arbitrary “five-point AP drop” as a universal page rule.

(e) Recall + stretch · 10 min

1. 990 negatives, 10 positives, all-negative predictions? Answer: accuracy 0.99, recall 0, precision undefined (reported 0 under the lab convention), F1 0.
2. Spam precision and recall? Answer: precision is the fraction of quarantined messages actually spam; recall is the fraction of all spam quarantined. Their consequences differ.
3. Raise threshold from 0.5 to 0.9? Answer: predicted positives shrink and recall cannot increase; precision is not guaranteed to rise on finite data.
4. Why add PR metrics for rare events? Answer: precision reflects prevalence and false-alarm workload, which a class-conditional ROC summary does not specify.
5. One comparison number? Answer: AP can be useful on a fixed rare-positive population, but name prevalence and follow with the relevant cost/capacity operating point. No summary replaces the decision.
6. Does F1 mean equal error costs? Answer: no; it symmetrically combines precision and recall, ignores TN and is nonlinear. Use explicit expected cost for fixed currency costs.
7. What does the confusion matrix miss? Answer: probability calibration and the full ranking. It summarizes one threshold's counts.
8. Validation recall 0.62, target 0.95? Answer: inspect lower validation thresholds first and measure precision/workload, then evaluate the frozen feasible choice independently. A validation target is not a guaranteed SLA.
9. Precision@K? Answer: positives among the top K divided by K, with a tie rule; meaningful when review or display capacity is K.
10. Stretch: perfect separation and the rare-ranking fixture. Answer: with both classes present and every positive strictly above every negative, all positive–negative pairs are correctly ordered (AUC 1), and every recall increment occurs at precision 1 (AP 1). The explicit twelve-negatives/ten-positives/1,188-negatives ordering gives AUC 0.99 and AP about 0.295; the top of the ranking still creates many false alarms relative to the ten positives.

Explain-out-loud test

Reconstruct all four cells for the 8/2/4/86 example, derive precision from prevalence/TPR/FPR, and explain why AP and trapezoidal PR area differ. Then say which split selected the threshold and which population the final precision describes.

What comes next

Continue with Feature Engineering, keeping the evaluation objective fixed while changing the representation and testing for leakage.

Sources and execution boundary

Checked 2026-10-07: scikit-learn 1.7 classification metrics, precision–recall curve API, average precision API, and calibration. The CPU run checks exact arithmetic, ranking fixtures and small pipeline fits. It does not validate a hospital screen, a fraud SLA, confidence intervals under arbitrary dependence, or performance at a real production prevalence.