Feature Engineering — Encoding, Scaling, Missing

Why this session exists

The same information can be easy or difficult for a model to use depending on its representation. An hour encoded as an integer makes 23 and 0 look far apart to a Euclidean-distance model; a sine/cosine pair puts them next to each other on a circle. A string category needs a representation appropriate to the estimator. Missing values require a policy, not an accidental conversion to zero.

Feature engineering combines those representation choices with an information constraint: every feature and fitted preprocessing statistic must be available at prediction time. A pipeline helps enforce the fitting boundary, but it cannot detect that a “final outcome” column contains the future. We will compare encoders, scalers and imputers, derive target-encoding leakage, build and round-trip a fitted pipeline, and test its state on held-out rows.

Prerequisites

Know NumPy/pandas indexing, logistic regression, evaluation metrics, and why model selection belongs on development data. Inputs below are dataframes with explicit numeric and categorical columns. The categorical branch normalizes values to strings with np.nan for missingness; mixed numeric/string categories in one column are not assumed safe.

(a) Intuition · 5 min

A kitchen's preparation station turns inconsistent ingredients into labeled, usable components before service. The ML equivalent is a fitted transformation that takes raw columns to a stable numeric schema. The important difference is that the preparation recipe is learned only from training data, then reused unchanged at inference.

For a linear classifier, an interaction such as amount divided by recent average amount may expose a simple signal that raw amount alone does not. A tree can approximate ratios using rectangles but does not perform division at a split. Engineered features are therefore often about making a relationship economical to represent, not introducing information absent from the raw inputs.

The three problems feature engineering solves

1. Representation: encode nominal/ordinal categories and useful relationships without silently imposing inappropriate geometry.
2. Scale: choose units that make distances, penalties and optimization behave as intended.
3. Completeness: distinguish missing, zero, unknown and not-applicable, while preserving relevant missingness information.

A fourth discipline cuts across all three: availability. A perfectly scaled and encoded future outcome is still leakage. Write the prediction timestamp and feature-ready timestamp next to each feature definition before evaluating a model.

A quick history so you know why the world looks like this

The current APIs embody a progression from individual preprocessing functions to composable fitted estimators. ColumnTransformer applies different transformations to heterogeneous columns, and a Pipeline joins those transformations to the estimator so CV refits them together. Scikit-learn's TargetEncoder, introduced in 1.3, adds an internal cross-fitting distinction between training-row encoding and later transformation.

These documented capabilities replace an unsupported timeline about when statisticians first invented encodings. The enduring distinction is between stateless transformations such as a fixed hash and fitted state such as category vocabularies, medians, means, selected columns or target statistics. One-hot encoders do have learned vocabularies to version; they are not inherently free of a fit boundary.

The end-to-end feature pipeline

In each fold, fit learns state and transform applies it. The outer CV estimator must include the preprocessing; transforming the whole dataset once before cross-validation violates that boundary. In the final workflow, select settings from development folds, refit the chosen pipeline on development data, and evaluate the untouched test population.

A pipeline prevents some accidental preprocessing leakage when used correctly. It does not automatically fix group leakage, time leakage, supervised features computed before the pipeline, or an estimator that performs inner CV after upstream preprocessing was fitted once. The exact nesting matters.

The encoding decision tree

One-hot encoding. For low or manageable cardinality, create one indicator per category. Nominal labels do not acquire a false order, and each coefficient is interpretable relative to the full representation. With an intercept, all category indicators are linearly dependent; regularization or dropping a reference category handles identifiability, with different interpretive consequences. Sparse output makes many-category one-hot feasible in some workloads; “more than 20 categories is impossible” is not a meaningful cutoff. Estimate actual matrix and model memory.

Ordinal encoding. Use an explicitly ordered category list for low/medium/high or other genuine order. In a linear model, integer codes also assume equal-sized steps unless transformed further. For an ordinary numeric threshold tree, arbitrary codes for nominal categories impose contiguous-code splits: swapping codes can change the partitions available at a shallow depth. Trees do care about order, even when they do not use Euclidean distances. Native categorical split support is different from passing arbitrary integers as ordinary numeric values.

Target encoding. Replace a category with a smoothed target statistic. It can keep dimensions small, but rare groups can leak their own labels and estimates can be unstable. Use cross-fitted training encodings, an outer split boundary, smoothing and a fallback for unseen categories. Binary/continuous targets and multiclass targets can produce different output dimensionality; “always one output column” is not a general API promise.

Hashing. Map tokens into a fixed number of buckets, commonly with signs to reduce collision bias. It can be stateless with respect to a vocabulary and supports streaming/high-cardinality inputs. Collisions trade bounded memory for information loss; the hash seed/function, token construction and dimension must remain stable. It does not prove an identifier has useful generalizable signal.

Frequency grouping and embeddings. Frequency encoding or grouping rare categories can be useful baselines, but counts and rare-category rules are learned on training folds, not globally. A neural embedding learns vectors jointly with a task; it requires a strategy for rare/unseen IDs and enough data to estimate useful representations. Similar categories do not automatically end up near each other in a causally meaningful way, and adding an embedding stack has training/serving costs.

For ten million mostly unique user IDs, first ask whether the validation split tests new users or returning users. Hashing, target encoding and embeddings can all fail to generalize if the feature only memorizes identity. One-hot is not logically forbidden, but storing and estimating that many dimensions may be impractical; a bounded representation or dropping the ID may be preferable.

The scaler zoo

StandardScaler computes $z=(x-\mu_{train})/s_{train}$ using training mean and population-standard-deviation convention (ddof=0). It does not require Gaussian inputs and does not make a skewed distribution Gaussian. Outliers can strongly influence mean and scale. Training columns with variation become mean zero/unit variance; a constant column cannot have unit variance and is handled without division by zero. A three-row display or test subset need not have mean zero or unit variance.

MinMaxScaler applies an affine transformation based on training extrema. New values can fall outside the fitted output interval unless clipping is configured; it does not guarantee every future value is in [0,1]. Extreme observations compress the remaining range.

RobustScaler uses median and quantile-range statistics, reducing sensitivity of those fitted statistics to extreme observations. It does not remove outliers, prove robustness of the entire model, or always win on long-tailed features. Log transforms may be appropriate for positive skewed quantities, but must have a defined policy for zeros/negatives and a reason grounded in the signal.

Scale-sensitive methods include distance-based models, regularized linear models and many gradient-based optimizers. Unpenalized OLS predictions can be invariant to invertible rescaling in exact arithmetic; standardization can still improve numerical conditioning. Threshold trees generally do not require scaling because observed ordering determines candidate partitions, with finite-precision, nonlinear midpoint and histogram qualifications. Binary indicators can be scaled intentionally; it is not categorically prohibited. Centering a sparse matrix can destroy sparsity, so use a sparse-compatible policy.

Missing-value imputation strategies

• Median/mean: a simple numeric fallback learned from training. It changes variance/correlation and understates uncertainty; median is less sensitive to extreme values than mean but is not automatically unbiased under missing-at-random assumptions.
• Mode: fills categorical missings with a common level. It can overrepresent that level and discard the fact that a value was absent.
• Explicit missing category: use a reserved token for categorical values, ensuring it cannot collide with a real category. Numeric imputation plus a missingness indicator retains the distinction between an observed median and an imputed one.
• KNN imputation: borrows from nearby rows; distance scale, available features, missingness pattern and compute cost matter. Neighbors must come from fitting data, not future/test observations.
• Iterative imputation: predicts missing columns from others under an iterative modeling procedure. It needs assumptions and convergence/validation checks; in scikit-learn 1.7 it is experimental and must be enabled explicitly. One imputed matrix is not a full account of missing-data uncertainty.

Some estimators can route NaNs natively. That is an estimator-specific option, not permission to skip a missingness contract. Entirely missing training columns and columns newly missing at inference need deliberate handling; add_indicator=True generally adds indicators for columns observed missing during fitting, not every possible future missingness pattern. Test those edge cases and preserve the intended schema.

Why target encoding needs cross-fitting

For category $c$ with $n_c$ training rows, naive encoding is $\bar y_c=\sum_{i:c_i=c}y_i/n_c$. Each row's own label contributes $y_i/n_c$. A unique category has $n_c=1$, so its feature equals the target. That relation cannot be reproduced for a future unlabeled row.

A simple smoothed statistic is

Here $m$ is a prior-strength parameter. Smoothing reduces but does not eliminate self-label contribution. For a training row, construct the statistic from another fold or a permitted earlier window. On a held-out validation/test row, use the fitted training mapping and a declared global fallback for unseen categories.

In scikit-learn, TargetEncoder.fit_transform(X,y) uses cross-fitting; fit(X,y).transform(X) is intentionally not equivalent. Inside a pipeline's training fit, the transformer's fit_transform provides cross-fitted training features. It still needs an outer CV boundary to estimate the complete procedure. Its built-in internal folds do not automatically honor arbitrary customer groups or temporal constraints; build an appropriate custom cross-fitting scheme when those are required. Leave-one-out alone can also retain problematic target dependence in some designs; exclusion is necessary reasoning, not a universal leakage certificate.

Features that make structure easy to represent

For cyclic hour $h$, use $(\sin(2\pi h/24),\cos(2\pi h/24))$. Hours 23 and 0 are then separated by chord length $2\sin(\pi/24)\approx0.261$, like adjacent hours elsewhere. One sine alone confounds multiple times, so the pair matters. A weekend flag can generalize weekly structure more directly than memorizing date intervals in a raw timestamp. A tree can approximate periodic patterns inside its observed range, but piecewise-constant extrapolation beyond that range is a poor way to encode repeated future weekends.

Ratios need a zero-denominator policy; log transforms need domain checks; “time since last event” needs the correct event-time and availability-time cutoff. A seven-day count at time $t$ should not include the outcome at $t+1$. Useful domain representation and information hygiene are complementary: neither substitutes for the other.

(c) Hands-on · 25 min

This complete CPU lab uses a local Titanic-shaped synthetic fixture, not Titanic outcomes. It compares numeric-only and mixed-feature pipelines on identical rows and identical folds, then tests held-out schema behavior and a trusted in-memory persistence round trip. The optional OpenML loader is included for the original Titanic exercise but is not called by the bounded check.

import pickle
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, TargetEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import roc_auc_score
 
NUM = ['age','fare','sibsp','parch']
CAT = ['sex','embarked','pclass']
 
 
def load_titanic_optional():
    # Network exercise only: not executed by the local fixture check.
    from sklearn.datasets import fetch_openml
    ds = fetch_openml('titanic', version=1, as_frame=True, parser='auto')
    frame = ds.frame
    y = frame['survived'].astype(int)
    X = frame[NUM+CAT].copy()  # excludes post-outcome boat/body columns
    for c in NUM:
        X[c] = pd.to_numeric(X[c], errors='coerce')
    for c in CAT:
        X[c] = X[c].astype(object).map(lambda v: str(v) if pd.notna(v) else np.nan)
    return X, y
 
 
def make_fixture():
    rng = np.random.default_rng(42)
    n = 240
    age = rng.uniform(16,80,n)
    fare = rng.lognormal(3, .6, n)
    sex = rng.choice(['female','male'], n)
    logits = -.03*(age-40) + .015*(fare-25) + .8*(sex=='female')
    y = rng.binomial(1, 1/(1+np.exp(-logits)))
    X = pd.DataFrame({'age':age,'fare':fare,'sibsp':rng.integers(0,4,n),
        'parch':rng.integers(0,3,n),'sex':sex,'embarked':rng.choice(['S','C','Q'],n),
        'pclass':rng.choice(['1','2','3'],n)})
    X.loc[rng.choice(n,24,replace=False),'age'] = np.nan
    X.loc[rng.choice(n,15,replace=False),'embarked'] = np.nan
    return X, pd.Series(y, index=X.index)
 
 
def pipeline(mixed=True):
    numeric = Pipeline([('impute', SimpleImputer(strategy='median', add_indicator=True)),
                        ('scale', StandardScaler())])
    categorical = Pipeline([('impute', SimpleImputer(strategy='constant', fill_value='__MISSING__')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse_output=False))])
    branches = [('num',numeric,NUM)]
    if mixed:
        branches.append(('cat',categorical,CAT))
    pre = ColumnTransformer(branches, remainder='drop', verbose_feature_names_out=True)
    return Pipeline([('pre',pre), ('model',LogisticRegression(max_iter=1500))])
 
 
X,y = make_fixture()
Xd,Xt,yd,yt = train_test_split(X,y,test_size=.25,stratify=y,random_state=42)
cv = list(StratifiedKFold(4,shuffle=True,random_state=9).split(Xd,yd))
scores = {}
for mixed in [False,True]:
    values = cross_val_score(pipeline(mixed),Xd,yd,cv=cv,scoring='roc_auc',n_jobs=1,error_score='raise')
    scores[mixed] = values.mean()
    print('mixed',mixed,'fold ROC-AUC',values,'mean',values.mean())
selected = max(scores, key=scores.get)
chosen = pipeline(selected).fit(Xd,yd)
print('selected mixed',selected,'final test ROC-AUC',roc_auc_score(yt,chosen.predict_proba(Xt)[:,1]))
 
# A separate mixed pipeline for schema tests; not selected from final-test scores.
p = pipeline(True).fit(Xd,yd)
pre = p.named_steps['pre']
Z = pre.transform(Xd)
names = pre.get_feature_names_out()
assert Z.shape[1] == len(names) and np.isfinite(Z).all()
imputer = pre.named_transformers_['num'].named_steps['impute']
assert np.isclose(imputer.statistics_[0], Xd['age'].median())
scaler = pre.named_transformers_['num'].named_steps['scale']
old_mean = scaler.mean_.copy()
probe = Xt.iloc[:2].copy()
probe['age'] = [np.nan, 1000.]
probe['embarked'] = 'NEVER_SEEN_PORT'
Zprobe = pre.transform(probe)
assert np.isfinite(Zprobe).all() and Zprobe.shape[1] == Z.shape[1]
assert np.array_equal(old_mean, scaler.mean_)  # transform must not refit
port = np.array([name.startswith('cat__embarked_') for name in names])
assert np.all(Zprobe[:,port] == 0)
print('feature names', names.tolist())
print(pd.DataFrame(Z[:3],columns=names).round(2).to_string(index=False))
 
# Only deserialize bytes just created locally; never load untrusted pickle.
artifact = pickle.dumps(p)
restored = pickle.loads(artifact)
assert np.allclose(p.predict_proba(probe), restored.predict_proba(probe))
 
# Prove TargetEncoder's training and transform paths differ on unique IDs.
ids = np.array([f'id_{i}' for i in range(40)]).reshape(-1,1)
labels = np.tile([0,1],20)
te = TargetEncoder(target_type='binary', smooth=0, cv=4, shuffle=True, random_state=7)
oof = te.fit_transform(ids,labels)
naive = te.transform(ids)
assert np.allclose(naive[:,0],labels)
assert not np.allclose(oof[:,0],labels)
assert np.allclose(te.transform([['unseen']]), [[labels.mean()]])
print('target encoding naive versus cross-fit first rows', naive[:4,0], oof[:4,0])
 
hours = np.array([23.,0.])
cyclic = np.column_stack([np.sin(2*np.pi*hours/24),np.cos(2*np.pi*hours/24)])
assert np.isclose(np.linalg.norm(cyclic[0]-cyclic[1]),2*np.sin(np.pi/24))

Anatomy of the script

The optional loader selects only predeclared columns and does not claim every other Titanic column is leakage. boat and body are excluded as post-outcome information for this prediction exercise; group structure and the historical target definition still need care before interpreting Titanic as a deployment task.

Both compared models use the same rows, target, folds and classifier family. The numeric-only baseline now imputes rather than dropping rows; comparing a complete-case baseline against a full-population mixed pipeline would confound feature engineering with a changed evaluation population. A positive score delta is measured, not guaranteed. Even with matched rows, adding categorical information is different from proving that scaling alone caused the gain.

ColumnTransformer defines branch membership and remainder='drop' avoids accidentally passing a new unreviewed field into the model. handle_unknown='ignore' creates all-zero indicators for an unseen category. That is a deliberate fallback, not universally the safest policy: an application might instead reject unknown categories or map them into an infrequent group. Unknown and missing categories are different; the lab's missing token is learned if observed in training.

Dense output is acceptable for this tiny fixture. For high cardinality, inspect sparse density and memory rather than converting to a dense dataframe by default. Feature names are recovered from the fitted transformer, not guessed from raw column names. The first three rows are an inspection sample; they are not expected to exhibit the mean/variance of the whole training set.

The state assertions are a stronger leakage-boundary demonstration than hoping an invalid pipeline produces a larger score. Introducing a huge held-out age must not alter training means or medians. A preprocessing leak can change a finite score in either direction or not at all; the violation exists even when an optimistic gap is not visible.

(d) Production reality · 15 min

Four leakage mechanisms and their distinct defenses

1. Preprocessing fit on evaluation rows: global means, medians, vocabularies, PCA or feature selection influence training. Fit learned transformations inside development folds. PCA ordinarily does not use labels, but still learns from inputs and belongs within the fit boundary.
2. Target leakage: category statistics or selected features encode validation labels or each row's own outcome. Use outer fold separation and cross-fitted/temporally valid training representations. Smoothing alone is insufficient.
3. Future information: an aggregate includes events, corrections or label availability after prediction time. Build point-in-time joins using both event and availability semantics; a timestamp filter on only one clock can still leak late-arriving data.
4. Group/duplicate contamination: the same event, person or near-duplicate observation appears across splits when deployment requires new-event/new-person generalization. Deduplicate or group-split according to the actual question. Repeated customers across time are not always invalid, but the evaluation must match whether returning-customer information is available.

A pipeline addresses the first and helps organize the second; it does not infer the right split or detect an outcome disguised as a raw column. Unseen categories can occur even in a random same-period test split, especially with a long tail, so explicitly test them rather than claiming offline evaluation cannot expose them.

Training/serving skew and a single artifact

Serialize preprocessing and the model together so inference reuses the training vocabulary, imputation statistics and ordering. Record library versions, raw schema, allowed categories/ranges, output feature names, prediction timestamp semantics and model/feature version IDs. Test missing fields, unexpected fields, types, NaNs, entirely missing columns and unseen categories against the declared contract.

A pickle/joblib artifact may execute code when loaded. Only deserialize trusted artifacts; never treat model files from unknown sources as inert data. Cross-version loading is not guaranteed, and a successful in-memory round trip does not prove portability to another runtime. The persistence guide discusses these boundaries and alternatives.

A feature store can organize shared definitions, historical retrieval and online values. It does not automatically guarantee point-in-time correctness, source availability, fresh values or equality of batch/streaming implementations. Shared logic plus replay/contract tests can also address skew without a feature store. Test the same recorded events through offline and online paths and compare values with defined tolerances; do not invent a company's percentage recall loss as motivation.

Manual preprocessing stretch, done safely

A manual implementation is possible: inside each fold, fit an imputer/scaler on training numeric columns and an encoder on training categoricals; transform validation with those fitted objects; concatenate in a stored order; fit the classifier only on transformed training rows. At final fitting, persist all those objects and the column order together.

Calling get_dummies independently on training and test can produce mismatched columns. A safer manual version fits the training schema and reindexes test indicators to that schema, handling unknown categories explicitly. It is not intrinsically invalid to write manual code; it is easier to omit or reorder a step. A pipeline reduces that bookkeeping surface, rather than making all possible leakage impossible.

Where this shows up in the rest of the plan

Imbalanced learning must resample within the training portion of each fold, never the final test population. Model-selection CV must enclose preprocessing. Feature-store designs extend the same state/availability contract to batch and online systems. Tokenization and neural embeddings add different representation choices, but data boundaries, versioning and inference compatibility remain necessary.

(e) Recall + stretch · 10 min

1. Why is a global scaler a leak? Answer: its learned mean/variance depend on evaluation inputs. The score need not always improve, but the independent-evaluation contract is violated.
2. Ordinal encoding on nominal categories? Answer: numeric threshold trees impose an order too. Use explicit categorical support, one-hot, or a justified encoding and validate code-permutation sensitivity where relevant.
3. Unknown one-hot categories? Answer: ignore produces an all-zero block, not a learned estimate for the new category. Rejection, explicit unknown buckets or infrequent-category handling are other valid policies.
4. When skip scaling? Answer: ordinary threshold trees usually do not need it. For linear/distance/gradient methods choose scaling deliberately; a binary feature is not forbidden from scaling.
5. Ten million IDs? Answer: consider whether identity generalizes under the intended split, then evaluate memory, hashing collisions, cross-fitted target statistics or embedding cost. Dropping the feature can be the right choice.
6. Median versus missing token? Answer: median supplies a numeric fallback; a categorical token or indicator preserves missingness as a distinguishable event. Neither assumes all missings truly equal that fallback.
7. Skew defense? Answer: persist preprocessing with the model, share feature definitions, record timing/version contracts and compare offline/online results. A feature store helps organization but is not a proof of equality.
8. Why CV around the whole pipeline? Answer: each fit learns preprocessing on that fold's training rows. Multiple folds reveal split variability, but overlapping folds are not independent replicates and their standard deviation is not automatically a confidence interval.
9. Four leakage types? Answer: evaluation-statistic fitting, target leakage, future information and invalid group/duplicate overlap. Their defenses are related but not interchangeable.
10. Stretch: manual equivalent. Answer: fit all transformation state inside each fold and persist the final schema/order/objects; independently generated dummy columns or a global imputer violate that discipline. Verify state invariance and unseen-category behavior, not a promised score gap.

Explain-out-loud test

Draw where every fit happens in the nested pipeline/CV workflow. Derive the unique-category target leak and explain why fit_transform differs from fit().transform() for TargetEncoder. Then trace a new missing age and unseen port through the fitted branches without changing training state.

What comes next

Continue to Imbalanced Data and Model Selection & CV, carrying forward the same population, timestamp, preprocessing and final-test boundaries.

Sources and execution boundary

Checked 2026-10-07: scikit-learn 1.7 preprocessing, composition, imputation, TargetEncoder, common pitfalls, and persistence. The bounded CPU check uses generated rows, validates fitted state/schema and a trusted local round trip, and does not download Titanic or establish a real-data performance gain, service deployment, feature-store correctness or cross-version artifact compatibility.