S093 · Feature Engineering — Encoding, Scaling, Missing

Module M11: Classical ML · Session 93 of 130 · Track: ML 🛠️

What you'll be able to do after this session

• Explain Feature Engineering to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S084 — The ML Mental Model — Features, Labels, Train/Val/Test

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Krish Naik: Feature Engineering — All Techniques — 10-minute tour of the whole toolkit.
• 🎥 Deep dive (30–60 min) · Abhishek Thakur: Approaching (Almost) Any ML Problem — Feature Engineering — a 4× Kaggle grandmaster shows his real workflow.
• 🎥 Hands-on demo (10–20 min) · StatQuest: One-Hot, Label & Target Encoding — cleanest 10-min walkthrough of the three encodings.
• 📖 Canonical article · scikit-learn User Guide: Preprocessing data — reference for scalers, encoders, imputers; bookmark it.
• 📖 Book chapter / tutorial · Feature Engineering for ML (Zheng & Casari) — free chapter — O'Reilly's chapter 1 nails the mental model.
• 📖 Engineering blog · Uber Michelangelo — machine learning platform — how Uber industrialises features across hundreds of models.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The unsexy work that wins Kaggles.

Models don't see the world — they see a matrix of numbers. Whatever transformations you apply on the way from raw data to that matrix is your feature engineering, and it usually determines model quality more than which algorithm you pick. Andrew Ng's line: "applied machine learning is basically feature engineering."

Think of it as prepping ingredients before cooking. You wouldn't throw a whole raw onion into a pan and call it dinner — you dice, salt, and sweat it first. Feature engineering is the same: numbers on wildly different scales (age 0–100 vs income 0–1,000,000) confuse distance-based and gradient-based models, so you scale. Text categories like country = "IN" mean nothing to a matrix multiply, so you encode. Missing cells will crash most models, so you impute. And subtle domain-specific transforms (log of a skewed variable, day-of-week from a timestamp, ratios of two columns) are where the Kaggle-winning magic lives.

Gotcha to remember forever: every fit-and-transform (scaler mean/std, encoder categories, imputer median) must be learned on the training set only and then applied to validation/test. Fitting on the whole dataset leaks information from the future and inflates your metrics. Wrap everything in a sklearn Pipeline or ColumnTransformer so it's impossible to leak by accident.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The three big families

The pipeline diagram

Every arrow into Model.fit learns parameters only from the train fold. Test is transformed with the already-fitted CT — never re-fitted.

Worked example: target encoding done right

You have city with 5000 unique values. One-hot is a disaster (5000 columns). Target encoding replaces each city with the mean target value for rows in that city — powerful, but a huge leak if you use the row's own target in its own encoding.

The fix (out-of-fold target encoding): split train into 5 folds. For each fold, compute per-city means on the other 4 folds and apply to this fold. Every training row is encoded using targets it did not itself contribute to. sklearn.preprocessing.TargetEncoder (added in 1.3) does this automatically with cv=5.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# pip install scikit-learn pandas
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, TargetEncoder
from sklearn.ensemble import GradientBoostingClassifier
 
rng = np.random.default_rng(42)
n = 5000
df = pd.DataFrame({
    "age":       rng.normal(40, 12, n).clip(18, 90),
    "income":    rng.lognormal(10, 1, n),
    "city":      rng.choice([f"city_{i}" for i in range(200)], n),
    "device":    rng.choice(["ios", "android", "web", None], n, p=[.35, .4, .2, .05]),
    "signup_dt": pd.to_datetime("2024-01-01") + pd.to_timedelta(rng.integers(0, 365, n), unit="D"),
})
df.loc[rng.choice(n, 300, replace=False), "income"] = np.nan
y = (((df["age"] > 45) & (df["income"] > 20000)).astype(int).values
     ^ (rng.random(n) < 0.15).astype(int))
 
df["dow"] = df["signup_dt"].dt.dayofweek
df["month"] = df["signup_dt"].dt.month
df = df.drop(columns=["signup_dt"])
 
num_cols = ["age", "income", "dow", "month"]
low_card_cat = ["device"]
high_card_cat = ["city"]
 
Xtr, Xte, ytr, yte = train_test_split(df, y, test_size=0.2, stratify=y, random_state=42)
 
pre = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale",  StandardScaler()),
    ]), num_cols),
    ("low_cat", Pipeline([
        ("impute", SimpleImputer(strategy="constant", fill_value="MISSING")),
        ("ohe",    OneHotEncoder(handle_unknown="ignore")),
    ]), low_card_cat),
    ("high_cat", TargetEncoder(cv=5, smooth="auto"), high_card_cat),
])
 
pipe = Pipeline([("pre", pre), ("clf", GradientBoostingClassifier(random_state=42))])
pipe.fit(Xtr, ytr)
print(f"Train acc: {pipe.score(Xtr, ytr):.3f}")
print(f"Test  acc: {pipe.score(Xte, yte):.3f}")

What to observe when you run it:

1. ColumnTransformer routes each column to its own preprocessing branch — never conflates numeric with categorical.
2. TargetEncoder(cv=5) does out-of-fold encoding on train, single encoder on test — no leak.
3. OneHotEncoder(handle_unknown='ignore') prevents test-time crashes on unseen categories.
4. Median imputation is robust to the log-normal income tail; mean would be pulled up by outliers.
5. The whole thing is one pipe.fit(Xtr, ytr) — no leakage possible even if you refactor.

Try this modification: replace TargetEncoder(cv=5, ...) with OneHotEncoder(handle_unknown='ignore') on city. Observe how training memory spikes and accuracy often drops because the model can't share statistical strength across cities.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1: the timezone bug that ate a launch. A ride-sharing team engineered hour_of_day from a UTC timestamp for a demand-forecast model. In production, the service fed it local timestamps. Predictions were 5.5 hours off for India, 8 hours off for California. Rule: pick a canonical timezone (usually UTC) at the feature store level and enforce it in schemas.

War story #2: silent target leak. An analytics team added a feature avg_purchase_amount_last_90d — computed on the entire dataset before splitting. For every row, the "past 90 days" window included the row's own future purchases. Offline AUC 0.98; online 0.71. Rule: any time-window aggregate must be computed with a strict cutoff at prediction time, ideally in a feature store that snapshots by event time.

Common bugs:

• StandardScaler fit on test. Shows up as "train acc 0.7, test acc 0.9" — the tell-tale sign that test benefited from being seen during scaling.
• One-hot with drop='first' on a tree model. Trees don't need collinearity fixes; you just threw away a category.
• Feature engineering per-row in a Python loop. Use vectorised pandas or Polars.
• Different feature definitions online vs offline. The #1 cause of "great offline, garbage in prod." Fix: a shared feature library called from both places.

How top teams handle it:

• Uber Michelangelo Palette, Airbnb Zipline, Meta FBLearner, Netflix Metaflow, DoorDash Riviera all converged on the same architecture: a feature store that computes features once, versions them, and serves them consistently to offline training and online inference.
• Kaggle grandmasters spend 60–80% of competition time on feature engineering, 10% on models. That ratio tells you where the value is.

(e) Quiz + exercise · 10 min

1. Why must you fit scalers, encoders, and imputers on the training set only, and how does sklearn.Pipeline prevent leaks?
2. When would you prefer target encoding over one-hot encoding, and what's the standard trick to avoid target leakage during target encoding?
3. Name a downside of StandardScaler on data with extreme outliers, and one alternative that handles them better.
4. Give three features you could engineer from a single ISO-8601 timestamp column, and one common bug that ruins them in production.
5. Your test accuracy is higher than your training accuracy. Name two feature-engineering pitfalls that could cause this.

Stretch (connects to S084): Sketch the difference between doing feature engineering in a Jupyter notebook vs a feature store like Feast or Uber Palette. Which failure modes does each guard against, and what does a feature store add that a sklearn.Pipeline cannot?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Feature Engineering? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S094): Imbalanced Data — SMOTE, Class Weights, Thresholds
• Previous (S092): Evaluation Metrics — P/R/F1/ROC/PR/AUC
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.