Classical ML Fundamentals — Linear, Trees, Metrics
A deep-dive on Classical ML Fundamentals — part of a 36-topic evergreen learning series.
Why this session matters
Part of a 36-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.
Session 06 · ML track 🌱
Before you touch a Transformer, you should be able to explain linear regression, logistic regression, decision trees, and random forests — how they work, when they break, what their metrics mean, and how feature engineering changes results. This session grounds you in the classical ML that still runs 80% of production ML systems (fraud, ads, credit, ranking).
Pre-read (30 min before session)
Watch 1–2 of these before the deep-dive:
- Andrew Ng Machine Learning Specialization (intro)
- StatQuest Linear Regression
- StatQuest Random Forests
Then skim chapter 1-3 of ISLR.
Deep-dive (90 min)
1. Linear & logistic regression (20 min)
OLS derivation. Assumptions (linearity, homoscedasticity, no multicollinearity, normally-distributed residuals). Regularization: L1 (Lasso) for sparsity, L2 (Ridge) for stability, ElasticNet. Logistic regression as GLM with logit link. Interpretability via coefficients + odds ratios.
2. Decision trees + ensembles (25 min)
CART algorithm (Gini vs entropy). Overfitting on single trees. Bagging → Random Forests (row + column subsampling). Boosting intuition (fits residuals sequentially). Feature importance: MDI vs permutation.
3. Feature engineering (25 min)
Numerical: scaling (standard, robust, min-max), log transforms, binning. Categorical: one-hot, target encoding (with smoothing to avoid leakage), embedding lookups. Time: cyclical encoding for hour/dayofweek, lag features, rolling stats. Text: TF-IDF, hashing trick.
4. Evaluation (20 min)
Regression: MAE, RMSE, MAPE (when target is nonzero), R². Classification: precision/recall/F1, ROC-AUC vs PR-AUC (imbalanced), log-loss, calibration. Cross-validation folds. Time-series CV (walk-forward, not random).
Reading list
- An Introduction to Statistical Learning — James et al (ISLR, free PDF)
- Hands-On ML — Aurélien Géron (chapters 1-7)
- scikit-learn user guide — canonical reference
Research links
Hands-on drill
Load scikit-learn's Titanic dataset. Train logistic regression, random forest, and XGBoost. Compare accuracy, ROC-AUC, calibration. Explain which features matter and why.
Post-session checklist
- Can you explain the bias-variance tradeoff in 60 seconds?
- Can you explain when to use ROC-AUC vs PR-AUC in 60 seconds?
- Can you explain why target encoding needs smoothing in 60 seconds?
- Did you complete the hands-on drill above?
- Did you write 3 flashcards for tomorrow's recall?
- What's the one thing you'd want to revisit in the next revision session?
What's next
Session 07 continues the series. See the hub page for the full sequence and revision pattern.