S090 · Random Forest & Bagging

Module M11: Classical ML · Session 90 of 130 · Track: ML 🌲

What you'll be able to do after this session

• Explain Random Forest & Bagging to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S089 — Decision Trees — Gini, Entropy, Splits

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Random Forests — Part 1: Building — StatQuest — the trees + votes picture in one shot.
• 🎥 Deep dive (30–60 min) · Random Forests — Part 2: Missing Data & Sample Clustering — StatQuest — the extras that make RF industrial-grade.
• 🎥 Hands-on demo (10–20 min) · Random Forest in Python — Krish Naik — end-to-end sklearn with tuning.
• 📖 Canonical article · scikit-learn — Ensemble methods: Bagging & Forests — the reference and defaults.
• 📖 Book chapter / tutorial · ISLR ch.8.2 — Bagging, Random Forests, Boosting (free PDF) — the go-to textbook treatment.
• 📖 Engineering blog · Uber Engineering — Predicting bad rides with random forests — a real ops story where RF was the workhorse before deep models.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Trees + randomness = surprisingly strong.

A single decision tree is high-variance — its predictions swing wildly depending on the exact training sample. Bagging ("bootstrap aggregating") is the fix in one sentence: train many trees on different random subsets of the data, then average (regression) or majority-vote (classification) their predictions. If each tree is decent and their errors are somewhat independent, averaging cancels out the noise. The forest is far more stable than any individual tree.

Random Forest takes bagging one step further with a second randomisation: at every split, only a random subset of features is considered (typically √p for classification, p/3 for regression). This decorrelates the trees — otherwise every tree would greedily pick the same strong feature at the root and the ensemble would be redundant. Bagging + feature subsampling = Random Forest.

Two knobs matter above all:

• n_estimators (number of trees): more is better, with diminishing returns. Common: 100-500 for prototyping, 1000+ for production.
• max_features (features per split): defaults are usually fine. Lower = more decorrelation but more bias; higher = closer to individual trees but more correlated.

You almost never need to tune tree depth — RF is famously robust to overfitting because averaging kills variance. Just let each tree grow deep.

The other superpower is OOB (out-of-bag) evaluation: because each tree only sees ~63% of the data (bootstrap sampling with replacement), the other ~37% is a "free" held-out set for that tree. Average OOB predictions give you a validation score without needing a separate val set. Free cross-validation.

Random Forest was for many years the first model to try on any tabular dataset. In 2026 it's still one of the best baselines — trains fast, tunes trivially, gives you feature importance for free, and handles missing values and mixed feature types. It's only been dethroned in top Kaggle spots by gradient-boosted trees (XGBoost, LightGBM, CatBoost) — but RF is simpler, more robust, and often within a hair of the best.

Gotcha to carry forever: Random Forest reduces variance, not bias. If a single deep tree already underfits your problem, a forest of them will still underfit. Bagging is a variance killer, not a magic wand.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The full picture — bootstrap, subspace, vote:

Why bootstrap sampling naturally gives you ~37% held-out. Probability a specific row is not picked in one draw = (N-1)/N. Not picked in any of N draws = ((N-1)/N)^N → 1/e ≈ 0.368 as N → ∞. That's your OOB sample per tree, free.

Comparison — single tree vs bagged trees vs random forest:

Worked example — why decorrelation matters. Suppose 100 trees each have 70% accuracy. If they made independent errors, the majority vote would have accuracy ≈ 99% (binomial math). If they made identical errors, the vote would still be 70%. Reality is in between; the more decorrelated, the better. Random feature subsampling is what pushes trees toward the independent end.

Feature importance in RF. Two flavours: (a) Mean Decrease in Impurity — how much each feature reduces Gini across all splits (biased toward high-cardinality features), and (b) Permutation Importance — shuffle a column and see how much accuracy drops (more reliable, model-agnostic). Sklearn provides both.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Build a random forest, compare to a single tree, and use OOB scoring instead of a val set.

# pip install scikit-learn matplotlib
import numpy as np
from sklearn.datasets import fetch_california_housing, load_breast_cancer
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier, BaggingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, accuracy_score
from sklearn.inspection import permutation_importance
 
# ---------- 1. Regression: single tree vs forest ----------
X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
 
tree = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
rf   = RandomForestRegressor(n_estimators=200, random_state=0, n_jobs=-1, oob_score=True).fit(X_tr, y_tr)
 
print(f"single tree  test RMSE = {np.sqrt(mean_squared_error(y_te, tree.predict(X_te))):.3f}")
print(f"random forest test RMSE = {np.sqrt(mean_squared_error(y_te, rf.predict(X_te))):.3f}")
print(f"random forest OOB R^2   = {rf.oob_score_:.3f}   (free validation!)")
 
# ---------- 2. Effect of number of trees ----------
print(f"
{'n_est':>6} {'test RMSE':>10} {'time~':>10}")
for n in [1, 5, 25, 100, 500]:
    m = RandomForestRegressor(n_estimators=n, random_state=0, n_jobs=-1).fit(X_tr, y_tr)
    r = np.sqrt(mean_squared_error(y_te, m.predict(X_te)))
    print(f"{n:>6} {r:>10.4f}")
 
# ---------- 3. Effect of max_features ----------
p = X_tr.shape[1]
print(f"
{'max_features':>15} {'test RMSE':>10}")
for mf in [1, 3, int(np.sqrt(p)), p//3, p]:
    m = RandomForestRegressor(n_estimators=200, max_features=mf,
                              random_state=0, n_jobs=-1).fit(X_tr, y_tr)
    r = np.sqrt(mean_squared_error(y_te, m.predict(X_te)))
    print(f"{mf:>15} {r:>10.4f}")
 
# ---------- 4. Classification with OOB + permutation importance ----------
Xc, yc = load_breast_cancer(return_X_y=True)
names = load_breast_cancer().feature_names
rfc = RandomForestClassifier(n_estimators=300, random_state=0, oob_score=True,
                             n_jobs=-1).fit(Xc, yc)
print(f"
Breast cancer OOB accuracy = {rfc.oob_score_:.4f}")
 
perm = permutation_importance(rfc, Xc, yc, n_repeats=5, random_state=0, n_jobs=-1)
top = sorted(zip(names, perm.importances_mean), key=lambda t: -t[1])[:5]
print("Top-5 features by permutation importance:")
for n, s in top:
    print(f"  {n:35s} {s:.4f}")
 
# ---------- 5. Bagging with any base estimator ----------
from sklearn.linear_model import LogisticRegression
bag = BaggingClassifier(estimator=LogisticRegression(max_iter=2000, solver='liblinear'),
                        n_estimators=50, random_state=0, n_jobs=-1).fit(Xc, yc)
print(f"
Bagged Logistic Regression accuracy = {accuracy_score(yc, bag.predict(Xc)):.4f}")

Observe when you run it:

1. Single tree RMSE ~0.72; RF (200 trees) drops it to ~0.50 — that's pure variance reduction.
2. OOB R² is close to the true test score — proving the "free validation" claim.
3. Test RMSE improves rapidly from 1 → 100 trees, then plateaus. Diminishing returns are visible.
4. max_features=p (equivalent to bagged trees, no feature subsampling) is worse than √p — decorrelation matters.
5. Permutation importance is stable across runs and matches medical intuition (worst area, worst concave points).

Try this modification: compare RandomForestClassifier to sklearn.ensemble.GradientBoostingClassifier and HistGradientBoostingClassifier on the same dataset. HistGBM is what LightGBM/XGBoost do internally — usually the winner on tabular data, at the cost of more tuning.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Random Forest's enduring niche. Fast to train (parallel across trees), no scaling needed, no missing-value preprocessing, no hyperparameter tuning marathon, gives feature importance, gives calibrated probabilities. This makes it the perfect baseline: any new tabular ML project should compare its fancy model to RF, and if RF is close, ship RF.

Where RF loses to boosting. Boosted trees (XGBoost, LightGBM, CatBoost) build trees sequentially, each correcting the residual of the previous. They achieve lower bias than RF and typically 1-3% better accuracy on well-tuned tabular problems. Cost: more hyperparameters, more overfitting risk, sequential (harder to parallelise). Every Kaggle top solution since ~2015 is boosted trees.

Memory footprint. 500 trees × 30 features × 1M rows can easily be 5+ GB in memory. Production RF services either (a) shrink trees (max_depth, max_leaf_nodes), (b) use warm_start to grow incrementally, or (c) prune post-training.

Serving latency. RF prediction is O(n_trees × depth) — parallelisable but not free. If serving < 10 ms per prediction is required, either shrink the forest or switch to a linear model for the online path.

Distributed training. Spark MLlib's RandomForest, H2O, and Dask-ML all offer distributed RF. Actual industry usage: Uber, Netflix, Airbnb all had RF pipelines in Spark for years before migrating to deep models.

Feature importance is not causal. RF importance says "the model used this feature", not "this feature caused the outcome". Two correlated features split their importance ~50/50 even if only one is causal. Use SHAP values for a more principled local explanation.

Class imbalance handling. class_weight="balanced" or class_weight="balanced_subsample" re-weights each bootstrap sample. For extreme imbalance (< 1%), combine with undersampling — see imbalanced-learn's BalancedRandomForestClassifier.

Concept drift monitoring. RF trained in Jan, deployed in June — the feature distribution may have shifted (COVID, holidays, marketing pushes). Monitor input distributions and OOB error on incoming batches; retrain when either drifts too far. Every mature ML platform (Airbnb Bighead, Uber Michelangelo, Netflix Metaflow, Google Vertex) automates this.

(e) Quiz + exercise · 10 min

1. What are the two sources of randomness in a Random Forest, and what does each accomplish?
2. Why does averaging many decision trees reduce variance? Give the intuition, not the math.
3. What is out-of-bag (OOB) evaluation and why do you get it for "free"?
4. Why does Random Forest use random feature subsets at every split rather than a random subset once per tree?
5. Give two reasons boosted trees (XGBoost/LightGBM) often outperform Random Forest, and one reason you might still pick RF.

Stretch (connects to S088 — Bias-Variance): Using the bias-variance decomposition, explain why Random Forest reduces variance but not bias. Then argue why this makes RF a better fit for problems where individual trees overfit, and a worse fit for problems where individual trees underfit.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Random Forest & Bagging? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S091): Gradient Boosting — XGBoost, LightGBM
• Previous (S089): Decision Trees — Gini, Entropy, Splits
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.