S020 · Calculus II — Gradients & Gradient Descent from Scratch

Module M02: Math Foundations · Session 20 of 130 · Track: Math ⛰️

What you'll be able to do after this session

• Explain Calculus II — Gradients & Gradient Descent from Scratch to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S019 — Calculus I — Derivatives & Chain Rule

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown — Gradient descent, how neural networks learn — the video that makes the entire idea click.
• 🎥 Deep dive (30–60 min) · Andrej Karpathy — Building micrograd from scratch — 2 hours; you'll write gradient descent by hand.
• 🎥 Hands-on demo (10–20 min) · StatQuest — Gradient Descent, Step-by-Step — a linear-regression fit worked out with pencil.
• 📖 Canonical article · Sebastian Ruder — An overview of gradient descent optimization algorithms — the canonical map of SGD, Momentum, Adam, RMSProp.
• 📖 Book chapter / tutorial · Deep Learning Book, Chapter 4 — Numerical Computation — gradient-based optimisation from the ground up.
• 📖 Engineering blog · distill.pub — Why Momentum Really Works — interactive, gorgeous, definitive on momentum.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The one optimisation algorithm behind all of ML.

Imagine you're blindfolded on a hillside and want to reach the lowest point. You can't see far, but you can feel which direction the ground slopes downward under your feet. Take a small step downhill. Feel again. Step again. Repeat. Eventually you settle into a valley. That's gradient descent — the entire mental model.

For a function f(x, y, z, …) of many variables, the gradient ∇f is a vector whose components are the partial derivatives — [∂f/∂x, ∂f/∂y, ∂f/∂z, …]. It points in the direction of steepest ascent. To descend, you go in the negative direction: params ← params - η · ∇f(params), where η (eta) is the learning rate. That single line, iterated, is how every neural network in the world learns.

The forever-gotcha: the learning rate is the single most important knob. Too small and training takes forever (or gets stuck). Too big and you overshoot the valley and bounce, or diverge to infinity. There is no universally correct value — it depends on the loss surface. Modern optimisers (Adam, AdamW) automatically adapt it per parameter, which is why they've won.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The full algorithm in one loop:

Gradient = vector of partial derivatives. For f(x, y) = x² + 3y²:

• ∂f/∂x = 2x, ∂f/∂y = 6y
• ∇f = [2x, 6y]
• At point (3, 1), ∇f = [6, 6] — direction of steepest ascent.
• To descend: [3, 1] - 0.1 · [6, 6] = [2.4, 0.4]. One step closer to the minimum at (0, 0).

Worked example — fit a line to data by gradient descent:

Model: ŷ = w·x + b. Loss (mean squared error): L(w, b) = (1/N) Σ (y - (wx + b))².

Gradients (partials, from chain rule):

• ∂L/∂w = -(2/N) Σ x · (y - (wx + b))
• ∂L/∂b = -(2/N) Σ (y - (wx + b))

Iterate w ← w - η · ∂L/∂w, b ← b - η · ∂L/∂b. After a few thousand steps, w and b converge to what numpy.polyfit would give — but you built it by hand.

Learning-rate zoo:

Variants you'll hear about:

• Batch GD — gradient over the full dataset every step. Accurate but slow on big data.
• Stochastic GD (SGD) — gradient from one random sample per step. Noisy but fast; noise sometimes helps escape local minima.
• Mini-batch SGD — best of both; batch of 32–1024 samples. What everyone actually uses.
• Momentum — accumulate a running average of past gradients so you "roll through" small bumps. Velocity + friction analogy.
• Adam (2015) — momentum + per-parameter adaptive learning rates. The default optimiser of the 2020s.

Local vs global minima: GD only guarantees a local minimum. In high-dim non-convex surfaces (like neural nets), saddle points and flat regions are the bigger problem — but most local minima have very similar loss to the global one, which is one of the deep-learning surprises of the last decade.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

"""Gradient descent from scratch: fit a straight line to noisy data."""
import numpy as np
 
rng = np.random.default_rng(42)
 
# ---- 1. Fake data with a known ground-truth line -------------------
true_w, true_b = 2.5, -1.0
N = 200
X = rng.uniform(-3, 3, size=N)
y = true_w * X + true_b + rng.normal(0, 0.5, size=N)
 
# ---- 2. Loss and gradients (by hand) -------------------------------
def mse(w, b):
    return float(np.mean((y - (w * X + b)) ** 2))
 
def grad(w, b):
    err = y - (w * X + b)                # (N,)
    dw = -2.0 * np.mean(X * err)
    db = -2.0 * np.mean(err)
    return dw, db
 
# ---- 3. Gradient descent -------------------------------------------
def train(lr, steps=500, log_every=100):
    w, b = 0.0, 0.0
    for t in range(steps):
        dw, db = grad(w, b)
        w -= lr * dw
        b -= lr * db
        if t % log_every == 0 or t == steps - 1:
            print(f"  step {t:4d}  loss={mse(w,b):.4f}  w={w:+.3f}  b={b:+.3f}")
    return w, b
 
print(f"true: w={true_w}, b={true_b}\n")
print("== lr = 0.05 (just right) ==");  train(0.05)
print("\n== lr = 0.001 (too small)  =="); train(0.001)
print("\n== lr = 0.9 (too big)     ==");  train(0.9, steps=20, log_every=2)
 
# ---- 4. Bonus: momentum in 5 lines ---------------------------------
def train_momentum(lr=0.05, beta=0.9, steps=500):
    w, b = 0.0, 0.0
    vw, vb = 0.0, 0.0                     # velocity
    for t in range(steps):
        dw, db = grad(w, b)
        vw = beta * vw + (1 - beta) * dw
        vb = beta * vb + (1 - beta) * db
        w -= lr * vw
        b -= lr * vb
    print(f"momentum final: w={w:.3f}, b={b:.3f}, loss={mse(w,b):.4f}")
 
print("\n== momentum ==")
train_momentum()

Observe when you run:

1. With lr=0.05, loss drops from ~4.0 to ~0.25 in 500 steps; w converges to ~2.5 and b to ~-1.0 — matching the ground truth despite the noise.
2. With lr=0.001, after 500 steps w is still only ~1.0 — nowhere near converged. Too-small step, too-slow training.
3. With lr=0.9, loss blows up to NaN or millions within a few steps — overshoot, then overshoot the overshoot.
4. The gradient shrinks as you approach the minimum — that's how the algorithm "knows" to slow down.
5. Momentum reaches the same answer in noticeably fewer effective updates because it averages out gradient noise.

Try this modification: replace MSE with Huber loss (quadratic near zero, linear far away) and add three big outliers to y. Compare the fitted w under MSE vs Huber — MSE will be pulled by the outliers, Huber will resist. This is why robust losses exist.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Gradient descent (specifically mini-batch SGD with Adam) is the reason ChatGPT, DALL-E, AlphaFold, and Tesla's autopilot exist. Training GPT-4 involved trillions of parameter updates across tens of thousands of GPUs, all coordinated through variants of this exact algorithm. The optimiser is often the only thing standing between "this architecture works" and "this architecture fails" — the AdamW paper (2017) alone unlocked large-scale transformer training.

What top teams actually do in production:

• Learning-rate schedules — start high, decay over training (cosine, warmup + linear decay, 1/√step). A constant LR is almost never optimal. Every LLM paper reports its schedule.
• Gradient clipping — torch.nn.utils.clip_grad_norm_(params, max_norm=1.0) prevents one bad batch from wrecking your weights. Standard for RNNs and LLMs.
• Warmup — for the first ~1000 steps, ramp LR from 0 to target linearly. Prevents early divergence when weights are random.
• Mixed precision (FP16/BF16 forward, FP32 master weights) — 2× throughput, half the memory. Requires loss scaling to prevent underflow.
• Distributed training — DDP, FSDP, ZeRO. Gradients are averaged across GPUs each step (all-reduce). Network becomes the bottleneck at scale.

Common bugs:

• Loss goes to NaN: usually LR too high, unclipped gradients, or log(0)/sqrt(neg) in the forward pass. Enable detect_anomaly and lower LR.
• Training loss decreases, val loss increases — classic overfitting. Add weight decay, dropout, more data, or early stopping.
• Loss oscillates without decreasing — LR too high, or batch size too small (noisy gradient). Halve LR or double batch.
• Model doesn't learn at all — dead ReLUs (all activations = 0), bad init, or an accidentally-frozen layer (check requires_grad).
• "Wrong" convergence — often a bug in the loss (predicting the wrong target, wrong reduction), not the optimiser. Print a single batch's loss by hand.

At Microsoft on the OneNote Copilot pipelines, we don't train foundation models — but every retrieval index we build tunes hyperparameters via Bayesian optimisation, which internally uses gradient-based methods over a surrogate model. Understanding gradient descent lets you read any optimisation paper.

Beyond gradient descent: for non-differentiable problems (hyperparameter search over discrete options, prompt tuning against black-box APIs, RL with sparse rewards) we use gradient-free methods — random search, Bayesian optimisation, evolutionary strategies, RLHF's PPO. Knowing when GD doesn't apply is as important as knowing when it does.

(e) Quiz + exercise · 10 min

1. Write the one-line update rule for gradient descent. What does each symbol mean?
2. What is the gradient of f(x, y) = 2x² + xy + y² at the point (1, 1)?
3. Describe three symptoms of a learning rate that is too high, and three of one that is too low.
4. What's the difference between Batch GD, Stochastic GD, and Mini-batch SGD? Which is used in practice and why?
5. What does momentum add over plain SGD? Give an analogy from physics.

Stretch (connects to S019 Chain Rule): for the simple neural net y = sigmoid(w·x + b) with MSE loss L = (y - t)², derive ∂L/∂w step by step using the chain rule. You should get ∂L/∂w = 2(y - t) · y(1 - y) · x. Now implement it and confirm PyTorch autograd gives the same number.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Calculus II — Gradients & Gradient Descent from Scratch? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S021): Probability — Random Variables, Distributions, Expectation
• Previous (S019): Calculus I — Derivatives & Chain Rule
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.