Calculus II — Gradients & Gradient Descent from Scratch

Review boundary and corrected contracts

GD descent needs a suitable step size and smoothness assumptions; it need not converge globally or even reach a stationary point. Uniform minibatch gradients are unbiased under the fixed-data objective, but noise is not a universal escape/generalization guarantee. Replaced normal-equation solve with lstsq for the OLS reference.

The examples below teach the stated input contracts; fragments and deliberately incorrect counterexamples are exercises, not production implementations. Complexity counts elementary operations unless a bit-cost or allocation caveat is stated. Practice times are suggestions, not measured performance or hiring guarantees.

Learning objective: Implement gradient descent from scratch on linear regression, feel what learning rate does, and know when to reach for momentum, Adam, or a schedule.

Why this session exists

Many differentiable models, including neural networks, are trained by repeated gradient-based updates. Trees, nearest-neighbour methods and closed-form least squares are counterexamples to “all ML uses gradient descent.” A two-parameter regression teaches the update loop, not the whole of GPT training: objectives, data, architecture, numerical precision and distributed execution still matter.

You will be able to

• Compute the gradient of a multi-variable function by hand and interpret it as ‘direction of steepest ascent’.

• Implement the core vanilla GD, SGD and momentum loops, then add validation and measurements; line count is not a correctness criterion.

• Choose a learning rate that neither diverges nor stalls (and diagnose which failure you're seeing).

• Explain what Adam/AdamW, cosine schedules and gradient clipping each change, without treating them as universal defaults.

• Use a loss curve and controlled comparisons to investigate step size, gradients and numerical failures.

Prerequisites

• S019 · Derivatives & chain rule — a gradient is just a vector of partial derivatives.
• S017/S018 · Linear algebra — you'll do vector arithmetic and matrix updates.

Pre-read · skim before the session (30 min total)

Further reading: Deep Learning · Chapter 8 · Optimization

Further reading: An overview of gradient descent optimization algorithms

Further reading: Why Momentum Really Works

(a) Intuition · 5 min

You're on a foggy hillside; find the valley

Real: You're on a mountain in dense fog. You want to reach the valley but can only see the ground under your feet. Every few seconds you check which direction is most downhill and take a step that way. If the slope is steep, big step. If the ground is flat, tiny step. With a suitable step size you may approach a stationary region; the local slope alone cannot promise a valley or a global optimum. That fog is what an ML model sees. It doesn't know the shape of the loss landscape globally. It only knows the local slope. Gradient descent is the ‘check underfoot, step downhill, repeat’ strategy — repeated a million times for a million-parameter landscape.

 
    In code, the "slope in every direction" is the gradient — a vector where entry i is ∂loss/∂parameter_i. You step against it: params -= lr · gradient. The learning rate lr is your step size. Too big and you jump over the valley. Too small and progress may be slow within your budget. Adaptive methods still need a learning rate; inspect the data, objective and gradients as well.
    Every optimiser you'll ever use — SGD, momentum, RMSprop, Adam, AdamW, Lion — is a variation on this one theme.
  

The gradient in one line

What the gradient IS

• For a scalar function f(x, y, z), the gradient ∇f = [∂f/∂x, ∂f/∂y, ∂f/∂z]. A vector of partial derivatives.

• For a differentiable real scalar function in Euclidean coordinates, a nonzero ∇f gives the steepest local ascent per unit Euclidean distance; −∇f gives descent. Other norms or parameterizations change this geometry.

• At a differentiable interior local minimum, ∇f = 0. The converse fails: a saddle or maximum is also stationary. Boundary minima need not have zero gradient, and GD need not converge.

• The gradient has one coordinate per differentiable trainable parameter. Frozen parameters, sparse updates and parameter sharding change what is stored or updated in an implementation.

The single most-critical hyperparameter

Learning rate strongly affects stability and speed. Rates such as 1e-3 for Adam or 1e-2 for SGD are possible starting trials, not safe defaults for every loss scale. A high rate can diverge; a small rate can make progress too slow within the budget. Numerical faults and bad data need separate checks.

A quick history

The useful historical progression is from deterministic steepest descent (associated with Cauchy), to Robbins–Monro stochastic approximation, to Polyak-style momentum, to per-coordinate adaptive methods. These are different mechanisms, not successive guarantees of better answers. AdaGrad's 2011 paper accumulates gradient information to adapt coordinate scaling; it is not a 2012 invention. RMSprop uses a decaying squared-gradient average rather than AdaGrad's growing sum. Adam combines first and second moments; decoupled weight decay separates shrinkage from those moments. Sign-based Lion and preconditioned methods such as Shampoo illustrate that newer optimisers can change the direction too, not just a scalar learning rate. No “still the winner” or market-share claim follows from this timeline.

A hand gradient and a safe quadratic step

For f(x,y) = x² + 4y², ∇f = (2x,8y). At (1,1), the gradient is (2,8); with learning rate 0.1 the next point is (0.8,0.2), and loss drops from 5 to 0.8. The directional derivative along unit u is ∇f·u. Cauchy–Schwarz bounds it below by −‖∇f‖, attained by u = −∇f/‖∇f‖ when the gradient is nonzero.

This is a local statement. If a differentiable loss has an L-Lipschitz gradient, the descent lemma gives f(θ−ηg) ≤ f(θ)−η(1−Lη/2)‖g‖² for g=∇f(θ). Thus 0 < η < 2/L decreases this deterministic loss when g is nonzero. For the quadratic above L=8; η=0.3 makes the y coordinate multiply by −1.4 and diverge. For MSE with design matrix A, the Hessian is 2AᵀA/n: compute its largest eigenvalue to understand the stable range. Rank deficiency allows multiple minimizers; separable unregularized logistic regression need not have a finite minimizer.

(b) From gradients to parameter updates · 15 min

A gradient describes the loss at the current parameters; an optimiser uses that information to choose the next parameters. To connect the hillside intuition to training code, separate the repeated loop from its two choices: which examples supply the gradient, and how that gradient becomes a step.

The update rule, forever

Trace one trip around the diagram, starting with the current weights. The forward pass measures the loss, the backward pass computes its sensitivity to those weights, and only the update changes the weights. Keeping these jobs separate is what lets us change optimisers without changing the model.

The return arrow matters: after an update, the next gradient must be evaluated at the new parameters rather than reused unchanged. The diagram leaves the data selection open. Using every example, one example, or a small batch changes the cost and noise of that gradient estimate without changing the loop's structure.

Batch · mini-batch · stochastic — three flavours

Name: Batch GD

Tagline: Full dataset per step

• Exact gradient of the specified finite-data objective, up to numerical error

• One step per epoch

• More work per update; accumulation can avoid holding all data at once

• Useful for small problems; compare hardware throughput and convergence for larger ones

Name: SGD (stochastic)

Tagline: One sample per step

• Potentially noisy; noise can help or hinder and can be zero even at a stationary point

• Cheap per step

• Requires a learning-rate policy matched to curvature and noise

• Single-example updates are a useful reference; mini-batches offer more parallel work

Name: Mini-batch SGD

Tagline: The workhorse

• A chosen subset per step; 32–1024 are examples, not a definition or universal optimum

• Compromise: gradient noise controlled but still cheap per step

• Can exploit parallel hardware when batch shape and memory permit

• Libraries often use “SGD” for the update rule regardless of the batch size

The optimiser family tree

Batch size chooses the evidence used to estimate the gradient. Optimiser choice addresses the next box in the loop: how to turn that estimate into a parameter update. Read the progression below by tracking what each method retains from earlier steps, from no history in vanilla SGD to momentum and adaptive scaling.

Optimisers, from simple to state-of-the-art

Name: SGD (vanilla)

Desc: θ ← θ − lr · g. Simple, works surprisingly well with the right lr + schedule. Still used for computer-vision training.

Name: SGD + momentum

Desc: Track a velocity v = β·v + g, then θ ← θ − lr · v. β ≈ 0.9 is a common trial. With suitable tuning it can damp oscillations and accelerate persistent directions.

Name: AdaGrad / RMSprop

Desc: AdaGrad divides by the square root of a cumulative squared-gradient sum (plus numerical stabilization); RMSprop uses an exponentially weighted squared-gradient average. The distinction prevents the denominator from growing forever in RMSprop.

Name: Adam

Desc: Momentum + adaptive learning rate. β₁≈0.9, β₂≈0.999, lr≈1e-3. Common starting choice; this is not a measured usage statistic.

Name: AdamW

Desc: Adam with decoupled weight decay. Common in transformer training; validate it for the chosen objective and schedule.

Name: Lion (experimental)

Desc: Sign-based updates with one momentum state rather than Adam’s two moment states. This alone halves equal-dtype moment storage, not total training memory; weights, gradients, activations and implementation buffers remain.

Learning-rate diagnostics from a loss curve

Whichever update rule you choose, the loss measured on repeated trips around the loop provides feedback about the step size. Compare measurements on a consistent dataset when possible: changing mini-batches can make loss bounce even when training is progressing. Use the patterns below as diagnostic clues, then check the gradients and numerical stability before blaming the learning rate alone.

Loss goes DOWN smoothly → lr is roughly right

Desc: You're descending the loss surface. Consider slightly larger lr to see if you can converge faster.

Loss oscillates or bounces wildly → investigate lr and sampling noise

Desc: On a deterministic quadratic, alternating growth indicates an unstable rate. For minibatches also compare a fixed evaluation loss; try a smaller rate in a controlled rerun.

Loss barely moves → lr too LOW (or vanishing gradient)

Desc: You're taking micro-steps. Double the lr. If still no movement, check gradient magnitudes — may be a vanishing-gradient problem.

Loss explodes to NaN → lr WAY too high or gradient issue

Desc: Locate the first nonfinite input or intermediate. Divergence/overflow is one cause, but an invalid log or corrupt batch can fail before any optimizer update.

Loss goes down then plateaus → investigate a schedule and other limits

Desc: Try a decay schedule only after checking capacity, gradients and the noise floor; a plateau alone does not identify its cause.

The LR-finder trick (Leslie Smith 2017)

A learning-rate range test gradually increases the rate in a short disposable run and inspects when loss becomes unstable. Use log learning-rate axes; loss need not be plotted logarithmically. Save/reset model and optimiser state afterward. “A fraction of the instability rate” is a heuristic, not a 90%-success theorem or a fixed 100-step recipe.

Myth: "Gradient descent finds the minimum. If training converged, I'm at the best solution the model can reach — and if it didn't converge, I should train longer."

Gradient descent may approach stationary points under appropriate smoothness, step-size and boundedness assumptions; it can also diverge or cycle. A plateau is a diagnostic observation, not proof of optimality.

Why: Convexity makes every attained local minimum global, but does not guarantee a unique or finite minimizer, a suitable step size, or convergence from every initialization. Nonconvex problems add stationary points that are not minima. Test the exact claim rather than using a plateau as a certificate.

The archived quartic illustrates a stationary inflection, not two different minima. Its derivative is x²(4x−9): x=0 is stationary without being a local extremum; x=9/4 is the minimum. Starting negative approaches zero from below under these small steps; starting at three approaches 9/4.

f = lambda x: x**4 - 3*x**3 + 2
df = lambda x: 4*x**3 - 9*x**2
 
def descend(x, lr=0.01, steps=2000):
    for _ in range(steps):
        x -= lr * df(x)
    return x
 
print(descend(-1.0), descend(3.0))  # near 0 from below, and near 2.25
print(descend(0.0))                # exactly 0: stationary, not a minimum

Question: What is gained and lost when a small batch estimates a fixed-data gradient?

Let gᵢ(θ) be per-example gradients and ḡ their mean. At a fixed θ, uniformly drawing B examples independently with replacement gives E[g_batch]=ḡ and covariance Cov(g_batch)=Cov(gᵢ)/B, when second moments exist. For a simple uniform sample without replacement with N>1 and 1≤B≤N, multiply by (N−B)/(N−1) if Cov(gᵢ) uses denominator N. For N=B=1 the variance is zero directly. A full batch has zero sampling variance. These statements do not say every noisy update decreases loss.

Our code instead reshuffles once per epoch. Each position is marginally uniform, but later batches and the evolving θ depend on earlier batches: conditional unbiasedness at every adaptive step does not follow from the fixed-θ argument. With per-example scalar gradients [−2,4], uniform single draws average to 1; always taking the first produces −2, a biased estimate. If every per-example gradient is zero, sampling adds no noise and cannot kick the iterate off a stationary point.

Small batches can improve update throughput and sometimes exploration; large batches exploit hardware and reduce noise. A positive inner product ḡ·d makes −d locally descending only for sufficiently small steps. Basin width, parameterization and anisotropic noise matter, so there is no universal flat-minimum or generalization guarantee. Tune batch size, step size and schedule together.

Name: Blindfolded on a hillside, feeling the slope

Picture: You are on a landscape in the dark. You cannot see the terrain — you can only feel which way is downhill under your feet, and take a step. That is the entire algorithm. The gradient is the local feel of the slope; the learning rate is your stride length; the loss curve is your altitude log. Everything that goes wrong has an obvious physical reading: too big a stride and you leap across the valley and up the far side (loss diverges); too small and you take a week to descend (slow convergence); a flat plateau and you can't tell which way to go (vanishing gradient); a narrow ravine and you bounce between the walls while barely advancing along it (which momentum can mitigate by accumulating consistent directions; an unsuitable momentum/step-size combination can also destabilize it).

• Once the data/loss/gradient checks pass, investigate rates across orders of magnitude before fine tuning; the useful interval is problem-dependent.

• A NaN can come from invalid data or an undefined operation even at zero learning rate. Inspect the first nonfinite forward value/gradient, then investigate scaling, learning rate and precision; clipping a NaN does not repair it.

• Features on wildly different scales create ravines, because the loss curves far more steeply along one axis than another. Normalising inputs is not cosmetic; it reshapes the landscape into something a single global step size can handle.

• Plot training and held-out loss over time. Their trajectories reveal information hidden by a final metric, but patterns are clues rather than unique diagnoses.

Trigger: Fire this model the moment you see: loss going to NaN · a training curve that is flat from step one · validation loss rising while training loss falls · results that swing between seeds · a "just train longer" suggestion · any hyperparameter search that didn't start with the learning rate.

Decision: Which optimiser: plain SGD, SGD with momentum, or an adaptive method like Adam?

Name: Plain SGD

Gain: one hyperparameter, minimal memory (no per-parameter state), and completely predictable behaviour — what you see is the raw gradient

Cost: painfully slow in ravines, oscillating across the steep direction while creeping along the shallow one; extremely sensitive to feature scaling and to the learning rate

Pick: convex or well-conditioned problems, or when you want a baseline with no confounding optimiser dynamics

Name: SGD + momentum (+ schedule)

Gain: accumulates consistent directions and cancels oscillation, so it traverses ravines efficiently; can be competitive after tuning, without a universal generalization ranking

Cost: two coupled hyperparameters plus a schedule to design; needs more tuning effort and more patience to get right

Pick: you have the budget to tune, the training run is long, and final quality matters more than time-to-first-result — the classic choice for large vision training

Name: Adam / AdamW

Gain: adaptive coordinate scaling can help differently scaled or sparse gradients; dense Adam state still costs memory, and sparse-gradient API support depends on the implementation

Cost: two moment tensors per parameter for Adam, versus one velocity for momentum and none for vanilla SGD. Total memory also includes weights, gradients and implementation buffers. An L2 loss penalty is coupled through Adam’s moments; AdamW intentionally implements a different, decoupled shrinkage rule.

Pick: gradients are sparse or badly scaled, or you need a good result quickly without a tuning campaign — the sane default for a first run on almost anything

Verdict: AdamW is a useful starting point for many differentiable models, but it does not remove learning-rate sensitivity. Compare it with a tuned SGD baseline using the same data and validation protocol. Fix incorrect labels, gradients and loss definitions before attributing failure to optimiser choice.

(c) Hands-on · 25 min

You'll fit a straight line to noisy data three ways — analytic solution, vanilla GD, and SGD with momentum — and watch the learning-rate knob live. Save as gd_demo.py.

"""gd_demo.py — gradient descent from scratch on linear regression."""
from __future__ import annotations
import numpy as np
 
rng = np.random.default_rng(0)
 
# ── 1. Make some noisy data around y = 2x + 1 ─────────────────────────
N = 200
X = rng.uniform(-3, 3, size=N)
Y = 2.0 * X + 1.0 + rng.normal(scale=1.0, size=N)  # true slope=2, intercept=1
 
# ── 2. Least-squares reference and explicit data contract ──────────────
def check_xy(X, Y):
    if (not isinstance(X, np.ndarray) or not isinstance(Y, np.ndarray)
            or X.ndim != 1 or Y.shape != X.shape or X.size == 0
            or not np.isfinite(X).all() or not np.isfinite(Y).all()):
        raise ValueError("X and Y must be finite nonempty arrays of shape (n,)")
 
def analytic_ols(X, Y):
    check_xy(X, Y)
    A = np.column_stack([X, np.ones_like(X)])   # design matrix
    theta = np.linalg.lstsq(A, Y, rcond=None)[0]  # avoid squaring the condition number
    return theta  # [slope, intercept]
 
# ── 3. Loss & gradient ─────────────────────────────────────────────────
def mse_loss(theta, X, Y):
    check_xy(X, Y)
    if np.shape(theta) != (2,):
        raise ValueError("theta must have shape (2,)")
    w, b = theta
    pred = w * X + b
    return float(np.mean((pred - Y) ** 2))
 
def mse_gradient(theta, X, Y):
    """dLoss/dw = 2·mean((pred-Y) · X);  dLoss/db = 2·mean(pred-Y)."""
    check_xy(X, Y)
    if np.shape(theta) != (2,):
        raise ValueError("theta must have shape (2,)")
    w, b = theta
    err = w * X + b - Y
    dw = 2.0 * np.mean(err * X)
    db = 2.0 * np.mean(err)
    return np.array([dw, db])
 
# ── 4. Vanilla gradient descent ────────────────────────────────────────
def gradient_descent(X, Y, lr=0.05, steps=200):
    check_xy(X, Y)
    if not np.isfinite(lr) or lr <= 0 or not isinstance(steps, int) or steps < 0:
        raise ValueError("positive finite lr and nonnegative integer steps required")
    theta = np.array([0.0, 0.0])
    history = [(theta.copy(), mse_loss(theta, X, Y))]
    for _ in range(steps):
        g = mse_gradient(theta, X, Y)
        theta = theta - lr * g
        history.append((theta.copy(), mse_loss(theta, X, Y)))
    return theta, history
 
# ── 5. Mini-batch SGD with momentum ────────────────────────────────────
def sgd_momentum(X, Y, lr=0.02, beta=0.9, batch_size=32, epochs=20):
    check_xy(X, Y)
    if (not np.isfinite(lr) or lr <= 0 or not 0 <= beta < 1
            or not isinstance(batch_size, int) or batch_size < 1
            or not isinstance(epochs, int) or epochs < 0):
        raise ValueError("invalid learning rate, momentum, batch size or epochs")
    theta = np.array([0.0, 0.0])
    v = np.zeros_like(theta)
    n = len(X)
    history = [(theta.copy(), mse_loss(theta, X, Y))]
    for epoch in range(epochs):
        perm = rng.permutation(n)
        for i in range(0, n, batch_size):
            idx = perm[i:i + batch_size]
            g = mse_gradient(theta, X[idx], Y[idx])
            v = beta * v + g
            theta = theta - lr * v
        history.append((theta.copy(), mse_loss(theta, X, Y)))
    return theta, history
 
# ── 6. Compare lr choices: too small, just right, too big ──────────────
def compare_learning_rates() -> None:
    optimum = mse_loss(analytic_ols(X, Y), X, Y)
    for lr in [0.001, 0.05, 0.3]:
        theta, hist = gradient_descent(X, Y, lr=lr, steps=200)
        final_loss = hist[-1][1]
        label = "near OLS" if abs(final_loss - optimum) < 1e-6 else "not near OLS"
        print(f"lr={lr:>6.3f}  final=[{theta[0]:.3f}, {theta[1]:.3f}]  "
              f"loss={final_loss:>8.3f}  {label}")
 
def diverging_run() -> None:
    print("\n── Diverging run: lr way too big ─────")
    theta, hist = gradient_descent(X, Y, lr=1.0, steps=10)
    for step, (th, l) in enumerate(hist):
        marker = "  ← NaN/inf" if not np.isfinite(l) else ""
        print(f"step {step:>2}  theta=[{th[0]:>+8.2e}, {th[1]:>+8.2e}]  "
              f"loss={l:>+10.3e}{marker}")
 
if __name__ == "__main__":
    theta_ana = analytic_ols(X, Y)
    print(f"analytic OLS:   slope={theta_ana[0]:.4f}  intercept={theta_ana[1]:.4f}")
 
    theta_gd, _ = gradient_descent(X, Y, lr=0.05, steps=200)
    print(f"vanilla GD:     slope={theta_gd[0]:.4f}  intercept={theta_gd[1]:.4f}")
 
    theta_sgd, _ = sgd_momentum(X, Y, lr=0.02, beta=0.9, epochs=20)
    print(f"SGD + momentum: slope={theta_sgd[0]:.4f}  intercept={theta_sgd[1]:.4f}")
 
    print("\n── Learning-rate sensitivity ─────")
    compare_learning_rates()
    diverging_run()

Run it:

uv run --with numpy python gd_demo.py

Expected output is generated from the seeded program above; the seed fixes this call order. Changing calls to the shared Generator changes later draws. NumPy versions may differ.

analytic OLS:   slope=1.9578  intercept=0.9206
vanilla GD:     slope=1.9578  intercept=0.9206
SGD + momentum: slope=1.9741  intercept=0.9022
 
── Learning-rate sensitivity ─────
lr= 0.001  final=[1.478, 0.383]  loss=   2.221  not near OLS
lr= 0.050  final=[1.958, 0.921]  loss=   1.042  near OLS
lr= 0.300  final=[-12.667, -0.559]  loss= 724.838  not near OLS
 
── Diverging run: lr way too big ─────
step  0  theta=[+0.00e+00, +0.00e+00]  loss=+1.549e+01
step  1  theta=[+1.35e+01, +2.77e+00]  loss=+4.546e+02
step  2  theta=[-6.39e+01, -6.40e+00]  loss=+1.472e+04
step  3  theta=[+3.78e+02, +3.96e+01]  loss=+4.783e+05
step  4  theta=[-2.14e+03, -2.16e+02]  loss=+1.554e+07
step  5  theta=[+1.22e+04, +1.24e+03]  loss=+5.048e+08
step  6  theta=[-6.96e+04, -7.04e+03]  loss=+1.640e+10
step  7  theta=[+3.97e+05, +4.01e+04]  loss=+5.327e+11
step  8  theta=[-2.26e+06, -2.29e+05]  loss=+1.731e+13
step  9  theta=[+1.29e+07, +1.30e+06]  loss=+5.622e+14
step 10  theta=[-7.35e+07, -7.43e+06]  loss=+1.827e+16

What each block does

Anatomy of the code

Name: analytic_ols

Desc: np.linalg.lstsq minimizes ‖Aθ−Y‖₂ and returns the minimum-norm solution when minimizers are nonunique. It avoids explicitly forming/inverting AᵀA. This is a numerical reference, not exact arithmetic; nonlinear optimization also has methods other than GD.

Name: mse_gradient

Desc: Hand-derived from MSE loss = mean((wx+b-y)²). Chain rule: dLoss/dw = 2·mean((wx+b-y) · x). This is what autograd would compute automatically.

Name: gradient_descent

Desc: Full-batch GD uses all 200 samples for every gradient. Its per-step cost grows with the dataset; whether it is practical depends on data size, throughput and convergence.

Name: sgd_momentum

Desc: Mini-batches of 32 and an unnormalized velocity sum v=βv+g. Setting beta=0 gives plain mini-batch SGD; batch_size=1 gives single-example updates. Compare examples processed, not just step counts; the final batch can be smaller.

Name: compare_learning_rates

Desc: Compare final loss with the least-squares reference, not an arbitrary threshold of 2. For this seeded dataset the largest Hessian eigenvalue is about 6.69974, so the stability ceiling 2/L is about 0.29852. Thus 0.3 grows slowly unstable; 1.0 diverges much faster.

Name: diverging_run

Desc: Watch the high-curvature error grow and change sign at lr=1.0. Ten steps need not overflow; divergence and nonfinite floating-point values are different observations.

What to observe

Vanilla GD approaches the least-squares reference for this full-rank problem and stable step size. This finite example is not proof of convergence for every convex problem. The 0.001 run has not reached the same accuracy after 200 steps; the 0.3 run is just above the stability boundary and eventually grows, but does not produce NaN in these 200 steps. Compare the SGD run using epochs and examples processed, not “fewer effective steps.”

The numerical demo uses finite real float64 arrays and θ of shape (2,), at moderate magnitudes; finite inputs alone do not prevent overflow during arithmetic. The guards check shape and finiteness, not arbitrary dtype coercion or numerical conditioning. The code leaves X and Y unchanged and copies every θ stored in history. With fixed parameter dimension 2, full-batch loss/gradient cost O(n); T GD steps cost O(Tn) time and O(T+n) memory including temporary residuals/history. One shuffled SGD epoch costs O(n), plus the permutation and residual arrays; E epochs and their loss measurements cost O(En). OLS uses numerical linear algebra with a fixed two-column matrix. Shape (n,1) labels would broadcast a vector prediction into an (n,n) residual matrix, so the guard rejects them.

Goal: Feel the loss surface as a 3D landscape

Hint: A high-rate trajectory can alternate across the steep axis and grow outward; it need not spiral. The red star is the generating parameter, not generally the noisy-sample minimizer. Use the OLS reference to locate the latter.

Visualise the loss landscape (requires matplotlib):

import matplotlib.pyplot as plt
 
ws = np.linspace(-1, 5, 50)
bs = np.linspace(-2, 4, 50)
W, B = np.meshgrid(ws, bs)
L = np.zeros_like(W)
for i in range(W.shape[0]):
    for j in range(W.shape[1]):
        L[i, j] = mse_loss(np.array([W[i, j], B[i, j]]), X, Y)
 
plt.figure(figsize=(7, 6))
plt.contour(W, B, L, levels=20)
plt.plot(2, 1, "r*", markersize=15, label="true (2, 1)")
plt.xlabel("slope w"); plt.ylabel("intercept b")
plt.title("MSE loss surface for y = wx + b")
for lr in [0.001, 0.05, 0.5]:
    _, history = gradient_descent(X, Y, lr=lr, steps=12)
    path = np.array([theta for theta, _ in history])
    plt.plot(path[:, 0], path[:, 1], label=f'lr={lr}')
ols = analytic_ols(X, Y)
plt.plot(*ols, 'ko', label='sample OLS')
plt.legend(); plt.savefig("loss_surface.png", dpi=100)
plt.close()

This is the completed trajectory exercise: each row of path is a stored (w,b), so column 0 is horizontal and column 1 vertical. A parameter-space contour is not the same plot as loss versus training step; it explains why that scalar curve moves. The unstable path can leave the plotted grid, requiring separate axis limits.

(d) Production reality · 15 min

Finite gradients, clipping and schedules

Before updating, check both loss and gradients for finiteness. For g=(3,4), norm clipping at c=1 produces (0.6,0.8); its direction is unchanged. It limits this gradient norm, not the loss, curvature, or the final Adam/momentum update norm. Save optimizer state with weights when checkpointing; otherwise a rewind does not restore the same dynamics. Investigate a failed batch rather than silently dropping it, which can bias training. The archived LLaMA cost/spike story is not independently established here.

A cosine schedule over steps 0≤t≤T uses η(t)=η_min+(η_max−η_min)(1+cos(πt/T))/2, giving η_max at zero and η_min at T. Warmup increases the rate before a later schedule; neither repairs an invalid loss. For Adam, m=β₁m+(1−β₁)g and v=β₂v+(1−β₂)g²; bias-correct with m/(1−β₁ᵗ), v/(1−β₂ᵗ), then divide the first by sqrt(second)+ε. These equations explain the two state tensors.

Shuffling is a sampling policy, not a slogan

For two scalar squared-error examples with targets 0 and 2, start at θ=0 and use η=0.25. Processing targets [0,2] gives θ=1; processing [2,0] gives θ=0.5. Order changes finite-step updates. Shuffling reduces systematic ordering effects for exchangeable training examples; it is not an unconditional rule for temporal sequences, stateful batches or evaluation data. The script's rng.permutation(n) explicitly chooses random reshuffling and preserves X/Y alignment. No invented 62%-to-89% benchmark is needed.

Coupled L2 versus decoupled decay

With a zero data gradient, θ=2, λ=0.1 and η=0.01, the first Adam update using the L2 gradient 0.2 is approximately 0.01 after bias correction (ignoring ε). Decoupled decay alone shrinks by ηλθ=0.002. The two rules are not equivalent under adaptive scaling. AdamW applies θ ← (1−ηλ)θ−η·adaptive_step using the old θ. Select parameter groups and decay intentionally; neither rule guarantees superior validation accuracy.

Where this shows up in the rest of the plan

Root: Gradient-based optimization connects these differentiable learning topics; labels below are conceptual, not authoritative session numbers

Label: Linear + logistic regression

Desc: Minimize negative log-likelihood or MSE with an appropriate optimizer; maximizing log-likelihood uses ascent.

Label: Neural networks

Desc: Backprop computes gradients; SGD/Adam updates weights. Same loop as this session, 10⁹ params.

Label: Advanced optimisers

Desc: Adam, AdamW, Lion and Sophia change update state, scaling or curvature approximations; compare rather than assume better defaults.

Label: Reinforcement learning · policy gradient

Desc: For differentiable parameterized policies, policy-gradient estimators optimize expected return, often using sampled trajectories and surrogate losses.

Label: LLM pretraining

Desc: Autoregressive pretraining uses a differentiable token-prediction objective and a large-scale update loop; optimizer/schedule and numerical details are additional choices.

Label: Fine-tuning + LoRA

Desc: Freeze most weights, train a few low-rank matrices with… gradient descent. Same recipe.

(e) Recall + stretch · 10 min

Self-check (answer follows each question).

Recall — questions with written answers

Q: In one sentence, what is a gradient?

A: The vector of partial derivatives of a differentiable real scalar function. A nonzero gradient points in steepest local ascent under the Euclidean norm; its norm is the maximal unit-direction slope.

Q: Why do we step against the gradient (θ ← θ − lr · ∇L), not with it?

A: The directional derivative along −∇L is −‖∇L‖². This is strictly negative when the gradient is nonzero, so a sufficiently small positive step decreases a differentiable loss locally; an arbitrary finite step need not.

Q: What are the three failure modes of learning rate, and what does each look like in the loss curve?

A: A high rate can cause oscillation/divergence; a low rate can make progress slow. A stable deterministic run can descend steadily. These are clues, not diagnoses: minibatch noise, invalid data and vanishing gradients can look similar.

Q: Batch, mini-batch, stochastic — what's the difference and which is used in modern deep learning?

A: Full batch uses all examples; single-example SGD uses one; mini-batch uses a subset. Batch sizes are workload choices, not restricted to 32–1024. The full-data average is the reference for unbiased fixed-parameter sampling.

Q: What does momentum add on top of plain SGD, and what's the intuition?

A: Momentum tracks a velocity v = β·v + g and steps in the direction of v. Intuition: like a ball rolling down the hill — it retains previous direction, so it smooths through small bumps and accelerates on long flat slopes.

Q: What's the difference between Adam and AdamW?

A: Classical Adam with an L2 term feeds that term through its adaptive moments. Decoupled weight decay applies a separate parameter shrinkage; library Adam/AdamW flags vary by version, so inspect the specific optimizer API.

Q: What is gradient clipping, and when do you need it?

A: Rescale the gradient to a maximum norm before the update: if ||g|| > c: g ← g · c/||g||. Limits finite gradient spikes; it cannot fix nonfinite inputs/gradients or guarantee a finite loss. Optimizer state can still affect the actual update magnitude.

Q: Why does SGD's noise sometimes help training?

A: Noise can perturb some stationary regions if it has a nonzero component in a useful direction, but can also prevent settling. If all per-example gradients are zero, minibatching supplies no escape; there is no universal generalization benefit.

Q: You're training a model and the loss goes NaN at step 4000. What do you check first?

A: Find the first nonfinite input, forward intermediate or gradient; inspect the failing batch and stable loss formulation. Check precision/loss scaling, step size and finite-gradient norms. A smaller rate or clipping is appropriate only for the diagnosed cause. Restore weights and optimizer state together; do not assume skipping corrupt batches is harmless.

Q: Stretch — derive the gradient of the MSE loss L = (1/n) Σ (w·xᵢ + b − yᵢ)² with respect to w and b.

Stretch: true

A: Let eᵢ = w·xᵢ + b − yᵢ. Then L = (1/n) Σ eᵢ². By chain rule: ∂L/∂w = (1/n) Σ 2·eᵢ · ∂eᵢ/∂w = (2/n) Σ eᵢ · xᵢ. Similarly ∂L/∂b = (2/n) Σ eᵢ · 1 = (2/n) Σ eᵢ. Those are the exact expressions used in mse_gradient in the hands-on code.

Explain-out-loud test

1. What does gradient descent do, in one paragraph, to a total beginner?
2. What is the learning rate and what happens if it's too big or too small?
3. Why does every deep-learning framework compute gradients automatically?

Explain-out-loud answer guide

1. GD measures a scalar loss, differentiates it with respect to each trainable parameter, and subtracts a scaled gradient; it uses local information and can fail with unsuitable steps or objectives.
2. The learning rate scales each update. On a quadratic it must respect curvature; overly large steps amplify error, while small stable steps can be slow within a finite budget.
3. Automatic differentiation applies the chain rule through a recorded computation graph; it avoids writing every partial derivative by hand but does not check whether the chosen loss or data are correct.

Optional archived videos

These are retained resource identities, not governing evidence. Playback, duration and suitability were not verified. The two 3Blue1Brown cards share one URL; neither is certified as an optimiser-comparison video.

• Gradient descent · Chapter 2 · Deep Learning — archived label, unverified.
• The Spelled-out Intro to Neural Networks and Backpropagation — archived label, unverified.
• Gradient descent, how neural networks learn | Deep Learning Chapter 2 — archived label, unverified.

What comes next

Next: S021 · Probability — Random Variables, Distributions, Expectation

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.

Corrective source notes

• NumPy lstsq — least-squares and minimum-norm contracts.

• Adaptive subgradient methods — per-coordinate adaptation.

• Decoupled Weight Decay Regularization — L2/decay distinction for adaptive optimizers.

• Lecture 7: Gradient Descent (and Beyond) — An excessively large gradient-descent learning rate can cause divergence. Checked 2026-10-07.

These citations support the named claims, not blanket certification of the lesson. Worked proofs are derived in the text; execution scope and finite-test limits are recorded separately.