R04 · Week 4 Recall & Drill

Weekly revision · Week 4 · Covers 5 sessions from Mon–Fri.

Sessions covered

• S016 — Discrete Math — Sets, Logic, Combinatorics, Graphs
• S017 — Linear Algebra I — Vectors, Dot Product, Geometry
• S018 — Linear Algebra II — Matrices, Transforms, Eigenvalues
• S019 — Calculus I — Derivatives & Chain Rule
• S020 — Calculus II — Gradients & Gradient Descent from Scratch

90-min structure

Blank-page reconstruction · 30 min

S016 · Discrete Math

1. Write set-builder notation for "the even numbers in S", then the equivalent Python comprehension and the equivalent SQL WHERE clause.
2. State both of De Morgan's laws, then simplify NOT (x AND (y OR NOT z)) by hand.
3. You have 10 candidates and need 3 for a team. How many teams? Now how many if the three roles are distinct and ranked?

Gotcha you probably forgot: you use discrete math constantly, just unnamed. A SQL JOIN is a filtered Cartesian product, DISTINCT is set semantics, GROUP BY is a partition into equivalence classes, a task scheduler is a directed acyclic graph, and hash partitioning is modular arithmetic. The notation is not academic decoration — it is the compressed form of things you already do.

S017 · Vectors

1. Give two equally valid descriptions of a vector, and say when each view is the useful one.
2. Write both the algebraic and the geometric formula for the dot product, and say what each factor means in the geometric one.
3. You compute cosine similarity and get a negative value. What does that tell you about the angle?

Gotcha you probably forgot: cosine similarity measures angle only and throws magnitude away entirely. Two vectors pointing the same direction have cosine similarity 1 even if one is a hundred times longer. Euclidean distance measures position. The two rankings only agree when every vector has been normalised to unit length — which is precisely why vector databases usually store pre-normalised embeddings.

S018 · Matrices

1. Say in one sentence what a matrix does to a vector, then say what the columns of the matrix are, geometrically.
2. State the shape rule for matrix multiplication, and explain why A times B generally differs from B times A.
3. Define an eigenvector and its eigenvalue in terms of what the transformation does to that direction.

Gotcha you probably forgot: to solve a linear system in code, you do not compute the inverse and multiply. Explicitly inverting a matrix is slower and numerically worse than using a dedicated solver, which factorises the matrix instead. Reach for the library's solve routine; the inverse is a mathematical concept you rarely want to materialise.

S019 · Derivatives & Chain Rule

1. Write the definition of the derivative as a limit, without looking.
2. State the chain rule for a composition, then compute the derivative of a function composed three deep, showing each factor.
3. Explain the difference between symbolic, numeric, and automatic differentiation, and where each one is used in practice.

Gotcha you probably forgot: a derivative is best read as a local linear approximation — "nudge the input a little, how much does the output move and in which direction?" — not as a fact about the geometry of a graph. That reading is what makes it the engine of optimisation and sensitivity analysis, and it is why backpropagation is nothing more than the chain rule applied repeatedly and cached.

S020 · Gradients & Gradient Descent

1. Define the gradient in one sentence, and say why the update rule subtracts it rather than adding it.
2. Distinguish batch, mini-batch, and stochastic gradient descent, and say which one modern deep learning actually uses.
3. Describe the three learning-rate failure modes and what each one looks like on a loss curve.

Gotcha you probably forgot: gradient descent does not find the minimum. It finds a point where the gradient is approximately zero that happens to be reachable from where you started. For a non-convex loss there are enormously many such stationary points, and which one you land in depends on initialisation, batch ordering, and the schedule. "It converged" and "it found the best solution" are entirely different claims.

Hands-on drill · 30 min

Task: implement gradient descent from scratch on linear regression, then check your answer against the closed-form solution. The closed form is the grader — this is the rare case where you can verify an optimiser exactly.

Step 1 — generate data with known parameters (5 min)

mkdir -p ~/projects/w4-drill && cd ~/projects/w4-drill
uv venv .venv --python 3.12 && source .venv/bin/activate
uv pip install numpy

# data.py
import numpy as np
 
rng = np.random.default_rng(0)
N = 500
X = rng.normal(size=(N, 3))
true_w = np.array([2.0, -3.0, 0.5])
true_b = 1.5
y = X @ true_w + true_b + rng.normal(scale=0.1, size=N)
np.savez("data.npz", X=X, y=y, true_w=true_w, true_b=true_b)
print("saved", X.shape, y.shape)

Expected outcome: saved (500, 3) (500,). You now know the answer the optimiser is supposed to find.

Step 2 — the closed form, as ground truth (5 min)

# closed_form.py
import numpy as np
 
d = np.load("data.npz")
X, y = d["X"], d["y"]
X1 = np.hstack([X, np.ones((len(X), 1))])   # append the bias column
 
# Solve, do NOT invert. lstsq factorises; np.linalg.inv would be slower and worse.
theta, *_ = np.linalg.lstsq(X1, y, rcond=None)
print("closed form w =", theta[:3], " b =", theta[3])
print("truth       w =", d["true_w"], " b =", d["true_b"])

Expected outcome: the recovered w sits very close to [2, -3, 0.5] and b close to 1.5. It will not match to the last decimal — noise was added on purpose — but it should agree to roughly two decimal places.

Step 3 — gradient descent by hand (12 min)

Derive the gradient before you code it. For mean squared error, the gradient with respect to w is the mean over samples of 2 * (prediction - target) * x, and with respect to b it is the mean of 2 * (prediction - target). Write that on paper first, then:

# gd.py
import numpy as np
 
d = np.load("data.npz")
X, y = d["X"], d["y"]
n = len(X)
 
def descend(lr, steps=2000):
    w = np.zeros(X.shape[1])
    b = 0.0
    history = []
    for _ in range(steps):
        pred = X @ w + b
        err = pred - y
        loss = np.mean(err ** 2)
        history.append(loss)
        grad_w = (2 / n) * (X.T @ err)
        grad_b = (2 / n) * err.sum()
        w -= lr * grad_w
        b -= lr * grad_b
    return w, b, history
 
for lr in (0.5, 0.1, 0.001):
    w, b, hist = descend(lr)
    first, last = hist[0], hist[-1]
    print(f"lr={lr:<6} loss {first:.4f} -> {last:.6f}   w={np.round(w, 3)} b={b:.3f}")

Expected outcome: at a learning rate of 0.1 the loss drops steadily and w and b land on the same values the closed form found — that agreement is the whole point of the drill. At 0.001 the loss is still visibly falling when the loop ends: it is not wrong, just unfinished, which is what "too low" looks like. At 0.5, watch whether the loss decreases smoothly, oscillates, or blows up to a non-finite value; whichever happens, that is your crossover behaviour on this particular problem and it is worth seeing rather than reading about.

Step 4 — verify the chain rule numerically (8 min)

Symbolic derivatives are easy to get subtly wrong. Check them:

# checkgrad.py
import numpy as np
 
def f(x):                      # a three-deep composition
    return np.sin(np.exp(x ** 2))
 
def df_symbolic(x):            # chain rule, outermost to innermost
    return np.cos(np.exp(x ** 2)) * np.exp(x ** 2) * 2 * x
 
def df_numeric(x, h=1e-5):     # central difference
    return (f(x + h) - f(x - h)) / (2 * h)
 
for x in (0.3, 0.7, 1.1):
    s, nmr = df_symbolic(x), df_numeric(x)
    print(f"x={x}  symbolic={s: .6f}  numeric={nmr: .6f}  diff={abs(s - nmr):.2e}")

Expected outcome: the symbolic and numeric columns agree to several decimal places and the difference column is tiny. This is the exact technique to use whenever you hand-derive a gradient for a model — if the two disagree, your algebra is wrong, and finding that out here is much cheaper than finding it out after a training run.

Gap analysis + next week preview · 10 min

• Did your hand-derived gradient match the numeric check on the first attempt? If not, the error you made is the one to write down — hand-derived gradients fail the same way twice.
• Did gradient descent at the working learning rate actually reproduce the closed-form answer? If it stopped short, the fix is more steps or a larger rate, and knowing which one is the diagnostic skill.
• Can you explain matrix multiplication as composition to someone who only knows the grid picture? If not, redraw the basis-vector diagram from memory.

Next week (S021–S025) finishes the math block and opens data structures: probability with random variables, distributions and expectation; statistics with the central limit theorem, hypothesis testing and confidence intervals; then arrays and strings with the two-pointer pattern; hashmaps and sets with hash functions and collisions; and linked lists. The dot-product and Big-O reflexes you have been building are exactly what the DSA sessions assume you already own.

Part of the 6-month evergreen learning plan.