S018 · Linear Algebra II — Matrices, Transforms, Eigenvalues

Module M02: Math Foundations · Session 18 of 130 · Track: Math 🔲

What you'll be able to do after this session

• Explain Linear Algebra II — Matrices, Transforms, Eigenvalues to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S017 — Linear Algebra I — Vectors, Dot Product, Geometry

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown — Linear transformations and matrices — the video that makes matrices click.
• 🎥 Deep dive (30–60 min) · 3Blue1Brown — Eigenvectors and eigenvalues — the entire "why do we care about eigen-anything" question, answered visually.
• 🎥 Hands-on demo (10–20 min) · StatQuest — PCA main ideas in only 5 minutes — PCA is the killer application of eigenvectors.
• 📖 Canonical article · Wikipedia — Matrix (mathematics) — reference for notation and operations.
• 📖 Book chapter / tutorial · Introduction to Linear Algebra — Gilbert Strang (MIT OCW) — the course every serious learner does.
• 📖 Engineering blog · Netflix — Matrix Factorization Techniques for Recommender Systems — how the Netflix Prize was won with SVD.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Neural nets are stacked matrix multiplies.

A matrix is a grid of numbers, but that's the boring definition. The interesting definition: a matrix is a function that transforms vectors. Multiply a matrix by a vector and out comes a new vector — usually rotated, stretched, squished, or reflected. [[0, -1], [1, 0]] is "rotate 90° counter-clockwise". [[2, 0], [0, 2]] is "double the size". Once you see matrices as transformations, linear algebra stops feeling like arithmetic and starts feeling like geometry.

Why does this matter? Because a neural network is nothing more than a chain of matrix transformations glued together with squishy non-linear functions. Every "hidden layer" is one matrix multiply. A GPT model with 175B parameters is really just a giant chain of matrix multiplies, each turning one vector of activations into the next. Understanding matrices means understanding what neural nets are actually doing.

The forever-gotcha: matrix multiplication is not commutative. A·B ≠ B·A in general — the order matters because you're composing transformations. "Rotate then stretch" is different from "stretch then rotate". This trips up everyone the first time.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A 2×2 matrix acts on the standard basis vectors — that's where every intuition comes from:

• Take î = [1, 0] and ĵ = [0, 1]. These are the "east" and "north" unit vectors.
• The columns of the matrix tell you where î and ĵ land after the transformation.
• Everything else follows because linear transformations preserve lines and the origin.

Shape rules — the only ones you ever need:

The rule to memorise: inner dimensions must match, outer dimensions become the result. (3, 5) · (5, 2) → (3, 2). (3, 5) · (2, 5) → error.

Worked example 1 — matrix-vector multiply by hand:

A = [[1, 2],       v = [3,
     [3, 4]]            4]

A·v = [1·3 + 2·4,   =  [11,
       3·3 + 4·4]       25]

Row-i of A dotted with v gives entry-i of the result. That's it.

Worked example 2 — composition of transforms:

• S = [[2,0],[0,2]] doubles everything.
• R = [[0,-1],[1,0]] rotates 90° counter-clockwise.
• S · R means "first rotate, then scale" (right-to-left, like function composition f(g(x))).
• R · S means "first scale, then rotate".

In this specific case they happen to be equal (uniform scaling commutes with rotation) but for R · shear ≠ shear · R you'd get different pictures.

Special matrices:

Eigenvectors, in one sentence: an eigenvector of A is a special vector that A only stretches (doesn't rotate) — it's the "axis" of the transformation. The stretch factor is the eigenvalue. A 2D rotation matrix has no real eigenvectors (nothing stays on its own line under rotation). A symmetric matrix always has a full set of perpendicular eigenvectors — this is why PCA works.

Eigenvectors of the covariance matrix of your data are the principal components — the directions of maximum variance. Projecting onto the top-k eigenvectors is dimensionality reduction (PCA). Whole recommender systems and image-compression schemes are built on this one idea.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

"""Matrices, transforms, and a mini PCA in ~50 lines."""
import numpy as np
 
# ---- 1. Shapes and the multiplication rule -------------------------
A = np.array([[1, 2], [3, 4]])          # 2×2
v = np.array([3, 4])                    # length 2
print("A·v      =", A @ v)              # [11, 25]
print("v·A      =", v @ A)              # different! [15, 22]  — non-commutativity
 
# ---- 2. Transformations of the unit square -------------------------
square = np.array([[0,0],[1,0],[1,1],[0,1],[0,0]]).T   # 2×5 points
scale2   = np.array([[2, 0], [0, 2]])
rotate90 = np.array([[0,-1], [1, 0]])
shearx   = np.array([[1, 1], [0, 1]])
 
for name, M in [("original", np.eye(2)), ("scale", scale2),
                ("rotate", rotate90), ("shear-x", shearx),
                ("scale then rotate", rotate90 @ scale2),
                ("rotate then scale", scale2 @ rotate90)]:
    pts = M @ square
    print(f"{name:20s} corners →", pts.T[:4].tolist())
 
# ---- 3. Eigenvalues / eigenvectors ---------------------------------
S = np.array([[4.0, 1.0], [1.0, 3.0]])                # symmetric ⇒ real eigenvals
vals, vecs = np.linalg.eig(S)
print("\neigenvalues :", vals.round(3))
print("eigenvectors :\n", vecs.round(3))
# check A·v = λ·v for the first eigenvector
v0 = vecs[:, 0]
print("A·v0 =", (S @ v0).round(3), "  vs  λ·v0 =", (vals[0] * v0).round(3))
 
# ---- 4. PCA from scratch on toy 2-D data ---------------------------
rng = np.random.default_rng(0)
X = rng.multivariate_normal(mean=[0,0],
                            cov=[[4, 3.5],[3.5, 4]],
                            size=500)
X -= X.mean(axis=0)                        # centre
cov = (X.T @ X) / (len(X) - 1)             # 2×2 covariance
eig_vals, eig_vecs = np.linalg.eigh(cov)   # eigh: sorted, for symmetric
order = np.argsort(-eig_vals)              # largest first
eig_vals, eig_vecs = eig_vals[order], eig_vecs[:, order]
X_pca = X @ eig_vecs                       # rotate onto principal axes
print("\ncovariance:\n", cov.round(3))
print("PC variances:", eig_vals.round(3), "(should be ~7.5 and ~0.5)")
print("first 3 rows PCA-projected:\n", X_pca[:3].round(3))

Observe when you run:

1. A@v ≠ v@A — proves non-commutativity in one line.
2. scale then rotate and rotate then scale give the same answer here (uniform scaling commutes with rotation). Swap scale2 for shearx and they diverge.
3. The check A·v0 == λ·v0 holds to floating-point precision — that's the definition of an eigenvector.
4. PC variances come out roughly [7.5, 0.5] — the first principal component captures 94% of the variance (7.5 / 8). Perfect setup for dimensionality reduction.
5. After PCA the data is rotated so the first axis is the direction of maximum variance. The columns of X_pca are decorrelated (off-diagonal of their covariance ≈ 0).

Try this modification: truncate to 1 dimension (X_pca[:, :1]) and reconstruct back to 2D (X_pca[:, :1] @ eig_vecs[:, :1].T). Compare with the original — how much info was lost? This is exactly what image compression via SVD/PCA does.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Matrix multiplication is the single most optimised operation in computing history. When you write A @ B in NumPy, it dispatches to a BLAS library (MKL, OpenBLAS, Apple Accelerate) hand-tuned in assembly for your CPU's SIMD instructions. On a GPU, the same operation goes to cuBLAS or ROCm and hits tensor cores. NVIDIA's H100 does ~2000 TFLOPS of FP16 matmul — a single second on that GPU is more matrix multiplies than a human could do by hand in millions of years.

The Netflix Prize (2006–2009, $1M prize) was won by matrix-factorisation approaches: represent the sparse "user × movie" ratings matrix as the product of two smaller matrices (users × latent-factors) and (latent-factors × movies). Predict a missing rating by multiplying the two row/column vectors. Simon Funk's blog post laying this out is the origin of modern recommender systems.

PCA is used everywhere you need to visualise or compress high-dim data — visualising a 1024-dim embedding in 2D (project to top-2 eigenvectors), speeding up nearest-neighbour by pre-reducing to 32 dims, denoising images by keeping only the top-k eigen-components.

At Microsoft on the OneNote Copilot LLM-eval framework, we use SVD (a generalised eigendecomposition) on the matrix of eval scores × model versions to identify systematic quality regressions — a single "hidden factor" (say, "handles code blocks poorly") shows up as a strong singular vector and we know exactly which capability we broke.

Gotchas in production:

• Order of multiplies matters for cost: A(BC) vs (AB)C may differ by 100× in FLOPs depending on shapes. Numpy has np.einsum and PyTorch has opt_einsum that pick the best order automatically.
• Numerical instability: inverting an ill-conditioned matrix (np.linalg.inv) amplifies floating-point error catastrophically. Prefer np.linalg.solve(A, b) over np.linalg.inv(A) @ b. In ML, prefer pseudo-inverse (np.linalg.pinv) or regularisation (add λI to the diagonal — "ridge").
• Non-symmetric input to eigh: np.linalg.eigh is faster and stabler than eig, but only for Hermitian/symmetric matrices. Using it on a non-symmetric matrix returns junk without error.
• Precision matters at scale: FP32 vs BF16 vs FP16 — LLM training now happens in BF16/FP8 to halve memory. Understanding matrix ops means understanding why "mixed precision" works: the numerically sensitive reductions stay in FP32, the bulk multiplies drop to BF16.
• A @ B is not A*B in NumPy — the second is element-wise Hadamard. This bug kills people daily.

(e) Quiz + exercise · 10 min

1. State the shape rule for matrix multiplication. Given a (3, 5) matrix times a (5, 2) matrix, what shape comes out? What about (3, 5) * (5, 2) element-wise?
2. Why is matrix multiplication not commutative? Give a geometric example.
3. What does it mean for v to be an eigenvector of A with eigenvalue λ? Write the defining equation.
4. Explain PCA in one sentence using the words "covariance", "eigenvectors", "variance".
5. Why does np.linalg.solve(A, b) beat np.linalg.inv(A) @ b in production?

Stretch (connects to S017 Vectors): show that if u and v are perpendicular eigenvectors of a symmetric matrix S, then u·v = 0 before and after applying S (i.e., (S·u)·(S·v) = 0). Why does this property make symmetric matrices so useful for PCA?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Linear Algebra II — Matrices, Transforms, Eigenvalues? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S019): Calculus I — Derivatives & Chain Rule
• Previous (S017): Linear Algebra I — Vectors, Dot Product, Geometry
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.