DL S002 · Vectors, Matrices, and the Geometry of Data

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 2 / 80 · Module M01 · ~2 hours

The story

Here's the puzzle I want you to sit with for a second.

Every recommender system that has ever suggested a movie to you did it, at some level, by computing the dot product between two lists of numbers — one representing you, one representing the movie — and picking the movies whose dot product with your list was highest.

Every semantic search box that finds the "closest" document to your query does it by computing the dot product between the query's embedding vector and each document's embedding vector.

Every attention head inside a transformer — literally every single one of GPT-4's ~1,000 attention heads on every forward pass — computes a dot product between a "query" vector and a bunch of "key" vectors to decide what to pay attention to.

The dot product is the operation. Not "one of many operations" — the operation. The single most consequential arithmetic idea in modern machine learning. And it's the arithmetic thing you (probably) learned in high school and immediately forgot.

Two hours today: we take the dot product apart geometrically, feel it in our bones as "how aligned are these two arrows?", derive it, compute it by hand on tiny numeric examples, and then re-derive attention as "a soft, differentiable, weighted dot-product lookup". If we do this well, the entire M07 transformer module — 8 sessions from now — will read like arithmetic.

Prerequisites

• Session 001 — you need .shape, .reshape, @ vs *, and axis= fluency from yesterday. If those aren't muscle memory, go back.
• High-school Pythagoras. You remember c² = a² + b². That's it.

1 · A vector is TWO things at once

The first mental leap. A vector is:

1. An arrow in space with a length and a direction (the geometric picture).
2. A list of numbers (the algebraic picture).

Both are correct. Both are the same thing. Learning to switch between them fluidly is the entire skill of linear algebra.

1.1 The geometric picture

Take a piece of paper. Draw two perpendicular axes: x horizontal, y vertical. The vector [3, 2] is the arrow that starts at the origin and ends at the point 3 units right, 2 units up.

Its length is √(3² + 2²) = √13 ≈ 3.61 — Pythagoras. Its direction is arctan(2/3) ≈ 33.7° above the x-axis.

That's the whole geometric picture.

1.2 The algebraic picture

The same vector is just the list [3, 2]. In NumPy: np.array([3, 2]). Shape (2,). Two numbers. No arrow required.

1.3 Why both pictures matter

If I ask "what's [3, 2] + [1, 4]?", the algebraic picture gives you the answer instantly: [4, 6]. Add corresponding entries.

But if I ask "why does that make sense?", the geometric picture gives you the answer: walk from the origin along arrow 1, then along arrow 2 (tail-to-tip). Wherever you end up is the sum. It's the same point either way.

1.4 In higher dimensions

The picture breaks down after 3 dimensions (you can't draw a 4D arrow), but the algebra just keeps working. A vector of length 768 (the embedding dimension for BERT-base) is a 768-number list. It's an arrow in 768-dimensional space. You can't visualize it, but you can still compute its length, add it to other 768-vectors, and take its dot product with them.

The trick: whenever you're stuck in high dimensions, mentally drop back to 2D, work out the intuition, then trust that it scales.

2 · Vectors are everywhere in disguise

Before we do more math, here's why any of this matters. Everything in machine learning is a vector. Once you see it, you can't un-see it.

Every one of these is a vector, and every operation we're about to learn — dot products, norms, distances — applies to every one of them. The math is the same. Only the semantic label changes.

3 · Vector operations — the daily-driver kit

Let's do the four operations you'll use every day.

3.1 Addition

Geometrically: tail-to-tip. Algebraically: entry-wise.

import numpy as np
a = np.array([3, 2])
b = np.array([1, 4])
a + b        # array([4, 6])

3.2 Scalar multiplication

Geometrically: stretch (or shrink, or flip) the arrow. Algebraically: multiply every entry.

2 * a        # array([6, 4])  — arrow twice as long, same direction
-1 * a       # array([-3, -2]) — same length, opposite direction
0.5 * a      # array([1.5, 1]) — half length

3.3 The L2 norm (length)

Geometrically: how long the arrow is. Algebraically: sqrt(sum of squares).

v = np.array([3, 4])
np.sqrt((v ** 2).sum())         # 5.0
np.linalg.norm(v)               # 5.0 — same thing, canonical form

The 3-4-5 right triangle. First worked example everyone learns. It generalizes to any dimension.

3.4 The DOT PRODUCT (the whole reason we're here)

Two definitions. They are equal. Both matter.

Algebraic:

Geometric:

where θ is the angle between the two arrows.

Read those two equations together and let the strangeness sink in. On the left is a formula that requires no geometry — just multiply and add. On the right is a formula that requires no coordinates — just lengths and an angle. They give the same number. That equivalence is one of the most beautiful and useful facts in all of applied math.

3.5 Compute one by hand

Let a = [3, 2, 1] and b = [1, 0, 2].

Algebraic:

Verify in NumPy:

a = np.array([3, 2, 1])
b = np.array([1, 0, 2])
a @ b            # 5
np.dot(a, b)     # 5
(a * b).sum()    # 5   — the manual way

All three produce 5. Use @ in modern code.

3.6 What does the sign tell you?

From the geometric definition, a · b = ‖a‖ ‖b‖ cos θ. Lengths are always ≥ 0. So the sign of the dot product is the sign of cos θ:

• Positive dot product → angle between the vectors is less than 90° → arrows point in similar directions.
• Zero dot product → angle is exactly 90° → arrows are perpendicular (orthogonal).
• Negative dot product → angle is more than 90° → arrows point in opposing directions.

That's it. That's the whole intuition. Every embedding-similarity system, every attention head, every recommender — they all use this one fact.

4 · Cosine similarity — the dot product's practical cousin

The dot product mixes two things: alignment (the cosine) and lengths. Sometimes we don't want the lengths. If you're comparing embeddings of documents, one long document shouldn't beat a short document just because "long" means bigger norm. We want pure "same-direction-ness".

Cosine similarity divides the dot product by both norms, isolating the cosine:

Its value is always in [-1, 1]. 1 = same direction. 0 = perpendicular. -1 = opposite.

def cosine_similarity(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
cosine_similarity(np.array([1, 0]), np.array([1, 0]))       # 1.0
cosine_similarity(np.array([1, 0]), np.array([0, 1]))       # 0.0
cosine_similarity(np.array([1, 0]), np.array([-1, 0]))      # -1.0
cosine_similarity(np.array([1, 1]), np.array([2, 2]))       # 1.0 (same direction, different length)

This is the exact operation OpenAI's embeddings API uses to rank search results. It's the exact operation Twitter's timeline uses to score which of the 500 tweets it just retrieved is "most similar" to your interests. It's… everywhere.

4.1 Worked example — comparing three "documents"

Say we have three 3-dimensional "embeddings" for three toy documents (in a real system they'd be 768-dim, but the math is identical):

d1 = np.array([1, 1, 0])   # "cats and dogs"
d2 = np.array([2, 2, 0])   # "many cats and many dogs" — longer, same direction
d3 = np.array([0, 0, 1])   # "quantum mechanics"
 
query = np.array([1, 1, 0])   # user searches "pets"
 
# Raw dot product
d1 @ query  # 2
d2 @ query  # 4  ← wins on raw dot product just because it's longer
d3 @ query  # 0
 
# Cosine similarity
cosine_similarity(d1, query)  # 1.0
cosine_similarity(d2, query)  # 1.0  ← tied — length no longer matters
cosine_similarity(d3, query)  # 0.0

Notice: raw dot product falsely privileged d2 (the longer document). Cosine similarity correctly identifies that d1 and d2 are equally relevant, and both far more so than d3.

Cosine similarity is the safer default when the vectors you're comparing have varying norms. In transformer attention, we deal with this a slightly different way (scaled dot product — we'll get there in Session 036), but the concern is the same.

5 · From vectors to matrices

A matrix is a rectangular grid of numbers. Shape (m, n) = m rows, n columns.

A = np.array([[1, 2, 3],
              [4, 5, 6]])
A.shape       # (2, 3)

Two ways to think about a matrix — again, dual:

1. A stack of row vectors. A is 2 vectors of length 3, stacked.
2. A linear transformation. A is a function that takes a 3-vector in and gives a 2-vector out.

Both matter. Let's do the second one, because it's the one that unlocks everything.

5.1 A matrix as a function

Multiplying A @ v where v is a 3-vector gives you a 2-vector. So A is literally a function: f(v) = A @ v maps ℝ³ → ℝ².

But not just any function — a linear function. That word means two very specific properties:

1. A @ (u + v) = (A @ u) + (A @ v) (distributes over addition)
2. A @ (c * v) = c * (A @ v) (scalars pass through)

These two properties are hugely restrictive. They mean the function is "flat" in a specific sense: no bending, no jumps. It can rotate space, stretch it, shear it, flip it, project it — but it cannot bend it.

5.2 What does the transformation look like?

Take the matrix

and apply it to the vector v = [1, 1].

Geometrically: A stretched the x-axis by 2 and the y-axis by 3. The vector [1, 1] ended up at [2, 3].

Try another one:

Apply R to [1, 0]:

The x-unit-vector went to the y-unit-vector. That's a 90° counterclockwise rotation. R is the rotation-by-90° matrix.

The point: every 2×2 matrix is some combination of rotate + stretch + shear + reflect. Every 3×3 matrix is the same in 3D. Every 768×768 matrix in a transformer's linear layer is doing the analogous thing in 768D space — moving points around in a very specific, linear way.

6 · Matrix multiplication — the operation that runs everything

Two matrices multiply as follows:

In words: entry (i, j) of the product is the dot product of row i of A with column j of B.

6.1 The shape rule

For A @ B to be defined:

The inner dimensions must match; the outer dimensions become the output shape. This is the single most important rule of ML. Tattoo it.

A = np.random.randn(3, 4)      # shape (3, 4)
B = np.random.randn(4, 5)      # shape (4, 5)
C = A @ B                      # shape (3, 5)  ✅
 
X = np.random.randn(3, 4)
Y = np.random.randn(3, 4)
Z = X @ Y                      # ERROR — inner dims don't match: (3, 4) @ (3, 4)
Z = X @ Y.T                    # ✅ (3, 4) @ (4, 3) = (3, 3)

6.2 Compute one by hand

Let

Then C = A @ B has shape (2, 2):

• C[0, 0] = row 0 of A · col 0 of B = [1, 2] · [5, 7] = 5 + 14 = 19
• C[0, 1] = row 0 of A · col 1 of B = [1, 2] · [6, 8] = 6 + 16 = 22
• C[1, 0] = row 1 of A · col 0 of B = [3, 4] · [5, 7] = 15 + 28 = 43
• C[1, 1] = row 1 of A · col 1 of B = [3, 4] · [6, 8] = 18 + 32 = 50

Verify:

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
A @ B
# array([[19, 22],
#        [43, 50]])

Every matrix multiply — even a 768×768 one inside GPT — is doing exactly this: dot products of rows of the left matrix with columns of the right. Once you see one, you've seen them all.

6.3 Matmul is NOT commutative

in general. This trips up beginners forever. Test it:

Order matters. When you're chaining matrix ops in a neural network, the order is the architecture. Swap two and you have a different model.

X = np.random.randn(32, 784)     # 32 flattened MNIST images
W1 = np.random.randn(784, 128)   # first hidden layer weights
W2 = np.random.randn(128, 10)    # output layer weights
b1 = np.random.randn(128)
b2 = np.random.randn(10)

7 · The transpose — swap rows and columns

A.T returns the matrix with rows and columns swapped. Shape (m, n) becomes (n, m).

A = np.array([[1, 2, 3],
              [4, 5, 6]])
A.T
# [[1, 4],
#  [2, 5],
#  [3, 6]]
A.T.shape       # (3, 2)

Transpose is important because it lets you turn matmul into "compute all-pairs dot products between two batches of vectors". Say you have:

• Q = 5 query vectors of dimension 3 → shape (5, 3)
• K = 7 key vectors of dimension 3 → shape (7, 3)

You want the 5×7 matrix S where S[i, j] is the dot product of query i with key j.

Then:

S = Q @ K.T           # shape (5, 3) @ (3, 7) = (5, 7)   ✅

That's it. Q @ K.T. One line. This is exactly what the attention mechanism computes on every layer of every transformer, only with d = 64 or 128 instead of 3, and with an extra / sqrt(d) scale factor:

scores = Q @ K.T / np.sqrt(d)     # attention "logits"

We just wrote attention. In one line. Nine sessions early. The rest of the transformer story (softmax the scores, multiply by V, add residuals, layer-norm, stack) is elaboration. This is the load-bearing operation.

7b · Eigenvectors, SVD, and why 2024–2025 fine-tuning is all about low rank

We've said “a matrix is a linear map — it rotates, scales, shears, reflects.” That's true. But some vectors, when you apply the matrix to them, come out pointing in exactly the same direction, just scaled. Those special vectors are the matrix's eigenvectors, and the scaling factor is the eigenvalue. This idea, unassuming as it sounds, is why LoRA (Low-Rank Adaptation) works, why we can fine-tune Llama 3.1 70B on a single 4090, and why PCA / SVD still show up in embedding pipelines in 2025.

7b.1 · The intuition, on a 2×2

Take

$A = \begin{pmatrix} 3 & 1 \\ 0 & 2 \end{pmatrix}.$

Most vectors change direction when we hit them with $A$. But $\mathbf{v}_1 = (1, 0)^T$ does not: $A \mathbf{v}_1 = (3, 0)^T = 3 \mathbf{v}_1$. So $\mathbf{v}_1$ is an eigenvector with eigenvalue $\lambda_1 = 3$. Similarly $\mathbf{v}_2 = (1, -1)^T$ satisfies $A \mathbf{v}_2 = (2, -2)^T = 2 \mathbf{v}_2$, eigenvalue $\lambda_2 = 2$. Every 2×2 matrix has (at most) 2 such directions. Every $d \times d$ matrix has (at most) $d$.

Why we care: if you know the eigenvectors and eigenvalues, you know everything about what the matrix does. Every input can be written as a sum of eigenvectors (assuming they span the space), and the matrix just scales each component by its eigenvalue. A complicated linear map becomes $d$ independent 1D scalings.

7b.2 · SVD — eigenvectors for rectangular matrices

Most matrices in ML aren't square. Your embedding matrix E is (vocab_size, d). Your attention weight W_Q is (d, d_k). Eigenvectors don't strictly exist for non-square matrices, but the singular value decomposition does, and it's the workhorse:

$W = U \Sigma V^T$

where $U$ is $(m, m)$ orthogonal (rotation), $\Sigma$ is $(m, n)$ diagonal with non-negative entries $\sigma_1 \geq \sigma_2 \geq \dots \geq 0$ (the singular values), and $V^T$ is $(n, n)$ orthogonal.

Geometrically: any linear map can be decomposed as rotate → stretch each axis by $\sigma_i$ → rotate again. The stretch factors $\sigma_i$ tell you how much “information” flows through each axis. If $\sigma_i$ is tiny, that axis is barely used.

import numpy as np
rng = np.random.default_rng(0)
W = rng.normal(size=(100, 100))
U, S, Vt = np.linalg.svd(W)
print(S[:5])   # top singular values
print(S[-5:])  # bottom ones (much smaller for a well-conditioned matrix)

7b.3 · Low-rank approximation — the theorem behind LoRA

The Eckart–Young theorem says: the best rank-$r$ approximation to $W$ (in Frobenius norm) is

$W_r = U_{:, :r} \; \Sigma_{:r, :r} \; V^T_{:r, :}$

— just keep the top $r$ singular values, throw the rest away. If the tail singular values are small (which they usually are for real-world weight matrices), $W_r$ is nearly indistinguishable from $W$ using $r(m+n)$ numbers instead of $mn$.

This is the entire mathematical basis for LoRA (Hu et al., 2021) and its 2024 successors:

$W_{\text{finetuned}} = W_{\text{pretrained}} + \Delta W \approx W_{\text{pretrained}} + BA, \quad B \in \mathbb{R}^{m \times r}, \; A \in \mathbb{R}^{r \times n}, \; r \ll \min(m, n).$

Instead of learning a full $m \times n$ update matrix during fine-tuning, you learn two thin matrices $B$ and $A$ whose product is rank-$r$. For Llama 3.1 70B with $r = 16$ on attention projections, this drops the trainable parameters from ~70B to ~30M — the reason you can QLoRA-tune a 70B model on a single 24GB consumer GPU.

The hypothesis — verified empirically across hundreds of papers 2021–2025 — is that the update $\Delta W$ needed for a downstream task is naturally low-rank, even if the pretrained $W$ is full-rank. In SVD language: the fine-tuning update lives in a $r$-dimensional subspace of the weight space.

7b.4 · 2024–2025 successors of LoRA

• DoRA (Weight-Decomposed LoRA, Liu et al., 2024) — decompose $W$ into magnitude and direction, LoRA only the direction. Nearly matches full fine-tuning at LoRA cost.
• QLoRA (Dettmers et al., 2023) — quantize the frozen backbone to 4-bit NF4, LoRA on top in bf16. Standard 2024–2025 recipe.
• VeRA (Kopiczko et al., 2024) — share random $B$ and $A$ across layers, learn only tiny scaling vectors. Even fewer trainable params than LoRA.
• LoftQ, PiSSA, LoRA-XS — initialize $B$ and $A$ from the SVD of $W$ so you start at a smart place rather than random.

All of these ride the same theorem you saw in 7b.3.

Further reading:

• Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models" (2021).
• Dettmers et al. "QLoRA: Efficient Finetuning of Quantized LLMs" (2023).
• Liu et al. "DoRA" (2024).
• 3B1B “Eigenvectors and eigenvalues” — the geometric intuition, 17 minutes.

8 · A worked example — recommending movies

Let's put every idea together. Say we have 4 users and 5 movies. Each user has a learned "taste vector" in some latent space of dimension 3 (in reality it'd be 32 or 128, but 3 is easier to see):

users = np.array([
    [ 1.0,  0.2,  0.0],   # user 0 — likes action
    [ 0.1,  1.0,  0.3],   # user 1 — likes drama
    [ 0.3,  0.9,  0.4],   # user 2 — like user 1
    [-0.5, -0.1,  1.0],   # user 3 — likes documentaries
])   # shape (4, 3)
 
movies = np.array([
    [ 1.0,  0.0,  0.0],   # movie 0 — pure action
    [ 0.0,  1.0,  0.0],   # movie 1 — pure drama
    [ 0.5,  0.5,  0.0],   # movie 2 — action-drama
    [ 0.0,  0.2,  1.0],   # movie 3 — documentary
    [ 0.8,  0.3, -0.2],   # movie 4 — action with a hint of drama
])   # shape (5, 3)

To score every user against every movie:

scores = users @ movies.T          # (4, 3) @ (3, 5) = (4, 5)
print(scores.round(2))

Row i of scores gives you user i's predicted rating for every movie. To recommend the top-2 for user 1:

top_movies_for_user_1 = np.argsort(scores[1])[::-1][:2]
print(top_movies_for_user_1)

This is the entire architecture of a matrix-factorization recommender. Netflix's original Netflix Prize model was, at its core, this exact operation with much bigger matrices. The "learning" part is figuring out what the numbers in users and movies should be, by fitting them to observed ratings — which is what neural networks and gradient descent will do for us starting in Session 007.

9 · Vectors in ML — a mental map

Here's the diagram I want on your wall for the next 78 sessions:

Every neural network — MLP, CNN, RNN, transformer — is some choreographed dance of matrix multiplies and non-linearities. Once you own §5–§7 of this session, you own the "matrix multiply" half. Session 007 gives you the "non-linearity" half.

9b · Embedding models in the wild — what “dot product” looks like in 2025

We've spent an hour saying “the dot product is the operation”. Now let's ground that in the models people actually deploy this year. If you skim any of the top embedding models on the MTEB leaderboard (Massive Text Embedding Benchmark, updated continuously through 2025), the top slots as of mid-2025 rotate between:

• NV-Embed-v2 (NVIDIA, Aug 2024) — 4096-dim embeddings, Mistral-7B backbone, 69.3 MTEB score on release.
• stella_en_1.5B_v5 (2024) — 8192-dim, uses Matryoshka Representation Learning so you can truncate to 512 dims and lose almost nothing.
• BGE-M3 (BAAI, 2024) — multi-lingual, three modes (dense, sparse, multi-vector).
• gte-Qwen2-7B-instruct (2024) — first Qwen2-based dense embedder, 4096 dim.
• Gemini Embedding (Google, 2025) — API-only, tops MTEB v2 for retrieval.

Every one of them, no matter the backbone, produces a vector v ∈ ℝ^d for each text. Retrieval is exactly the operation we did in §8's movie example: scores = Q @ D.T where D is (num_docs, d) and Q is (num_queries, d). That's it. Billions of dollars of infrastructure sit on top of that one line.

9b.1 · Matryoshka embeddings — dot products at every scale

A beautiful 2024 idea, “Matryoshka Representation Learning” (Kusupati et al., 2022; refined and widely adopted 2024): train the embedding so that the first 64 dimensions are already a decent embedding, the first 256 are better, the first 1024 are best. Then at retrieval time you pick the dimension you can afford.

Why does this even work? Because the dot product is linear:

$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{d} a_i b_i = \underbrace{\sum_{i=1}^{64} a_i b_i}_{\text{first-64 dot product}} + \underbrace{\sum_{i=65}^{d} a_i b_i}_{\text{rest}}$

So truncating both vectors to their first 64 dims just drops the second sum. If training pressures the model to put the most important “alignment signal” in those first dims, the truncated dot product is a good approximation of the full one. In production this lets you store (N, 64) int8 vectors in RAM for a fast first-pass filter, then re-rank the top-1000 with the full (N, 4096) float16 vectors on disk. Same math, three orders of magnitude cheaper.

9b.2 · ColBERT and late interaction — many dot products per query

ColBERT (Khattab & Zaharia, 2020) and its 2024 successor ColBERTv2/PLAID had a heretical idea: instead of squishing each document to a single vector, keep one vector per token. Then the query-document score is:

$\text{score}(q, d) = \sum_{i=1}^{|q|} \max_{j \in [1, |d|]} \; q_i \cdot d_j$

Read that: for each query token, find its best-matching document token (by dot product), then sum those best scores. It's MaxSim. It's still dot products. Just more of them. This is one of the ideas underneath modern hybrid retrieval systems used in production RAG stacks in 2025 — BGE-M3's multi-vector mode does exactly this.

9b.3 · Vector databases — dot products, but fast

When you have 100M documents and a query has to return the top-10 in <20ms, you can't afford to do Q @ D.T naively. This is why FAISS, ScaNN, HNSW, and vector DBs like Qdrant / Milvus / Weaviate / pgvector exist. They use approximate nearest neighbour indexes (IVF, HNSW graphs, product quantization) so you only compute the dot product against ~0.1% of the corpus per query. The math is still the dot product; the engineering is avoiding most of them.

A 2025 note: DiskANN's SPANN successor and Microsoft's ONNXruntime + FAISS on-disk indexes let you serve billion-scale retrieval from a single node with ~50ms p99. If you're building a RAG system in 2026, you'll touch one of these.

9b.4 · Modern attention is many small dot products, batched

The move from S002 to S036 (self-attention) is not a leap. It's a substitution. In retrieval:

• Q = matrix of query embeddings, one row per query.
• D = matrix of document embeddings, one row per document.
• scores = Q @ D.T = every-query-against-every-document.

In attention inside a transformer, at layer $\ell$:

• Q = matrix of token query vectors, one row per token in the sequence.
• K = matrix of token key vectors, one row per token.
• scores = Q @ K.T / sqrt(d_k) = every-token-against-every-token.

Same operation. The only difference is that in a transformer the queries and keys are both derived from the same input sequence (hence “self”-attention), and the scaling factor $1/\sqrt{d_k}$ (from Vaswani et al., 2017) keeps the softmax from saturating when d_k is large. That's it. If you groked §8's movie example, you have already met attention. We'll formalize the rest in M07.

Further reading:

• Kusupati et al. "Matryoshka Representation Learning" (2022; the paper that changed how 2024–2025 embedders are trained).
• Muennighoff et al. "MTEB: Massive Text Embedding Benchmark" (2022, still the leaderboard everyone chases in 2025).
• Nils Reimers' state-of-the-art embeddings series — practical, updated regularly.

10 · War stories

similarity = query * key       # I meant to compute a dot product

scores = Q @ K       # shape (batch, seq, seq)? No — ERROR

11 · Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: Redo §8's movie example from memory tomorrow. Revisit §6 (matmul shape rule) and §9b.4 (attention-as-retrieval) on day 7.

Next session (S003): Broadcasting — the invisible rule that made X @ W + b work despite X @ W being (32, 128) and b being (128,).

Sticky note: scores = Q @ K.T / np.sqrt(d) — attention in one line. Read as: "how aligned is each query with each key, rescaled so softmax doesn't saturate."

Recall — no scrolling

Stretch — for one extra hour

Load the GloVe 50d embeddings (or any pretrained embeddings you can grab) into a (V, 50) NumPy array. Then:

1. Find the top-5 nearest neighbours (by cosine similarity) of the word "king".
2. Find the top-5 nearest neighbours of the vector king - man + woman. What do you get?
3. Find the top-5 nearest neighbours of paris - france + japan.

You should see the classical analogy results (queen for #1, tokyo-ish results for #3). Congratulations — you just did the demo that made everyone lose their minds about word embeddings in 2013.

In your own words

Explain in one sentence to a friend what the dot product actually measures:

Spaced-review pointer

• From S001 — remember the (batch, seq, d) shape image? Every attention op you'll see this series uses the ideas from S001 §4 (axes) and S001 §5.2 (adding None axes) on top of what we did today.
• This session connects directly to S036 (Self-Attention Derived) — go back to §7 of this file when we get there. Q @ K.T / sqrt(d) will be your friend.

Next-session teaser

Now that we can @-multiply matrices and predict shapes, one shape-related rule was actually hiding all through today's examples: broadcasting. When we wrote X @ W1 + b1, X @ W1 had shape (32, 128) and b1 had shape (128,) — and somehow they added. That's broadcasting doing invisible work.

In Session 003 we drag broadcasting into the light. We derive the exact three rules NumPy uses, work through every gotcha, and by the end you'll predict any broadcast shape without running the code. This one session, more than any other in M01, is the difference between "shape errors take me 20 minutes" and "shape errors take me 20 seconds".

What to bring back tomorrow — sticky note

• Diagram: the arrow-with-cos-θ picture of the dot product.
• Equation: (m, k) @ (k, n) → (m, n). Inner match, outer survive.
• Snippet: scores = Q @ K.T / np.sqrt(d) — attention in one line.

Previous: ← DL S001 · NumPy Warm-up · Next: DL S003 · Broadcasting →