S017 · Linear Algebra I — Vectors, Dot Product, Geometry
The language ML speaks: vectors as arrows and lists of numbers, the dot product as similarity, norms as length, and cosine similarity as ‘how alike?’
🎯 Read any ML paper's ‘we compute the dot product between the query and key vectors’ sentence without stalling — and implement the same in NumPy from first principles.
Why this session exists
Every embedding model, every recommendation engine, every LLM attention head boils down to the same operation: take two vectors, compute their dot product, and interpret the number as "how similar are they?" Understand that one operation geometrically — as a projection, as an angle, as a length-scaled cosine — and 80% of ML notation becomes readable. You don't need to derive it from the axioms; you need to feel it as an arrow. That's what this session builds.
- Explain a vector two ways: as an arrow in space and as a list of numbers — and switch between the views fluidly.
- Compute a dot product by hand for 3D vectors, and predict its sign (positive / zero / negative) from the geometry.
- Compute vector length (L2 norm) and turn any vector into a unit vector.
- Compute cosine similarity between two vectors and use it as a similarity metric.
- Recognise vectors in unusual clothing: RGB pixels, one-hot encodings, word embeddings, feature rows.
Prerequisites
- S015 · Big-O — you'll analyse vector-op complexity.
- High-school algebra + Pythagoras. Recognise
sqrt(x² + y²)and don't run away.
(a) Intuition · 5 min
Walk 3 blocks east and 4 blocks north. That's a single motion — a vector — with two equivalent descriptions: the pair of numbers (3, 4), and the arrow drawn from your starting point to where you end up. The arrow's length is 5 blocks (Pythagoras). Its direction is roughly northeast.
Now add another vector: 2 east, 1 north. Total motion: (5, 5). You've just done vector addition. If a friend walks in your direction but only "0.6 as far," they walked (0.6·5, 0.6·5) = (3, 3). That's scalar multiplication.
In code, a vector is a list of numbers. Addition is element-wise. Scalar multiplication multiplies every element. That's it. The magic isn't the operations; it's what the list of numbers represents — an RGB colour, a movie's genre scores, a word's meaning after embedding, a user's feature row.
Once you accept "list of numbers" and "arrow in space" as the same object, ML notation stops being intimidating.
Three things a vector is, in three domains
- Geometrically — an arrow from the origin (or any base point) to a target point. Has length and direction.
- Algebraically — an ordered list of numbers [x₁, x₂, …, xₙ]. Lives in ℝⁿ.
- Computationally — a 1-D NumPy array `np.array([1, 2, 3])`. Shape (n,), dtype float.
A quick history
- 1637Descartes · coordinate geometryRené Descartes reduces geometry to algebra by inventing (x, y) coordinates.
- 1844Grassmann · vector algebraHermann Grassmann formalises vectors as objects that add and scale. Ignored for 50 years.
- 1901Gibbs & Heaviside · dot & cross productsPhysicists standardise notation. Every physics and engineering student since has seen these.
- 1958Perceptron · RosenblattFirst learning algorithm — literally a dot product against a weight vector, plus a threshold.
- 2013word2vec · MikolovWords become vectors. Semantic similarity becomes dot product. Modern NLP begins.
- 2017Attention is All You NeedTransformers: attention weights ARE scaled dot products of query and key vectors. Modern LLMs.
(b) Visual walkthrough · 15 min
The geometric meaning of a dot product
The five operations you'll do 10 000 times
Vector operations · Python + geometric meaning
Dot product two ways — the same number
How you compute it
- u·v = u₁v₁ + u₂v₂ + … + uₙvₙ
- O(n) elementary operations
- Trivially parallelisable — SIMD, GPU
- What NumPy `u @ v` calls under the hood
What it means
- u·v = ||u|| · ||v|| · cos(θ)
- || || is length, θ is angle between them
- Sign tells direction agreement
- Value depends on lengths — normalise for pure ‘similarity’
Cosine similarity — the ML workhorse
Element-wise multiply, sum.
cos_sim = (u·v) / (||u|| · ||v||). Ranges from -1 (opposite) to +1 (identical direction).
1 = identical, ~0 = unrelated, negative = anti-correlated. In text embeddings, real-world sim is usually 0.3–0.9.
If you'll query many vectors against one query, unit-normalise them once. Then cosine sim = plain dot product.
Where you'll see vectors in real code
Colour as a 3-vector
- [255, 128, 0] = orange
- Length ≈ 285 (arbitrary — just brightness)
- Dot with [1,0,0] extracts red channel
Class label as a vector
- ‘dog’ = [0,1,0,0] of 4 classes
- Length = 1 always
- Dot with logits picks that class's score
Meaning as a 768-vector
- BERT: each word → 768 floats
- ‘king’ - ‘man’ + ‘woman’ ≈ ‘queen’
- Cosine sim between embeddings ranks synonyms
One row of a dataset
- [age, income, clicks_last_week, …]
- Rows of the same table live in the same ℝⁿ
- Distance between rows = user similarity
"Cosine similarity measures how close two vectors are. High cosine similarity means the embeddings are near each other — it's basically normalised distance."
Cosine similarity measures angle only, discarding magnitude entirely. Two vectors pointing the same way have cosine 1 whether one is 100× longer than the other. Euclidean distance measures position. They agree only when all vectors are unit-normalised — and disagree sharply when they aren't.
Because in practice most embedding models emit vectors of roughly similar norm, and many pipelines normalise before indexing, so the two metrics rank results almost identically. You can work with embeddings for a long time before the distinction bites. It bites when magnitude carries real information: raw TF-IDF vectors where a long document has a large norm, count vectors where magnitude means volume, or a model whose norm correlates with confidence or token count. Then cosine says "these are the same topic" while Euclidean says "these are very different amounts of it" — and which one is right depends entirely on whether you care about direction or degree. The deeper trap is that switching metrics after building an index changes your recall set silently; nothing errors.
Identical direction, wildly different magnitude — the two metrics disagree completely:
import numpy as np
a = np.array([1.0, 1.0])
b = np.array([100.0, 100.0])
c = np.array([1.0, 1.1])
cos = lambda u, v: u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
print(cos(a, b), np.linalg.norm(a - b)) # 1.0 140.0 <- 'identical', far
print(cos(a, c), np.linalg.norm(a - c)) # ~0.9989 0.1 <- 'less similar', nearWhy does the dot product measure similarity at all? sum(a_i * b_i) is just an arithmetic sum of products — derive why it has anything to do with angle.
- 1Each term
a_i * b_iis positive when both vectors agree in sign on dimension i, and negative when they disagree.forced by · the product of two same-signed numbers is positive; opposite signs give a negative - 2So the sum is a signed tally of agreement across dimensions, weighted by how strongly each vector expresses each dimension.forced by · large components contribute more to the sum than small ones
- 3Independently, the law of cosines applied to the triangle formed by a, b, and a−b gives
|a−b|² = |a|² + |b|² − 2|a||b|cos θ.forced by · this is plain Euclidean geometry, with no reference to coordinates - 4Expanding
|a−b|²in coordinates gives|a|² + |b|² − 2·Σ a_i b_i. Setting the two expressions equal forcesΣ a_i b_i = |a||b| cos θ.forced by · the same quantity computed two ways must agree, and everything else cancels - 5Therefore dividing the dot product by both norms isolates
cos θ— a pure measure of direction, bounded in [−1, 1] regardless of scale.forced by · the magnitudes were the only scale-dependent factors in the identity
Therefore the coordinate formula and the geometric meaning are the same object: the dot product is simultaneously an agreement tally and a projection length. That is why one cheap operation serves as a similarity score, a projection, and an orthogonality test.
And note what this predicts, all verifiable: dot product zero ⇔ perpendicular ⇔ "no shared information" — which is why orthogonality means independence in PCA. It predicts that if you pre-normalise every vector to unit length, cosine similarity collapses to a plain dot product, which is exactly why vector databases store normalised embeddings and use inner-product search: it is the same ranking for less arithmetic. And it predicts that maximising a dot product with fixed norms is identical to minimising Euclidean distance, so under normalisation the two search problems are provably the same problem.
A vector is an arrow: it has a direction (what it's about) and a length (how much of it). Every operation you will use touches one or both. Normalising discards length and keeps meaning. The dot product asks "how much do these two agree?" and returns a single number that is positive for aligned, zero for unrelated, negative for opposed.
An embedding is this idea taken to hundreds of dimensions: each axis is some learned feature, the direction is the semantic content, and similarity is angular agreement. Everything else — nearest-neighbour search, clustering, recommendations, attention scores — is dot products with bookkeeping around them.
- Normalise before comparing, unless magnitude is genuinely part of what you mean. Decide this once and enforce it at write time, not per query.
- Dot product = 0 means orthogonal means "carries no shared information". This is why PCA components are orthogonal and why it makes them independently interpretable.
- In high dimensions, intuition fails: distances concentrate, so almost all random pairs are nearly equidistant and nearly orthogonal. That is the curse of dimensionality, and it is why raw Euclidean distance degrades as dimensions grow.
- A matrix–vector product is just a batch of dot products. If you can see that,
X @ win a model stops being notation and becomes "score every row against this direction".
Fire this model the moment you see: cosine vs. Euclidean in a vector DB config · embeddings that must be normalised before indexing · an attention score · a similarity search returning suspiciously uniform scores · PCA output · a recommender scoring items by dot product.
You are building similarity search over millions of embeddings. Exact nearest neighbour, approximate (HNSW/IVF), or reduce dimensions first?
ef_search, nprobe) you can tune per workloadStart exact, because you need it as ground truth. You cannot evaluate an approximate index without an exact baseline to measure recall against, so building it first is not wasted work — it is the measuring instrument.
Then move to an ANN index when latency demands it, and treat the recall dial as a product decision rather than an infrastructure one: the honest question is "how many results per thousand queries may be wrong?", and that question has an owner outside engineering. The failure mode to avoid is shipping an ANN index with default parameters and no recall measurement at all — the system will look fast and healthy while quietly missing the answers, and no alert will ever fire.
(c) Hands-on · 25 min
You'll implement vectors from scratch, verify the dot-product identity, and build a tiny "find similar item" using cosine similarity. Save as vectors_demo.py.
Run it:
uv run --with numpy python vectors_demo.pyExpected output:
What each block does
Anatomy of the code
Compute the projection of u = [3, 4] onto v = [1, 0]:
u = [3.0, 4.0]
v = [1.0, 0.0]
# proj_v(u) = ( (u·v) / (v·v) ) * v
# Since v is a unit vector, v·v = 1, so proj = (u·v) * v
k = dot(u, v) # k = u·v = 3
projection = scale(k, unit(v))
print(projection) # [3.0, 0.0]Now do the same for v = [1, 1] (the 45° direction). Predict the answer geometrically before running: u projected onto the 45° line should have length equal to how far along the 45° direction u ‘leans’. Check.
(d) Production reality · 15 min
Spotify's ‘Discover Weekly’ needs to rank millions of tracks against each user's taste vector, every day. Naive: for each user, dot-product against every track vector — n_users × n_tracks operations = 10¹² per refresh. Days of compute.
Cosine similarity is O(d) per pair. But d is 40-200 and n_pairs is astronomical.
annoy: build a forest of random-projection trees, query in O(log n) instead of O(n). Latency dropped from hours to milliseconds per user. The same trick powers Pinterest lens, Meta faiss, and every modern vector DB.text-embedding-ada-002 for their corpus, then swaps to text-embedding-3-large (1536 → 3072 dimensions) for new documents. Cosine sim between old and new vectors is now a dimension mismatch error at best, or (if they resize somehow) garbage rankings at worst.np.testing.assert_allclose(rtol=1e-6), not exact equality. Understand that float ops are not associative — (a+b)+c != a+(b+c) in general — so the order NumPy/GPU chose subtly changes the answer.Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- What is a vector, in two sentences?
- What does the sign of a dot product tell you about the two vectors?
- Why do RAG systems use cosine similarity instead of ‘just checking if strings match’?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.