S017 · Linear Algebra I — Vectors, Dot Product, Geometry

Module M02: Math Foundations · Session 17 of 130 · Track: Math ➡️

What you'll be able to do after this session

• Explain Linear Algebra I — Vectors, Dot Product, Geometry to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S015 — Big-O Notation — Reasoning About Scale

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · 3Blue1Brown — Vectors, what even are they? — the canonical geometric intuition video. Watch this even if you already know vectors.
• 🎥 Deep dive (30–60 min) · 3Blue1Brown — Essence of Linear Algebra, Ch 1-3 — vectors, linear combinations, spans, basis.
• 🎥 Hands-on demo (10–20 min) · StatQuest — Dot Product Clearly Explained — dot product with worked examples.
• 📖 Canonical article · Wikipedia — Euclidean vector — reference for notation and identities.
• 📖 Book chapter / tutorial · Deep Learning Book, Chapter 2 — Linear Algebra (Goodfellow, Bengio, Courville) — the ML-oriented crash course.
• 📖 Engineering blog · OpenAI — Vector embeddings and what makes them useful — why every LLM app is now a linear-algebra app.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The language ML speaks.

A vector is just a list of numbers with a direction interpretation. [3, 4] is "3 steps east, 4 steps north" — it points to a specific spot on a map. Add two vectors and you add the trips together. Scale a vector by 2 and you go twice as far in the same direction. That's it — that's the whole idea.

Why do we care? Because in machine learning, everything is a vector. A word like "king" becomes a 1536-dimensional vector (an OpenAI embedding). A user's music taste is a 200-dim vector of genre preferences. An image is a 224×224×3 = 150 528-dim vector. Once your objects are vectors, questions like "how similar are these two things?" become geometric: compute the angle between the vectors. That's the dot product — one number that tells you if two vectors point the same way (positive), opposite ways (negative), or perpendicular (zero).

The forever-gotcha: vectors have both magnitude (length) and direction. For similarity, direction matters most — that's why we usually normalise (divide by length) and compare via cosine similarity rather than raw dot product. Otherwise longer vectors "win" every comparison, which is a bias, not a feature.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A vector in 2D is an arrow from origin to a point:

Length (magnitude / norm): ||v|| = √(3² + 4²) = 5. Pythagoras. Same formula in any number of dimensions: √(Σ vᵢ²).

Vector operations at a glance:

Worked example — dot product as "alignment":

• a = [1, 0] (points east)
• b = [1, 0] (also east) → a·b = 1·1 + 0·0 = 1. Same direction, cos = 1.
• c = [0, 1] (north) → a·c = 0. Perpendicular, cos = 0.
• d = [-1, 0] (west) → a·d = -1. Opposite, cos = -1.

That single number (between -1 and 1 after normalising) is what powers semantic search: your query becomes a vector, every document is a vector, return the top-k by cosine similarity. This is how ChatGPT with RAG finds your relevant notes, how Spotify's "songs like this" works, how Netflix ranks recommendations.

Worked example — word embeddings (the famous one):

vec("king")   - vec("man")   + vec("woman")   ≈  vec("queen")
vec("Paris")  - vec("France") + vec("Italy")  ≈  vec("Rome")

These are 300-dim vectors trained by Word2Vec. The fact that arithmetic on them captures analogies is what convinced the field that dense vector representations were the future. Modern LLM embeddings (1536-dim from OpenAI, 4096 from Cohere) do the same trick at much higher fidelity.

Cosine vs Euclidean distance:

• Cosine similarity = 1 - cos(angle). Cares only about direction. Good for text/embeddings.
• Euclidean distance = ||a - b||. Cares about position. Good for physical coordinates, k-means clustering.

Choose based on what "similar" means in your problem.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

"""Vectors from scratch, then with NumPy, then a mini semantic-search demo."""
import math
import numpy as np
 
# ---- 1. From scratch: understand every step ------------------------
def dot(a: list[float], b: list[float]) -> float:
    assert len(a) == len(b), "dimension mismatch"
    return sum(x*y for x, y in zip(a, b))
 
def norm(a: list[float]) -> float:
    return math.sqrt(dot(a, a))
 
def cosine(a: list[float], b: list[float]) -> float:
    return dot(a, b) / (norm(a) * norm(b))
 
a = [3.0, 4.0]
b = [4.0, 3.0]
print(f"a·b = {dot(a, b)}, |a| = {norm(a)}, cos(a,b) = {cosine(a, b):.4f}")
 
# ---- 2. NumPy: the way you'll actually write it --------------------
A = np.array([3.0, 4.0])
B = np.array([4.0, 3.0])
print(f"numpy dot = {A @ B}, |A| = {np.linalg.norm(A):.4f}")
print(f"cosine = {A @ B / (np.linalg.norm(A)*np.linalg.norm(B)):.4f}")
 
# ---- 3. Toy semantic search on 4-D 'embeddings' --------------------
docs = {
    "dog"    : np.array([0.9, 0.8, 0.1, 0.0]),   # animal, furry
    "cat"    : np.array([0.8, 0.9, 0.1, 0.0]),
    "car"    : np.array([0.0, 0.1, 0.9, 0.8]),   # vehicle
    "truck"  : np.array([0.0, 0.1, 0.8, 0.9]),
    "puppy"  : np.array([0.85, 0.85, 0.0, 0.0]),
}
 
def cos_sim(x, y):
    return float(x @ y / (np.linalg.norm(x) * np.linalg.norm(y)))
 
query = docs["dog"]
scores = sorted(
    ((word, cos_sim(query, v)) for word, v in docs.items() if word != "dog"),
    key=lambda t: -t[1],
)
print("\nmost similar to 'dog':")
for word, score in scores:
    print(f"  {word:6s} {score:+.3f}")
 
# ---- 4. Analogy: king - man + woman ≈ queen (fake but illustrative) -
# Real embeddings would use 1536 dims; the direction of the arithmetic is the same.

Observe when you run:

1. a·b = 24 matches by hand: 3·4 + 4·3 = 24. |a| = 5 (the classic 3-4-5 right triangle).
2. NumPy's @ (matrix-multiply operator, doubles as dot) matches the hand implementation exactly.
3. In the semantic-search block, puppy (0.99) and cat (0.98) score much higher than car (0.02) and truck (0.02) — the "animal" direction dominates.
4. np.linalg.norm and @ are ~100× faster than the pure-Python versions once your vectors are more than a few hundred dims.
5. Cosine similarity is bounded to [-1, 1]; that's why it composes cleanly across billions of comparisons in a vector database.

Try this modification: compute Euclidean distance instead of cosine and re-rank. Do the rankings change? Now normalise every vector to unit length first (v / np.linalg.norm(v)) and recompute Euclidean — you'll find it produces the same ranking as cosine (they're monotonic transforms once normalised).

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Every modern LLM application is a linear-algebra application. When you use ChatGPT with your PDFs (RAG), here's what actually happens:

1. Each chunk of your PDF is passed to an embedding model (OpenAI text-embedding-3-large returns a 3072-dim vector).
2. Vectors are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant, Milvus). Storage cost: 4 bytes × 3072 dims = ~12 KB per chunk.
3. Your query gets embedded the same way.
4. The DB returns the top-k chunks by cosine similarity. That's it — that's the "magic".

At Microsoft on the OneNote Copilot pipeline we do this at scale: ~200 TB of note content becomes ~10 billion embeddings, indexed with HNSW (Hierarchical Navigable Small World — approximate nearest-neighbour that's O(log n) instead of O(n)). Exact search over 10B vectors would take minutes; HNSW answers in ~5 ms with 95%+ recall.

Common gotchas:

• Dimension mismatch = every ML engineer's daily pain. Shape errors like (768,) vs (1, 768) vs (768, 1) will silently broadcast or loudly crash. Always print .shape before matmul.
• Not normalising before cosine — most vector DBs assume unit-normalised vectors. If you feed raw embeddings, distance metrics behave surprisingly. Normalise on write.
• Mixed embedding models — a vector from text-embedding-ada-002 cannot be compared to one from text-embedding-3-large. Different models = different vector spaces. Re-embed the whole corpus when you upgrade.
• Curse of dimensionality — in very high dims (>10 000) all points become roughly equidistant. Which is why we use 512–4096 dim embeddings, not 1M-dim ones.
• Memory blowup — 1B vectors × 3072 dims × 4 bytes = 12 TB. Solutions: quantisation (int8 → 4× smaller with ~1% quality loss), Product Quantisation, or Matryoshka embeddings (truncate to the first 512 dims — still works surprisingly well).

Fun fact: the "attention" mechanism in Transformers is fundamentally three matrix multiplies + one softmax, all built on dot products between vectors. Understanding this session lets you read the Transformer paper.

(e) Quiz + exercise · 10 min

1. What does a dot product of zero mean geometrically? What about positive vs negative?
2. What is the range of cosine similarity? Why is it preferred over raw dot product for comparing embeddings?
3. Compute by hand: given a = [1, 2, 2], b = [2, 1, 2], what are a·b, ||a||, ||b||, and cos(a,b)?
4. Why does "king - man + woman ≈ queen" work in word-embedding space? What structural property of the space allows this?
5. Given 1 billion 1024-dim float32 vectors, how much RAM does exact k-NN need? What's one technique to reduce that?

Stretch (connects to S015 Big-O): naive nearest-neighbour search over N vectors is O(N·d) per query. HNSW is O(d·log N). For N = 10⁹ and d = 1024, how many operations per query for each? At 1 ns per multiply, how long does each take?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Linear Algebra I — Vectors, Dot Product, Geometry? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S018): Linear Algebra II — Matrices, Transforms, Eigenvalues
• Previous (S016): Discrete Math — Sets, Logic, Combinatorics, Graphs
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.