Linear Algebra I — Vectors, Dot Product, Geometry

Review boundary and corrected contracts

Cosine is undefined for a zero vector. Orthogonality is geometric; uncorrelated PCA coordinates are not necessarily statistically independent. Dot and matmul differ above two dimensions. Removed universal ANN complexity/recall promises. Numeric checks use tolerances, not exact equality.

The examples below teach the stated input contracts; fragments and deliberately incorrect counterexamples are exercises, not production implementations. Complexity counts elementary operations unless a bit-cost or allocation caveat is stated. Practice times are suggestions, not measured performance or hiring guarantees.

Learning objective: Read any ML paper's ‘we compute the dot product between the query and key vectors’ sentence without stalling — and implement the same in NumPy from first principles.

Why this session exists

Dot products are central to many embedding scorers, recommenders and attention mechanisms. They are learned or chosen scores—not automatic measures of semantic similarity for every model. Understand that one operation geometrically — as a projection, as an angle, as a length-scaled cosine — and many common ML expressions become readable. You don't need to derive it from the axioms; you need to feel it as an arrow. That's what this session builds.

You will be able to

• Explain a vector two ways: as an arrow in space and as a list of numbers — and switch between the views fluidly.

• Compute a dot product by hand for 3D vectors, and predict its sign (positive / zero / negative) from the geometry.

• Compute vector length (L2 norm) and turn any nonzero vector into a unit vector.

• Compute cosine similarity between two nonzero vectors and use it as a similarity score (not a mathematical distance metric).

• Recognise vectors in unusual clothing: RGB pixels, one-hot encodings, word embeddings, feature rows.

Prerequisites

• S015 · Big-O — you'll analyse vector-op complexity.
• High-school algebra + Pythagoras. Recognise sqrt(x² + y²) and don't run away.

Optional archived videos

These original learning links are retained as optional resources. Video identity, availability and duration were not reverified; the lesson and governing sources below stand independently of them.

• Vectors · Chapter 1, Essence of Linear Algebra
• Dot products and duality · Chapter 9, Essence of Linear Algebra
• NumPy Crash Course

Pre-read · skim before the session (35 min total)

Further reading: Mathematics for Machine Learning (Chapter 3.1–3.3)

Further reading: Dot product

Further reading: NumPy Quickstart

(a) Intuition · 5 min

A vector is an arrow with a name and a length

Real: Walk 3 blocks east and 4 blocks north. That's a single motion — a vector — with two equivalent descriptions: the pair of numbers (3, 4), and the arrow drawn from your starting point to where you end up. The arrow's length is 5 blocks (Pythagoras). Its direction is roughly northeast. Now add another vector: 2 east, 1 north. Total motion: (5, 5). You've just done vector addition. If a friend walks in your direction but only "0.6 as far," they walked (0.6·5, 0.6·5) = (3, 3). That's scalar multiplication.

 
    In code, a vector is a list of numbers. Addition is element-wise. Scalar multiplication multiplies every element. That's it. The magic isn't the operations; it's what the list of numbers represents — an RGB colour, a movie's genre scores, a word's meaning after embedding, a user's feature row.
    Once you accept "list of numbers" and "arrow in space" as the same object, ML notation stops being intimidating.
  

Three things a vector is, in three domains

A vector is always all three of these

• Geometrically — an arrow from the origin (or any base point) to a target point. Has a length; a nonzero vector has a direction, while the zero vector does not.

• Algebraically — an ordered list of numbers [x₁, x₂, …, xₙ]. Lives in ℝⁿ.

• Computationally — a 1-D NumPy array np.array([1, 2, 3], dtype=float), shape (n,). Without dtype=float, integer input normally creates an integer dtype.

The single most-confused sign convention

For nonzero real vectors in the same Euclidean space, the dot product is positive at an acute angle, zero at a right angle, and negative at an obtuse angle. A raw dot product of −0.9 alone does not determine how nearly opposite they are: norms matter. Cosine −0.9 does determine an angle of about 154.16°. Zero has dot product zero with every vector but no angle.

A quick history

Coordinate geometry connects arrows to numbers; Grassmann’s algebra of extension and the later Gibbs/Heaviside vector tradition developed ways to add, scale and combine those objects. See the St Andrews history of linear spaces. A perceptron scores inputs with a weighted sum and a threshold; it was not the first learning algorithm of every kind. Mikolov and colleagues (2013) studied efficient word-vector representations; they did not invent all vector representations of language. Vaswani and colleagues (2017) introduced the Transformer: scaled query–key dot products are scores, while softmax converts those scores into attention weights. The earlier timeline’s priority and “ignored for 50 years” claims are not used as facts.

(b) From arrow alignment to similarity scores · 15 min

Calling a vector an arrow gives us two separate properties to compare: direction and length. The dot product mixes both; cosine similarity will separate them. First, use the angle between two nonzero arrows to predict the sign of their dot product.

The geometric meaning of a dot product

Read the branches as three reference angles, not three different algorithms. The lengths contribute a positive scale factor, leaving the cosine to determine whether the result is positive, zero, or negative.

The middle branch is the useful boundary: perpendicular nonzero vectors have zero dot product regardless of their lengths. On either side of that boundary the sign changes, while stretching an arrow changes the magnitude of the score. The operations below give you control over those separate effects.

The five operations you'll do 10 000 times

Vector operations · Python + geometric meaning

Name: addition · u + v

Desc: Element-wise. Geometric: stack v onto the tip of u. np.array([1,2]) + np.array([3,4]) = [4,6].

Name: scalar mul · 3 · v

Desc: Element-wise. Geometric: stretch v to 3× its length (same direction). Negative scalar flips direction.

Name: dot product · u · v

Desc: Sum of element-wise products: Σᵢ uᵢ·vᵢ. Geometric: projection of one onto the other × the other's length.

Name: L2 norm · ||v||

Desc: sqrt(v · v) = sqrt(Σ vᵢ²). The vector's length. Pythagoras generalised to n dimensions.

Name: unit vector · v̂ = v / ||v||

Desc: For nonzero v, same direction as v, length 1. Cosine similarity operates on unit vectors — makes ‘similarity’ scale-free.

Dot product two ways — the same number

You can now read the dot product in either direction: sum coordinate-wise products to compute it, or inspect lengths and angle to interpret it. The two columns describe one scalar, not competing similarity measures.

Name: Algebraic

Tagline: How you compute it

• u·v = u₁v₁ + u₂v₂ + … + uₙvₙ

• O(n) elementary operations

• Trivially parallelisable — SIMD, GPU

• What NumPy u @ v calls under the hood

Name: Geometric

Tagline: What it means

• u·v = ||u|| · ||v|| · cos(θ)

• || || is length, θ is angle between them

• Sign tells direction agreement

• Value depends on lengths — normalise for pure ‘similarity’

Before normalizing, work a 3D example by hand: u = [1,2,3], v = [4,−1,2]. Their dot product is 1·4 + 2·(−1) + 3·2 = 8. Their squared norms are 14 and 21, so cosine is 8/√294 ≈ 0.46657: an acute angle, even though the second coordinate contributes negatively. Coordinates must refer to the same axes and units; a similarity interpretation additionally needs a modelling rationale.

Cosine similarity — the ML workhorse

If two arrows keep their directions but one doubles in length, their dot product doubles too. For a comparison meant to ignore that scale change, divide out both lengths. This requires nonzero vectors—the zero vector has no direction to compare.

Compute dot product u · v

Desc: Element-wise multiply, sum.

Divide by product of norms

Desc: cos_sim = (u·v) / (||u|| · ||v||). Ranges from -1 (opposite) to +1 (identical direction).

Interpret the number

Desc: 1 means identical direction, not identical vectors; 0 means orthogonal, not statistically independent or semantically unrelated. Negative means an obtuse angle. Useful thresholds depend on model, preprocessing, corpus and task—there is no universal text-embedding score range.

For efficiency at scale, pre-normalise

Desc: If you'll query many vectors against one query, unit-normalise them once. Then cosine sim = plain dot product.

Where you'll see vectors in real code

Name: RGB pixel

Tagline: Colour as a 3-vector

• [255, 128, 0] = orange

• Length ≈ 285.3 in raw RGB coordinates; this is not perceptual brightness

• Dot with [1,0,0] extracts red channel

Name: One-hot

Tagline: Class label as a vector

• ‘dog’ = [0,1,0,0] of 4 classes

• Length = 1 always

• Dot with logits picks that class's score

Name: Word embedding

Tagline: Meaning as a 768-vector

• BERT-base: each contextual token representation has 768 coordinates; words may split into tokens and other models use other widths

• Some learned word-vector spaces exhibit analogy patterns such as king − man + woman; this is empirical, not a vector-space law

• Cosine can rank task-related neighbours; a good synonym ranking must be evaluated

Name: User features

Tagline: One row of a dataset

• [age, income, clicks_last_week, …]

• Rows of the same table live in the same ℝⁿ

• Distance depends on feature units, scaling and a task-specific definition of user similarity

The one sentence that unlocks embeddings

An embedding is a function that turns something (a word, an image, a user) into a vector such that semantically similar things end up close in vector space. This is a learned objective, not a guarantee: RAG and recommendation can also combine lexical retrieval, filters, graphs and rerankers. Test whether the selected metric retrieves relevant items.

“Close in vector space” still needs a precise definition before it becomes a search rule. The next example holds direction fixed while changing length, making the difference between angular similarity and physical distance impossible to miss.

Myth: "Cosine similarity measures how close two vectors are. High cosine similarity means the embeddings are near each other — it's basically normalised distance."

Truth: Cosine similarity measures angle only, discarding magnitude entirely. Two vectors pointing the same way have cosine 1 whether one is 100× longer than the other. Euclidean distance measures position. For unit vectors, squared Euclidean distance equals 2 − 2·cosine, so maximizing cosine and minimizing distance give the same ranking (not the same number). Equal candidate norms also give this ranking equivalence for a fixed nonzero query. Without norm control, rankings can disagree sharply.

Why: Some pipelines normalize vectors, making the ranking equivalence exact in real arithmetic. Similar norms alone do not guarantee an identical ranking, especially near ties. You can work with embeddings for a long time before the distinction bites. It bites when magnitude carries real information: raw TF-IDF vectors where a long document has a large norm, count vectors where magnitude means volume, or a model whose norm correlates with confidence or token count. Then cosine says "these are the same topic" while Euclidean says "these are very different amounts of it" — and which one is right depends entirely on whether you care about direction or degree. The deeper trap is that switching metrics after building an index changes your recall set silently; nothing errors.

Test: Identical direction, different magnitudes. Execute this independent NumPy example:

import numpy as np
a = np.array([1.0, 1.0])
b = np.array([100.0, 100.0])
c = np.array([1.0, 1.1])
 
cos = lambda u, v: u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
print(cos(a, b), np.linalg.norm(a - b))   # 1.0      140.007  <- same direction, far
print(cos(a, c), np.linalg.norm(a - c))   # ~0.9989    0.1  <- 'less similar', near

Question: Why does the dot product measure similarity at all? sum(a_i * b_i) is just an arithmetic sum of products — derive why it has anything to do with angle.

Premise: Each term a_i * b_i is positive when both vectors agree in sign on dimension i, and negative when they disagree.

Because: the product of two same-signed numbers is positive; opposite signs give a negative

Premise: So the sum is a signed tally of agreement across dimensions, weighted by how strongly each vector expresses each dimension.

Because: large components contribute more to the sum than small ones

Premise: Independently, the law of cosines applied to the triangle formed by a, b, and a−b gives |a−b|² = |a|² + |b|² − 2|a||b|cos θ.

Because: this is plain Euclidean geometry, with no reference to coordinates

Premise: Expanding |a−b|² in coordinates gives |a|² + |b|² − 2·Σ a_i b_i. Setting the two expressions equal forces Σ a_i b_i = |a||b| cos θ.

Because: the same quantity computed two ways must agree, and everything else cancels

Premise: Therefore dividing the dot product by both norms isolates cos θ — a pure measure of direction, bounded in [−1, 1] regardless of scale.

Because: the magnitudes were the only scale-dependent factors in the identity

Orthogonality means zero dot product for real vectors. For centered random coordinates, zero covariance means uncorrelated, not necessarily independent. PCA produces uncorrelated components; independence requires additional assumptions such as joint Gaussianity.

Name: Direction, magnitude, and agreement

Picture: A vector is an arrow: it has a direction (what it's about) and a length (how much of it). Every operation you will use touches one or both. Normalising discards length; whether it preserves useful meaning is an empirical modelling decision. The dot product asks "how much do these two agree?" and returns a single number that is positive for acute angles, zero for orthogonality, negative for obtuse angles. An embedding is this idea taken to hundreds of dimensions: coordinates are learned features, not necessarily interpretable concepts. Direction-based similarity is one design choice, while norms can carry information. Nearest-neighbour search, clustering, recommendation and attention require additional objectives and algorithms.

• Normalise before comparing, unless magnitude is genuinely part of what you mean. Decide this once and enforce it at write time, not per query.

For example, let X be uniform on −1, 0, 1 and Y = X². E[X] = E[XY] = 0, so covariance is zero, but Y is determined by X. This is a concrete uncorrelated-but-dependent pair.

• Under models such as independent isotropic coordinates, high-dimensional random directions concentrate near orthogonality. Arbitrary data need not behave that way: repeating one direction stays parallel in any dimension. Irrelevant/noisy features can dilute useful distances; dimension alone does not prove retrieval failure.

• A matrix–vector product is just a batch of dot products. If you can see that, X @ w in a model stops being notation and becomes "score every row against this direction".

Trigger: Fire this model the moment you see: cosine vs. Euclidean in a vector DB config · embeddings that must be normalised before indexing · an attention score · a similarity search returning suspiciously uniform scores · PCA output · a recommender scoring items by dot product.

Decision: You are building similarity search over millions of embeddings. Exact nearest neighbour, approximate (HNSW/IVF), or reduce dimensions first?

Name: Exact brute-force search

Gain: perfect recall by construction, no index to build or tune, and one matrix multiply is extremely well optimised — for modest corpora on a GPU it is genuinely fast

Cost: cost is linear in corpus size per query, so it scales with the product of queries and documents; latency grows as your data does, with no knob to trade against it

Pick: the corpus is small enough that a full scan meets your latency budget, or the exact metric’s top-k neighbours are required, with an explicit tie policy — evaluation harnesses and offline scoring

Name: Approximate NN index (HNSW, IVF, PQ)

Gain: potentially fewer distance evaluations than a full scan; parameters such as ef_search or nprobe trade candidate work against measured recall. No universal sublinear or flat-latency guarantee follows for arbitrary data

Cost: possible missed exact neighbours, plus index build and memory overhead. Compare against an exact-neighbour baseline; relevance labels answer a different question. Update/deletion support depends on the implementation

Pick: online serving where p99 latency is a hard requirement and a small recall loss is acceptable — when measured scan cost exceeds the serving budget, not at a universal corpus-size cutoff

Name: Dimensionality reduction first (PCA, or a smaller model)

Gain: every downstream operation gets cheaper — less memory, faster distance computation, and possibly a different neighbour ranking; low-variance directions can contain signal, so validate the projection

Cost: irreversible information loss; a second artefact (the projection) to version and keep in sync with the query path; the reduction itself must be fit and can drift from the data

Pick: dimensionality is high relative to the intrinsic structure and you have measured that recall holds after projection — never applied blind

Verdict: Start exact, because you need it as ground truth. You cannot evaluate an approximate index without an exact baseline to measure recall against, so building it first is not wasted work — it is the measuring instrument.Then move to an ANN index when latency demands it, and treat the recall dial as a product decision rather than an infrastructure one: the honest question is "how many results per thousand queries may be wrong?", and that question has an owner outside engineering. The failure mode to avoid is shipping an ANN index with default parameters and no recall measurement at all — the system will look fast and healthy while quietly missing the answers, and no alert will ever fire.

(c) Hands-on · 25 min

You'll implement vectors from scratch, verify the dot-product identity, and build a tiny "find similar item" using cosine similarity. Save as vectors_demo.py.

Contract: equal-length, one-dimensional real finite inputs; cosine and unit direction require nonempty nonzero vectors. The educational sum-of-squares implementation is for moderate magnitudes where intermediate arithmetic stays finite and nonzero. Huge values can overflow and tiny ones can underflow; clamping a cosine before acos repairs small rounding excursions, not bad inputs. A production boundary should reject mismatched dimensions and nonfinite data explicitly. Complex vectors need a conjugate inner product, which this demo does not implement.

"""vectors_demo.py — vectors from scratch + NumPy check + a tiny similarity search."""
from __future__ import annotations
from math import acos, degrees, sqrt
import numpy as np
 
Vector = list[float]
 
# ── 1. From-scratch operations (educational) ─────────────────────────────
def add(u: Vector, v: Vector) -> Vector:
    if len(u) != len(v):
        raise ValueError("vectors must have the same dimension")
    return [a + b for a, b in zip(u, v)]
 
def scale(k: float, v: Vector) -> Vector:
    return [k * x for x in v]
 
def dot(u: Vector, v: Vector) -> float:
    if len(u) != len(v):
        raise ValueError("vectors must have the same dimension")
    return sum(a * b for a, b in zip(u, v))
 
def norm(v: Vector) -> float:
    return sqrt(dot(v, v))
 
def unit(v: Vector) -> Vector:
    n = norm(v)
    if n == 0:
        raise ValueError("zero vector has no direction")
    return scale(1.0 / n, v)
 
def cosine_similarity(u: Vector, v: Vector) -> float:
    denominator = norm(u) * norm(v)
    if denominator == 0:
        raise ValueError("cosine is undefined for a zero vector")
    return dot(u, v) / denominator
 
def angle_between_degrees(u: Vector, v: Vector) -> float:
    # Clamp for float safety before acos
    c = max(-1.0, min(1.0, cosine_similarity(u, v)))
    return degrees(acos(c))
 
# ── 2. Sanity checks against NumPy ───────────────────────────────────────
def compare_to_numpy() -> None:
    rng = np.random.default_rng(42)
    for _ in range(1000):
        n = rng.integers(2, 20)
        u = rng.normal(size=n).tolist()
        v = rng.normal(size=n).tolist()
        assert abs(dot(u, v) - float(np.dot(u, v))) < 1e-9
        assert abs(norm(u) - float(np.linalg.norm(u))) < 1e-9
    print("1000 random vectors match NumPy within the stated tolerance ✅")
 
# ── 3. Verify the algebra ↔ geometry equivalence ─────────────────────────
def verify_geometric_identity() -> None:
    """u·v == ||u|| ||v|| cos(θ)."""
    u = [3.0, 4.0]     # length 5, along ~53° from x-axis
    v = [1.0, 0.0]     # unit x-axis
    algebraic = dot(u, v)
    # The angle is known independently from the right triangle, not from dot().
    theta = np.arctan2(4.0, 3.0)
    geometric = norm(u) * norm(v) * np.cos(theta)
    assert abs(algebraic - geometric) < 1e-12
    print(f"u·v={algebraic}  angle={angle_between_degrees(u,v):.2f}°")
 
# ── 4. A tiny ‘find similar item’ using cosine similarity ────────────────
def demo_similarity_search() -> None:
    """Movie ‘embeddings’ — hand-tuned for [action, romance, comedy] scores."""
    movies = {
        "The Matrix":         np.array([0.95, 0.05, 0.20]),
        "Die Hard":           np.array([0.90, 0.10, 0.30]),
        "Notting Hill":       np.array([0.10, 0.90, 0.60]),
        "When Harry Met Sally":np.array([0.05, 0.85, 0.75]),
        "The Hangover":       np.array([0.30, 0.20, 0.95]),
        "Superbad":           np.array([0.25, 0.15, 0.95]),
    }
    def cos(a, b):
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
    query = "The Matrix"
    q_vec = movies[query]
    scored = sorted(
        ((name, cos(q_vec, v)) for name, v in movies.items() if name != query),
        key=lambda p: -p[1],
    )
    print(f"\nMovies most similar to «{query}»:")
    for name, score in scored:
        bar = "█" * int(score * 30)
        print(f"  {score:+.3f}  {bar:<30}  {name}")
 
if __name__ == "__main__":
    compare_to_numpy()
    verify_geometric_identity()
    demo_similarity_search()

Run it:

uv run --with numpy python vectors_demo.py

Expected output for the stated vectors (last-bit arithmetic and font bar widths may vary):

1000 random vectors match NumPy within the stated tolerance ✅
u·v=3.0  angle=53.13°
 
Movies most similar to «The Matrix»:
  +0.992  █████████████████████████████   Die Hard
  +0.491  ██████████████                  The Hangover
  +0.450  █████████████                   Superbad
  +0.246  ███████                         Notting Hill
  +0.218  ██████                          When Harry Met Sally

What each block does

Anatomy of the code

Name: add / scale / dot / norm from scratch

Desc: One-line definitions that show the operation IS just Σ arithmetic. Once you can read them, NumPy's @, np.dot, np.linalg.norm are exactly the same idea, faster.

Name: compare_to_numpy · property test

Desc: 1000 random vector pairs must agree between your implementation and NumPy's. This is a differential test on 1000 seeded moderate-magnitude samples, not a proof over all real or floating-point inputs. A systematic bug outside the sampled domain can survive.

Name: verify_geometric_identity

Desc: The right-triangle example obtains its angle from atan2 and checks a known geometry case. The general theorem is the analytic law-of-cosines derivation above; no single numerical evaluation proves it.

Name: Movie ‘embeddings’

Desc: Toy 3-D vectors where each dim is a genre score. Real embeddings are 100-1000 dims and learned by a model, but the arithmetic is identical.

Name: np.linalg.norm

Desc: The library call. The chosen norm depends on array shape, axis and ord. Backend, dtype and size affect speed and numerical behaviour; no universal speedup or immunity to overflow is promised.

What to observe

All 1000 random comparisons match to within 1e-9. A forgotten term is likely to be detected; explicit sparse-basis cases make that negative control deterministic. The Matrix and Die Hard have cosine similarity of about 0.992 — they're almost identical direction in the (action, romance, comedy) space. The two rom-com scores are about 0.246 and 0.218 against The Matrix—different directions in this hand-authored toy space. This is what "embedding space" means. Try changing The Matrix's comedy score to 0.95 and re-run — scores and ranking change, but predict and measure the actual order rather than assuming the top result becomes a comedy. The chosen coordinates encode our hand-authored genre assumptions.

Goal: Feel the dot product as ‘projection × length’

Hint: After you compute the projection length, verify it: draw the arrow (3, 4) on paper, drop a perpendicular to the x-axis, measure. Should be 3 units. Because v is the unit x-axis, the projection IS the x-coordinate. Dot product with a unit vector = coordinate in that direction. That's the whole trick behind PCA and every linear projection.

Compute the projection of u = [3, 4] onto v = [1, 0]:

u = [3.0, 4.0]
v = [1.0, 0.0]
 
# proj_v(u) = ( (u·v) / (v·v) ) * v
# Since v is a unit vector, v·v = 1, so proj = (u·v) * v
k = dot(u, v)                # k = u·v = 3
projection = scale(k, unit(v))
print(projection)            # [3.0, 0.0]

For v = [1, 1], do not reuse dot(u,v) * unit(v): that shortcut assumed a unit v. Orthogonal projection onto a nonzero direction v is (dot(u,v) / dot(v,v)) * v. It is the point kv whose residual satisfies (u−kv)·v = 0, so solving gives k = (u·v)/(v·v). Here k = 7/2 and the projection is [3.5, 3.5], of length 7/√2. The residual [−0.5, 0.5] has zero dot product with [1,1].

v = [1.0, 1.0]
k = dot(u, v) / dot(v, v)
projection = scale(k, v)
assert projection == [3.5, 3.5]
assert abs(dot([a - b for a, b in zip(u, projection)], v)) < 1e-12

(d) Production reality · 15 min

Exact neighbours, index recall and model identity

For N corpus rows of width d, scoring one query costs O(Nd) arithmetic and storing the dense matrix costs O(Nd) numbers. Full sorting costs O(N log N), while top-k selection can avoid that full sort. For U queries an exhaustive refresh costs O(UNd), not just U·N independent of dimension. Annoy exposes n_trees and search_k as accuracy/build/query tradeoffs; its own documentation does not give the archived “hash-lookup” guarantee. Establish exact metric neighbours first and measure recall@k = size of the intersection of approximate and exact top-k sets divided by k, with an explicit tie policy. Relevance quality still requires separate task evaluation.

An index needs a model/version, preprocessing, dimension, metric and normalization contract. Equal shape is necessary but insufficient: two independently trained models may have unrelated coordinate systems. Reject incompatible query vectors rather than silently resizing them. During model migration, build and evaluate a separate index or re-embed the corpus; do not mix spaces without a validated alignment.

Shape and finite-precision checks you can observe

For u and v of shape (d,), dot and @ return a scalar; * returns d elementwise products. For A of shape (2,3,4) and B of shape (2,4,5), A @ B has shape (2,3,5), while np.dot(A,B) has shape (2,3,2,5): dot contracts the last axis of A with the second-last of B and keeps both remaining batches.

A = np.ones((2, 3, 4))
B = np.ones((2, 4, 5))
assert (A @ B).shape == (2, 3, 5)
assert np.dot(A, B).shape == (2, 3, 2, 5)
a, b, c = 1e16, -1e16, 1.0
assert (a + b) + c == 1.0
assert a + (b + c) == 0.0

That last pair demonstrates non-associativity in binary64. Dtype, reduction order and libraries can affect results, but the former 92.3% versus 91.7% training anecdote is not established evidence. Log shapes/dtypes and choose absolute plus relative tolerances from expected scale; a fixed 1e−9 is appropriate for this bounded demo, not every numerical pipeline.

Where this shows up in the rest of the plan

Root: Vectors are the atom of numerical computing

Label: S018 · Matrices

Desc: For a matrix of shape (m,n) and a vector of shape (n,), matrix–vector multiplication produces m row dot products, shape (m,).

Label: S020 · Gradient descent

Desc: For a differentiable scalar function in Euclidean coordinates, a nonzero gradient gives steepest ascent. Gradient descent subtracts a scaled gradient; the step size still matters.

Label: S078 · Feature engineering

Desc: A feature row is a vector. Its distance to another row depends on scaling, units and the task, not automatically on user similarity.

Label: Word embeddings / GloVe / word2vec (related topic)

Desc: Word-vector width is model-dependent; cosine ranks angular neighbours, whose semantic usefulness must be evaluated.

Label: RAG + vector DBs (related topic)

Desc: Score compatible query/corpus vectors by the chosen metric, then filter or rerank for relevance.

Label: Attention in Transformers (related topic)

Desc: For matching key/query width d, scores are dot(query,key)/sqrt(d); softmax over eligible keys produces weights.

(e) Recall + stretch · 10 min

Self-check (answer follows each question).

Recall — click each to reveal

Q: Give two equally-valid descriptions of a vector in 3D.

A: Geometric: a displacement arrow with a length, and a direction if nonzero. Algebraic: an ordered list of 3 numbers [x, y, z] ∈ ℝ³. Both are the same object.

Q: Write the algebraic formula for the dot product of u = [u₁, u₂, u₃] and v = [v₁, v₂, v₃].

A: u·v = u₁v₁ + u₂v₂ + u₃v₃. Element-wise multiply, sum.

Q: State the geometric formula for the dot product, and what each factor means.

A: For nonzero real vectors, u·v = ||u|| · ||v|| · cos(θ). The norms are lengths and θ is their angle. If either vector is zero the dot product is zero, but θ and cosine similarity are undefined.

Q: What is the L2 norm (length) of [3, 4]?

A: sqrt(3² + 4²) = sqrt(25) = 5. Pythagoras in 2 dimensions.

Q: You compute cosine similarity between two vectors and get −0.7. What does that mean?

A: For nonzero real vectors, the angle is acos(−0.7) ≈ 134.43°, an obtuse angle. It is neither identical direction nor evidence of negative statistical correlation without a separate random-variable model.

Q: How do you turn a vector into a unit vector?

A: For a nonzero vector, divide every element by the L2 norm. Zero cannot be normalized to a direction. v_unit = v / ||v||. The result has length 1 and the same direction.

Q: Why do people pre-normalise embeddings before storing them in a vector database?

A: Because for unit vectors, cosine similarity equals plain dot product — you skip the division at query time. Also lets you use inner-product indexes (faster than cosine indexes in some libraries).

Q: In NumPy, what's the difference between np.dot(u, v), u @ v, and u * v?

A: For 1D arrays both dot and @ give an inner product; for 2D they give matrix multiplication. Their broadcasting/contraction behavior differs for higher-dimensional arrays. * is elementwise multiplication.

Q: You have 1M product embeddings of dimension 768 and get a query embedding. Naive cosine-sim search — what's the complexity? What's the fix?

A: Exhaustive scoring costs O(nd). Approximate nearest-neighbor methods trade recall, memory and latency; none of those named algorithms provides a universal logarithmic query time or 95–99% recall for arbitrary data. Measure recall against exact neighbors.

Q: Stretch — derive u · v == ||u|| ||v|| cos θ for 2D vectors using the law of cosines.

Stretch: true

A: For nonzero real u and v in Euclidean space, consider the triangle formed by u, v, and (u−v). By the law of cosines: ||u−v||² = ||u||² + ||v||² − 2 ||u|| ||v|| cos(θ). Expand the left side algebraically: ||u−v||² = (u−v)·(u−v) = u·u − 2 u·v + v·v = ||u||² + ||v||² − 2 u·v. Cancel and get u·v = ||u|| ||v|| cos(θ). The two definitions must agree.

Explain-out-loud test

1. What is a vector, in two sentences?
2. What does the sign of a dot product tell you about the two vectors?
3. Why do RAG systems use cosine similarity instead of ‘just checking if strings match’?

Explain-out-loud answer guide

1. In real 3D Euclidean space, a vector has three ordered coordinates and can be drawn as a displacement arrow. The zero displacement has length zero and no direction.

2. For nonzero vectors, a positive dot means an acute angle, zero means a right angle and negative means an obtuse angle. Norms also affect the number; sign alone does not establish semantic relatedness.

3. A learned embedding can retrieve semantically related wording without exact string overlap, if training and evaluation support that use. Cosine compares nonzero compatible directions; lexical retrieval and reranking may still be needed.

What comes next

Next in sequence: S018 · Matrices. The destination is a navigation link, not part of this review’s acceptance.

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.

Corrective source notes

• OpenStax, The Dot Product: coordinate sum, length/angle relationship, orthogonality and projection.
• NumPy dot, matmul and norm: exact shape-dependent API contracts, including dot’s lack of complex conjugation.
• Annoy’s own README: n_trees/search_k are measured quality/cost tradeoffs, not universal logarithmic guarantees.
• Python floating-point tutorial: representation and rounding error; tolerances do not turn finite experiments into proofs.

The real-arithmetic law-of-cosines and projection derivations are proofs under the named domains; the NumPy, basis-vector and finite-precision examples are bounded implementation checks. No approximate index, embedding service, GPU or training benchmark was run. Saved documentation is a review-time snapshot, not a claim that mutable stable URLs name the installed package release.