Vectors, Matrices, and the Geometry of Data

Course map · DL002 · Linear algebra with geometric meaning

The story

A dot product can score a user against a movie, a query against a document, or a token query against a key. These are useful patterns—not claims that every recommender is matrix factorization, every retrieval system emits one vector, or every private model has a known number of attention heads. The operation measures geometry in a chosen representation. Learning and evaluation determine whether that geometry serves the task.

We will connect arrows to coordinates, work matrix products by hand, derive projection and attention scaling, then distinguish SVD approximation from low-rank adaptation and dense retrieval from late interaction.

Prerequisites

DL001's shapes, dtype, views, @ versus *, and Pythagoras. Use an ordinary real Euclidean coordinate system with an orthonormal basis for the dot/norm formulas below. Not every abstract vector space comes with these exact coordinates or inner product.

1 · A vector is TWO things at once

A vector can be represented as an arrow or by coordinates in a chosen basis. The list is the representation, not a coordinate-free definition of every possible vector. Tail-to-tip addition and coordinatewise addition describe the same linear operation.

1.1 The geometric picture

The arrow(3,2) points three units right and two up, with length√13≈3.60555. Its angle in this quadrant is arctan(2/3)≈33.69°. For arbitrary quadrants use atan2(y,x) rather than an unqualified arctan(y/x). The zero vector has length0 and no defined direction.

1.2 The algebraic picture

np.array([3,2]) has shape(2,), not an explicit row or column axis. In equations we may treat it as a column; in a NumPy batch, rows usually hold observations. State which convention a matrix uses before transposing it.

1.3 Why both pictures matter

(3,2)+(1,4)=(4,6). Geometrically chain two displacements; multiplying by2 doubles displacement, by−1 reverses direction, and by.5 halves it. A displacement can begin anywhere, whereas a point requires a chosen origin. This distinction helps separate linear maps, which fix the origin, from affine maps, which add a translation.

1.4 In higher dimensions

A768-coordinate embedding supports the same algebra even though it cannot be drawn as a literal768-axis sketch. Two-dimensional examples build intuition, but high-dimensional concentration and neighborhoods need their own analysis. Equal dimensionality does not make two independently learned embedding spaces comparable.

2 · Vectors are everywhere in disguise

An RGB pixel has three channel values; a flattened28×28 grayscale image has784; a row of50 numerical features has50; a vocabulary-V one-hot has V−1 zeros and one1. TF-IDF encodes a document by weighted vocabulary coordinates, usually sparse; those engineered counts differ from a dense learned embedding. A categorical probability vector lies on the simplex: nonnegative entries summing to1. Learned embeddings and user/item factors are also coordinate arrays, but their coordinate semantics are learned rather than guaranteed human labels.

BERT-base's hidden width768 is a particular documented configuration, not a universal embedding dimension. Flattening a structured object can discard useful inductive structure: a graph, image or variable-length sequence is not adequately characterized by saying “everything is a vector.” Geometry also depends on units and preprocessing; kilometres versus metres change raw inner products.

3 · Vector operations — the daily-driver kit

3.1 Addition

Add corresponding coordinates: (3,2)+(1,4)=(4,6). Shapes and coordinate meaning must agree; broadcasting a row over a matrix is a different, intentionally shared operation.

3.2 Scalar multiplication

Positive scalars preserve direction, negative ones reverse it, and zero removes direction. $(ca)\cdot b=c(a\cdot b)$ shows how raw scores respond to magnitude.

3.3 The L2 norm (length)

$\|v\|_2=\sqrt{\sum_i v_i^2}$. For(3,4), length5 and unit vector(.6,.8). Zero cannot be normalized to a unit direction. Tiny or huge magnitudes can also create numerical issues; use a documented norm implementation and a deliberate zero policy.

3.4 The DOT PRODUCT (the whole reason we're here)

Why do the definitions agree? Expanding $(a-b)\cdot(a-b)$ gives $\|a\|^2+\|b\|^2-2a\cdot b$. The law of cosines gives $\|a-b\|^2=\|a\|^2+\|b\|^2-2\|a\|\|b\|\cos\theta$. Equate the two. For nonzero b, the vector projection of a onto b is $(a\cdot b)/(b\cdot b)\,b$; subtract it from a and the residual has zero dot product with b.

3.5 Compute one by hand

For a=(3,2,1),b=(1,0,2), the products3,0,2 sum to5. Norms√14 and√5 give cosine5/√70≈.597614. a*b returns the three products, while a@b returns their scalar sum.

3.6 What does the sign tell you?

For nonzero vectors, positive dot means an acute angle, zero a right angle, and negative an obtuse angle. Zero dotted with anything is zero, but does not have a right-angle direction. Alignment is not semantic truth; it is a geometric statement in the current space.

4 · Cosine similarity — the dot product's practical cousin

Cosine divides the dot product by both nonzero norms, giving a value in[−1,1] in exact arithmetic. Normalizing removes magnitude, which can be useful or can discard trained signal. A bounded score does not make thresholds portable across queries, datasets or models. Normalizing embeddings from unrelated encoders does not align their coordinate systems.

4.1 Worked example — comparing three documents

Use d1=(1,1,0),d2=(2,2,0),d3=(0,0,1),query=(1,1,0). Raw dots are2,4,0; cosines1,1,0. This illustrates a norm effect. It does not prove cosine is “correct” for every task or that text length always increases embedding norm. For unit-normalized vectors, squared Euclidean distance is2−2cosine, so their rankings coincide apart from ties and rounding.

A sharper negative control uses q=(1,0),a=(1,0),b=(3,4): dot prefers b,3 versus1; cosine prefers a,1 versus.6. Neither ranking is universally wrong; match the metric to the model's trained objective and evaluated relevance.

5 · From vectors to matrices

An m×n real matrix maps n-coordinate columns to m-coordinate columns. Its j-th column is the image of basis vector e_j, so multiplying Ax reconstructs a linear combination of the transformed basis directions. A row-batch implementation XW expresses the same idea under a transposed storage convention.

5.1 A matrix as a function

A(u+v)=Au+Av and A(cv)=cAv. In particular A0=0. Adding b produces an affine map, which need not preserve the origin. Stacking affine maps without nonlinearities collapses to another affine map. A classifier typically needs nonlinear representation changes to bend its boundary; a single nonlinear monotone output does not change the affine level sets.

5.2 What does the transformation look like?

Diag(2,3) maps(1,1) to(2,3). The matrix with rows(0,−1),(1,0) maps(1,0) to(0,1), a90° counterclockwise rotation. Shears, reflections and singular projections are also linear. A zero singular value can collapse a direction; not every square matrix is an invertible rotation-plus-nonzero-stretch.

6 · Matrix multiplication — compose maps and contract indices

$(AB)_{ij}=\sum_kA_{ik}B_{kj}$. The k index disappears, leaving i and j. With column vectors ABx means apply B first, then A. This order explains noncommutativity without memorizing an arbitrary rule.

6.1 The shape rule

(m,k)@(k,n)→(m,n). A(3,4) times B(4,5) is(3,5); two(3,4) matrices cannot multiply directly, but X@Y.T gives a3×3 all-pairs table. Shape compatibility says the operation is defined, not that it is the one intended.

6.2 Compute one by hand

For A with rows(1,2),(3,4), and B with rows(5,6),(7,8): C00=1·5+2·7=19; C01=1·6+2·8=22; C10=3·5+4·7=43; C11=3·6+4·8=50. Every entry is a row-column dot product.

6.3 Matmul is NOT commutative

BA has rows(23,34),(31,46), so it differs from AB. Some particular matrices commute; the point is the operation is not commutative in general.

For X:(32,784),W1:(784,128),W2:(128,10),b1:(128,),b2:(10,), the shapes of X@W1, X@W1+b1, (X@W1)@W2, and X@W1@W2+b2 are respectively(32,128),(32,128),(32,10),(32,10). The last expression omits the first bias; a complete two-affine-layer expression includes b1@W2 as well. No intermediate activation means no added nonlinear expressive power.

7 · The transpose — swap rows and columns

For a2-D matrix, .T swaps its axes. For higher-rank NumPy arrays it reverses all axes; use swapaxes(-2,-1) for batched matrix axes. PyTorch's two-argument transpose is a different API.

Q:(5,3),K:(7,3) gives Q@K.T:(5,7), with entry i,j equal to query i dotted with key j. This computes attention logits, not an entire attention mechanism. Scaling, mask, normalization and values still matter.

7b · Eigenvectors, SVD, and low-rank adaptation

7b.1 · The intuition, on a 2×2

For A with rows(3,1),(0,2), (1,0) is an eigenvector of eigenvalue3 and(1,−1) of eigenvalue2. Eigenvectors are nonzero; negative eigenvalues reverse direction and zero eigenvalues annihilate it. A d×d matrix has at most d linearly independent eigenvectors, not at most d eigenvector directions: the identity has infinitely many. Some matrices have no real eigenvectors, and a defective matrix may lack a full eigenbasis. Diagonalization describes the whole map only when a suitable eigenbasis exists.

7b.2 · SVD — a decomposition for rectangular matrices

For W:(m,n), $W=U\Sigma V^T$ with orthogonal U,V and nonnegative ordered singular values. In the full form U is(m,m),Σ(m,n),V(n,n); the reduced form uses r=min(m,n) columns/rows. Orthogonal factors can reflect as well as rotate. The singular values measure directional gain, not automatically semantic “information.” A small minimum singular value relative to the maximum means poor conditioning when an inverse is sought, contrary to the old SVD comment.

7b.3 · Low-rank approximation — what the theorem actually says

Keeping the top-r singular values minimizes Frobenius approximation error among rank-at-most-r matrices, with squared error $\sum_{i>r}\sigma_i^2$. This is a theorem about approximating a given matrix, not proof that a task's fine-tuning update is low rank or that the approximation preserves model quality.

LoRA freezes W and learns an update BA with B:(m,r),A:(r,n), usually with a scale. It adds r(m+n) trainable entries instead of mn. Rank is at most r; not every trained solution attains r. For4096×4096 and r=16, the update uses131,072 parameters versus16,777,216, a128-fold parameter ratio. Optimizer/activation memory is separate. Even ideal four-bit storage for70B frozen weights takes35GB decimal before overhead, so LoRA alone cannot establish that70B QLoRA training fits a24GB GPU. The QLoRA paper reports65B on a48GB GPU, a different claim.

7b.4 · Adaptation methods have different mechanisms

• DoRA: separates weight magnitude and direction; applies low-rank adaptation to directional updates, with learned magnitude. It is not just SVD truncation.
• QLoRA: backpropagates through a frozen quantized backbone into adapters, introducing NF4, double quantization and paged optimizers in its study. Quantized storage does not mean every operation computes in four bits.
• VeRA: shares frozen random low-rank matrices and learns small scaling vectors. The constrained learned parameterization differs from ordinary LoRA.
• LoftQ: jointly addresses quantization and low-rank initialization to reduce their combined error; it is not merely “initialize from an SVD.”
• PiSSA: initializes trainable principal singular components and retains a residual part. The update/retained partition is explicit.
• LoRA-XS: uses SVD-derived fixed factors with a very small trainable matrix in the low-rank subspace. Its parameter count is not ordinary r(m+n).

Read the freeze/trainable contract before comparing quality or memory. These primary-paper mechanisms are not reproduced pretrained-model experiments here.

8 · A worked example — recommending movies

Use four invented user vectors and five movie vectors, retaining the original numbers. Scores are not calibrated ratings; genre labels are an explanatory assignment for this toy, not an interpretation guaranteed for learned latent coordinates.

import numpy as np
 
a=np.array([3.,2.,1.]);b=np.array([1.,0.,2.])
assert a@b==5 and np.isclose((a@b)/(np.linalg.norm(a)*np.linalg.norm(b)),5/np.sqrt(70))
projection=(a@b)/(b@b)*b;assert np.isclose((a-projection)@b,0)
A=np.array([[1,2],[3,4]]);B=np.array([[5,6],[7,8]])
np.testing.assert_array_equal(A@B,[[19,22],[43,50]])
np.testing.assert_array_equal(B@A,[[23,34],[31,46]])
assert not np.array_equal(A@B,A*B)
np.testing.assert_array_equal(np.array([[0,-1],[1,0]])@np.array([1,0]),[0,1])
E=np.array([[3.,1.],[0.,2.]])
np.testing.assert_allclose(E@np.array([1.,-1.]),2*np.array([1.,-1.]))
W=np.diag([5.,2.,.5]);U,S,Vt=np.linalg.svd(W,full_matrices=False)
Wr=(U[:,:1]*S[:1])@Vt[:1]
assert np.isclose(np.linalg.norm(W-Wr,'fro')**2,2**2+.5**2)
assert 16*(4096+4096)==131072 and 70_000_000_000/2>24_000_000_000
 
def cosine_rows(A,B):
    A=np.asarray(A,float);B=np.asarray(B,float)
    assert A.ndim==B.ndim==2 and A.shape[1]==B.shape[1]
    na=np.linalg.norm(A,axis=1,keepdims=True);nb=np.linalg.norm(B,axis=1,keepdims=True)
    if np.any(na==0) or np.any(nb==0): raise ValueError('zero direction')
    return (A/na)@(B/nb).T
q=np.array([[1.,0.]])
assert np.argmax(q@np.array([[1.,0.],[3.,4.]]).T)==1
assert np.argmax(cosine_rows(q,[[1.,0.],[3.,4.]]))==0
try: cosine_rows([[0.,0.]],[[1.,0.]])
except ValueError: pass
else: raise AssertionError('zero direction accepted')
users=np.array([[1.,.2,0],[.1,1.,.3],[.3,.9,.4],[-.5,-.1,1.]])
movies=np.array([[1.,0,0],[0,1.,0],[.5,.5,0],[0,.2,1.],[.8,.3,-.2]])
scores=users@movies.T
np.testing.assert_allclose(scores[1],[.1,1.,.55,.5,.32])
order=np.lexsort((np.arange(5),-scores[1]))[:2]
assert order.tolist()==[1,2]
assert np.array([4,1,5])@np.array([5,0,4])==40
assert np.array([4,1,5])@np.array([1,5,0])==9
print('geometry, SVD residual, metric disagreement and movie scores passed')

User1's scores are(.1,1,.55,.5,.32), so movie1 then movie2 are top two. A real recommender additionally needs training data, bias terms/objective, candidate policy, evaluation and often other models. This is the scoring kernel of a simple factor model, not the entire Netflix Prize system.

9 · Vectors in ML — a mental map

An affine map combines features; an activation changes the representation; an output head expresses a task-specific quantity. Outputs can be real regression values, scores or distributions—not every model ends in probabilities. A matrix's columns describe basis images, while a row-batch score table describes pairwise comparisons; track the convention instead of assigning one meaning to every rectangular array.

9b · Embedding models — read the representation contract

Official model cards, not a timeless leaderboard, identify how to obtain and compare representations:

• NV-Embed-v2: its card describes4096-dimensional embeddings and model-specific processing. A release MTEB score is an evaluation snapshot, not a universal current ranking.
• stella_en_1.5B_v5: the author card supports multiple dimensions512 through8192 and describes separate dense projection folders. It reports1024d nearly matching8192d on its benchmark; that does not establish arbitrary512-coordinate truncation is lossless.
• BGE-M3: dense, sparse and multi-vector modes are distinct representations; it cannot be described solely as one vector per document.
• gte-Qwen2-7B-instruct: instructions and pooling belong to its model contract; its official card lists 3584 dimensions for the Qwen2-7B model, not the4096 of the older Qwen1.5-7B entry. Equal width would not make independently learned spaces interchangeable either.
• Gemini embedding: the provider documents configurable dimensionality and model-specific task use. An API service has its own version and operational boundary; no provider call is made here.

9b.1 · Matryoshka embeddings — learned prefixes

A dot product decomposes into a prefix sum and a remainder. That algebra is true for any vectors, but useful prefix retrieval requires a training objective that makes selected prefixes informative. After truncation, renormalize if using cosine. The omitted dot term is bounded in absolute value by the product of the tail norms; small tails can help, but a small score error can still reorder nearly tied candidates.

For100M documents,64 int8 coordinates need6.4GB decimal of raw vector payload.4096 float16 coordinates need819.2GB, a128-fold payload ratio—not1000-fold. Index edges, IDs, scales, metadata, replicas and buffers are extra. A two-stage design retrieves candidates from the compact representation, loads full vectors for reranking, and measures whether relevant documents survive stage one. Reranking cannot recover a document never shortlisted.

9b.2 · ColBERT and late interaction — many vectors per document

ColBERT2020 retains token-level representations and aggregates best matches: $\sum_i\max_j q_i\cdot d_j$ under its representation/masking contract. For query rows(1,0),(0,1) and document rows(1,0),(.6,.8), the score is1+.8=1.8. Averaging each text to one vector is a different operation. ColBERTv2 and PLAID were2022 developments, not a single2024 successor. BGE-M3's multi-vector score includes its documented normalization; do not silently interchange sums and means across implementations.

9b.3 · Vector indexes — budget against recall

Exact Q@D.T scans every candidate; approximate methods spend less work by navigating/pruning candidate sets or compressed representations. FAISS supplies multiple exact and approximate index types; HNSW, IVF and product quantization are not identical algorithms. SPANN's primary paper studies a memory/disk-oriented billion-scale design; it is not evidence for the old invented “DiskANN successor plus ONNXRuntime guarantees50ms p99” statement.

Measure recall@k against a bounded exact search, query distribution, index build cost, RAM/disk bytes, update/filter behaviour and latency percentiles. There is no universal0.1% scanned fraction or20ms guarantee. Inner product, cosine and Euclidean indexes require consistent preprocessing and score conventions.

9b.4 · Modern attention is many dot products, batched

Scaled dot-product attention uses $S=QK^T/\sqrt{d_k}$, then masking, row softmax and multiplication by V. Self-attention derives Q,K,V from the same sequence through different projections; cross-attention may use different sequences.

If independent zero-mean query/key components each have variance1, each product has variance1 and the sum has variance d_k. Dividing by√d_k gives variance1. With variances σq² and σk², the unscaled variance is d_kσq²σk² under the independence assumptions. Correlations, trained weights and normalization alter this picture; it is not a forced universal scale. Division by d_k overcorrects this specific model, giving variance1/d_k. Scaling by a positive constant leaves a pure top-k ordering unchanged but changes the following softmax distribution. L2 normalization changes the statistics again and is not equivalent to the standard attention scale.

rng=np.random.default_rng(2)
Q=rng.normal(size=(5,3));K=rng.normal(size=(7,3))
S=Q@K.T/np.sqrt(3)
np.testing.assert_allclose(S,[[sum(Q[i,k]*K[j,k] for k in range(3))/np.sqrt(3) for j in range(7)] for i in range(5)])
q=np.array([[1.,0.],[0.,1.]]);d=np.array([[1.,0.],[.6,.8]])
assert np.isclose((q@d.T).max(1).sum(),1.8)
assert 100_000_000*64==6_400_000_000
assert 100_000_000*4096*2==819_200_000_000
# Prefix counterexample: ordinary truncation can destroy the best match.
q=np.array([[.1,1.]]);docs=np.array([[.1,-1.],[0.,1.]])
assert np.argmax(cosine_rows(q,docs))==1
assert (q[:,:1]@docs[:,:1].T).argmax()==0
print('all-pairs scalar oracle, MaxSim, memory and prefix-ranking counterexample passed')

10 · Failure modes and metric tradeoffs

query*key produces a vector; summing it yields a dot product, but selecting its first coordinate does not. A forgotten transpose may fail, or accidentally succeed on square shapes—use rectangular and numeric tests. Normalizing unrelated model spaces cannot fix a coordinate mismatch. Norm is not intrinsically confidence, and cosine is not intrinsically semantic truth.

Choose raw dot when the learned scoring contract includes magnitude; cosine when normalized directional comparison matches the model; and scaled dot when a downstream softmax needs its tested scale. Plot norm distributions and evaluate labeled retrieval; bounded cosine scores alone do not calibrate a universal relevance threshold.

11 · Retention scaffold

1. Two definitions of dot product? Sum coordinate products and norm product times cosine. Expanding the squared difference and applying the law of cosines proves their agreement in an orthonormal Euclidean basis.
2. Movie arithmetic? u=(4,1,5) scores40 against(5,0,4) and9 against(1,5,0); the first wins under raw dot. Cosine or additional business rules would define a different scoring task.
3. Two bugs? * lacks the contracted sum; missing attention scaling changes score spread under the stated statistics. A large head does not automatically prove saturation in every trained model.
4. All-pairs cost forN=10000,d=128? About2N²d=25.6 billion multiply/add FLOPs by the two-FLOP convention; the100M float32 scores alone need400MB decimal. Bulk matmul often beats Python loops, but no speedup is guaranteed without measurement, dtype and threading.
5. When prefer cosine? When the representation contract/evaluation calls for normalization, with nonzero norms and a same-model coordinate space. Do not choose it to mix unrelated embeddings or transfer thresholds without calibration.
6. Two-stage100M pipeline? Budget6.4GB compact vector payload plus index/scales/IDs in the50GB RAM allowance; keep819.2GB full-vector payload on disk and rerank a bounded candidate set. Measure stage-one recall and disk fetch costs; the budget alone establishes neither latency nor relevance.

Recall — no scrolling

Compute the sign for nonzero perpendicular/same/opposite vectors, give the(m,k)@(k,n) shape, and explain Q:(5,3) times K.T:(3,7). Include the zero-vector exception and distinguish logits from complete attention.

Stretch — for one extra hour

For trusted local GloVe vectors, map words to rows, validate finite nonzero norms, normalize rows, and rank cosine scores while excluding query/source words. For the analogy king-man+woman, compute that vector in the same embedding space and normalize it before scoring. Repeat paris-france+japan. A nearest neighbor of king is not automatically queen; analogy success depends on vocabulary, training, normalization and exclusion policy. No weights or dataset are downloaded here.

The mechanism can be tested on an explicitly invented embedding dictionary without claiming semantic evidence:

words=['king','man','woman','queen','paris','france','japan','tokyo']
E=np.array([[1,1,1],[1,0,0],[0,1,0],[0,2,1],[2,0,1],[1,0,1],[0,1,1],[1,1,1]],float)
def analogy(a,b,c):
    ids=[words.index(w) for w in (a,b,c)];v=E[ids[0]]-E[ids[1]]+E[ids[2]]
    scores=cosine_rows(v[None,:],E)[0];scores[ids]=-np.inf
    return words[int(np.argmax(scores))]
assert analogy('king','man','woman')=='queen'
assert analogy('paris','france','japan') in ('king','tokyo')  # deliberately tied invented rows
print('local analogy mechanism passed; invented coordinates are not GloVe evidence')

In your own words

Explain a matrix as transformed basis vectors and explain why the metric is meaningful only with a representation contract.

Spaced-review pointer

Tomorrow reconstruct all four entries of AB and the movie row; next week derive the SVD tail error and the attention variance assumption. A heading you recognize is not a derivation you can reproduce.

Next-session teaser

DL003 makes the implicit sharing in X@W+b explicit, including memory costs and backward reductions.

What to bring back tomorrow — sticky note

Contract the inner index; name every axis; compare norms deliberately; test zero and rectangular cases. A valid shape is necessary, not sufficient.

Primary references and boundary

Mathematics for Machine Learning, Attention Is All You Need, LoRA, QLoRA, DoRA, VeRA, LoftQ, PiSSA, LoRA-XS, Matryoshka, ColBERT, SPANN and the named official model cards support the comparisons. The programs are local numerical witnesses; model rankings, pretrained retrieval, adapter training and large indexes are not executed.

Model-card references: NV-Embed-v2, stella, BGE-M3, GTE-Qwen2-7B, and Gemini embeddings. Roughgarden–Valiant, lecture9, April30 2024 states the Frobenius low-rank approximation result.