DL S001 · NumPy Warm-up — Tensors Are Just Arrays

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 1 / 80 · Module M01 · ~2 hours

The story we're starting with

Here's a thing I wish someone had told me on day one: every large language model you've ever heard of is, at the bottom of the stack, a program that takes some numbers, arranges them in a rectangular grid, and multiplies rectangular grids together in a very specific pattern. That's it. GPT-4 is a lot of grids. Llama is a lot of grids. The magic — such as it is — comes from how many grids, what the numbers inside them are, and what data those numbers were shaped by. But the operation, always, is grids of numbers being multiplied.

We call these grids "tensors" when we want to sound impressive, "arrays" when we're programming, and "matrices" when we want to sound like mathematicians. They are the same thing.

So we start here. Before we write a single line of PyTorch, before we hear the word "attention", before we even define a neuron — we're going to become fluent with the object that everything else is built out of. In Python that object is numpy.ndarray. Get comfortable with it and the next 79 sessions become 30% easier. Skip this and you will spend the next month fighting shape errors instead of learning ideas.

That's the deal. Two hours today. Everything gets easier tomorrow.

Prerequisites

• Python. You can write a for-loop, a list comprehension, a function, and a class. That's it. If you can't yet, Python foundations of the 6-month plan (sessions S005–S014) will get you there.
• A terminal + Python 3.11 or newer. No other tooling needed today. We'll add PyTorch in Session 014.

1 · What is a NumPy array, really?

Before we type anything, let's build a mental model. Open a text editor, not a Python REPL, for the next 3 minutes.

A Python list of 4 numbers is a table of pointers to boxed integer objects scattered across the heap:

Each list slot holds a pointer to a Python integer object that lives somewhere else in memory. To sum a Python list of 4 numbers, the interpreter has to dereference 4 pointers, look up the type of each, dispatch a + method for each, and construct a new integer. Every operation is a mini-lookup.

A NumPy array of the same 4 numbers is one flat, contiguous run of bytes:

Four int64s laid out contiguously in memory, no pointers, no per-element type lookup. To sum them, NumPy runs a tight C loop over 32 contiguous bytes. This is why NumPy is ~100× faster than a Python loop.

The point: every performance intuition about NumPy — and later PyTorch — traces back to this one image. Contiguous typed memory. That's the trick.

1.1 · The stride trick — how one array can be many shapes

Here's the second thing nobody tells you on day one, and it will pay off every time you read a modern kernel. A NumPy array is not really the numbers. It's a small header that says: "the numbers live at memory address 0x7f…, there are 12 of them, they're int64, and to walk one step along axis 0 skip 24 bytes, to walk one step along axis 1 skip 8 bytes." Those per-axis skip distances are called strides.

Watch:

Now the magic: reshaping and transposing are almost always free. They don't move a single byte. They just build a new header pointing at the same underlying buffer with different shape and stride values.

This is why x.transpose(0, 2, 1, 3) in a transformer's attention block is not the expensive operation it looks like. It rewrites the header. The bytes stay put. The next matmul might have to make a contiguous copy (via .contiguous() in PyTorch, or an implicit copy in NumPy) if the resulting stride pattern is bad for the BLAS kernel, but the transpose itself is O(1).

Modern parallel: every deep-learning framework's .view(), .reshape(), .transpose(), .permute(), .expand() is doing the same stride bookkeeping. When Karpathy writes q.transpose(1, 2) in nanoGPT to reshape (B, T, nh, hd) into (B, nh, T, hd), he's not moving 100MB of memory around — he's editing a 40-byte header. This is also why torch.compile() and jit-style optimizers can fuse away most reshape ops entirely: they never existed as work.

1.2 · Same story on the GPU (a preview of S014–S015)

Fast-forward to when we hit PyTorch on the GPU. A CUDA tensor is a header (on the CPU) pointing at a big contiguous buffer of numbers (on GPU HBM memory). The header says: dtype bfloat16, shape (1, 4096, 4096), device cuda:0, strides (...). When you write x.transpose(1, 2), PyTorch updates the header. Zero GPU work. When you then do x.reshape(-1), if the stride pattern is now non-contiguous, PyTorch launches a cudaMemcpy2DAsync under the hood to lay out the bytes contiguously. That copy shows up in your profiler as "reshape" but it's really the delayed cost of the earlier free transpose.

Why we care today: understanding strides means understanding why the same three-line attention snippet can be 3× slower or faster depending on the order of transposes. It's not the algorithm — it's the memory layout. The people who write the fastest kernels on Earth (FlashAttention-3 authors, DeepSeek's kernel team, the vLLM PagedAttention folks) all spend most of their time thinking about strides.

Further reading:

• NumPy internals: "NumPy internals · Multidimensional array indexing" — official docs, 15 minutes, worth every one.
• Chip Huyen's "How to make Python 100x faster with less than 100 lines of Rust" — shows why the stride/BLAS model dominates.
• PyTorch strides tutorial — same idea, PyTorch dialect.

2 · Making arrays (five ways)

Let's actually type some code. Open a Python REPL and follow along.

2.1 From a Python list

import numpy as np
 
a = np.array([1, 2, 3, 4])
print(a)          # [1 2 3 4]
print(a.shape)    # (4,)   — one axis of length 4
print(a.dtype)    # int64
print(a.ndim)     # 1
print(a.size)     # 4

Notice the .shape is (4,) — a tuple with a trailing comma. That trailing comma is Python's way of saying "this is a tuple of length 1, not a parenthesized expression". You'll see it a lot.

2.2 A 2D array — nested lists become a matrix

b = np.array([[1, 2, 3],
              [4, 5, 6]])
print(b)
# [[1 2 3]
#  [4 5 6]]
print(b.shape)    # (2, 3)   — 2 rows, 3 columns
print(b.ndim)     # 2
print(b.size)     # 6

The shape convention (rows, cols) for 2D arrays is universal in NumPy and PyTorch. In the world of images we'll extend it to (batch, channels, height, width). In the world of language models we'll use (batch, sequence_length, embedding_dim). But it's always the same idea: a tuple of axis lengths, outermost first.

2.3 Zero-filled, one-filled, uninitialized

When you're allocating buffers for a training loop, you often want a shape-only array without caring about the values.

z = np.zeros((3, 4))          # 3×4 array of 0.0 (default dtype=float64)
o = np.ones((2, 2), dtype=np.float32)
e = np.empty((5,))            # uninitialized — contents are garbage from RAM
print(z.dtype, o.dtype)       # float64 float32

Small gotcha: np.empty does not initialize the memory. It's marginally faster than np.zeros because it skips the zero-fill, but you will use it approximately never. Just remember it exists so you know why colleagues use it in hot loops.

2.4 Ranges

Two range functions you'll use every day:

r  = np.arange(10)            # like range(10): [0 1 2 3 4 5 6 7 8 9]
rf = np.arange(0, 1, 0.1)     # start, stop (exclusive), step: 10 values
lin = np.linspace(0, 1, 11)   # 11 evenly spaced values from 0 to 1 inclusive
print(lin)
# [0.  0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1. ]

Use arange when you have an integer step. Use linspace when you have a target count. linspace is the safer choice for anything involving floats — arange(0, 1, 0.1) will sometimes give you 10 elements and sometimes 11 depending on floating-point rounding, whereas linspace(0, 1, 11) always gives exactly 11.

2.5 Random arrays (the workhorse)

Neural network weights start life as random arrays. Get comfy with these.

rng = np.random.default_rng(seed=42)     # modern NumPy random API — use this, not np.random.*
u = rng.uniform(0, 1, size=(3, 4))       # uniform in [0, 1)
n = rng.normal(loc=0, scale=1, size=(3, 4))  # Gaussian with mean 0, stddev 1
i = rng.integers(0, 10, size=(3, 4))     # integers in [0, 10)
print(n.mean(), n.std())  # ~0.0, ~1.0

Always seed your RNG. Reproducibility isn't optional — it's how you'll debug training runs starting in Session 019. The default_rng(seed=...) idiom is the modern way; the old np.random.seed() API is deprecated for new code.

3 · Indexing and slicing — where the shapes come from

This is the section where the muscle memory happens. Take it slow. Do every example in your REPL as you read.

3.1 The one-dimensional case

a = np.arange(10)     # [0 1 2 3 4 5 6 7 8 9]
 
a[0]                  # 0             — a scalar
a[-1]                 # 9             — last element
a[2:5]                # [2 3 4]       — half-open interval, like Python
a[:3]                 # [0 1 2]       — from the start
a[7:]                 # [7 8 9]       — to the end
a[::2]                # [0 2 4 6 8]   — every second element
a[::-1]               # [9 8 ... 0]   — reversed

Every one of these is a view, not a copy. That means a[2:5] doesn't allocate new memory — it just remembers "start at offset 2, length 3, stride 1". If you modify the view, you modify the original:

This is one of NumPy's most beloved features and one of its most notorious bug sources. If you want a real copy, ask for one: v = a[2:5].copy().

3.2 The two-dimensional case — this is where beginners get lost

b = np.arange(12).reshape(3, 4)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]
 
b[0]                  # [0 1 2 3]         — the first row (returns a 1D array)
b[0, :]               # [0 1 2 3]         — same, more explicit
b[:, 0]               # [0 4 8]           — the first column
b[1:3, 1:3]           # [[5 6]
                      #  [9 10]]
b[::-1, :]            # rows reversed
b[:, ::-1]            # columns reversed

The rule: NumPy uses one set of brackets and comma-separates axes. Python lists chain brackets: l[0][1]. NumPy does arr[0, 1]. Both notations work in NumPy (b[0][1] is legal), but the comma form is idiomatic and faster (no intermediate array).

Predicting shapes: every slice with : keeps that axis; every scalar index drops it.

That last one is subtle and important. b[0] gives you a 1D array of shape (4,). b[0:1] gives you a 2D array of shape (1, 4). Slicing preserves the axis; scalar indexing collapses it. You will use both.

3.3 The three-dot ellipsis ...

For high-rank tensors (4D images, 4D transformer activations) you don't want to type : five times. The ellipsis fills in "as many : as needed":

img = rng.normal(size=(64, 3, 224, 224))   # (batch, channels, H, W)
img[..., 0]                                # all batches, all channels, all rows, column 0
                                           # shape: (64, 3, 224)
img[0, ...]                                # first image, all channels, all pixels
                                           # shape: (3, 224, 224)

You'll see ... everywhere in real ML code. Get used to it.

3.4 Boolean masks — the killer feature

Say you want all the elements greater than 5:

a = np.arange(10)
mask = a > 5           # [False False ... False  True  True  True  True]
a[mask]                # [6 7 8 9]
a[a > 5]               # [6 7 8 9]   — the idiomatic one-liner
a[a > 5] = 0           # zero out everything > 5:  [0 1 2 3 4 5 0 0 0 0]

Boolean masks are how you'll filter datasets, compute accuracies, and mask out padding tokens in transformers. Learn them cold.

3.5 Fancy indexing with integer arrays

a = np.array([10, 20, 30, 40, 50])
a[[0, 2, 4]]           # [10 30 50]   — pick these positions
a[[0, 0, 0]]           # [10 10 10]   — same position 3 times

This lets you gather arbitrary sets of positions. It's how embedding lookups work: embedding_matrix[token_ids] where token_ids might be an array like [42, 17, 8, 42, 9].

4 · Axes — what does axis=0 actually do?

This is the concept that trips up every beginner. Let's kill it right now.

Take a 2D array:

Now try:

X.sum(axis=0)   # [5 7 9]     — shape (3,)
X.sum(axis=1)   # [ 6 15]     — shape (2,)

Read that twice. axis=0 returned a shape-(3,) array. axis=1 returned a shape-(2,) array. The axis you name is the one that gets collapsed.

Think of it as "which axis do we walk down?" To compute sum(axis=0) on a 2D matrix, you walk down the columns (axis 0 = row index) and produce one number per column. The 3 columns → 3 numbers → shape (3,).

To compute sum(axis=1), you walk across the rows (axis 1 = column index) and produce one number per row. The 2 rows → 2 numbers → shape (2,).

4.1 Worked numeric example

Let's do it by hand. Given

X = [[1, 2, 3],
     [4, 5, 6]]

X.sum(axis=0) — collapse the row axis:

• Column 0: 1 + 4 = 5
• Column 1: 2 + 5 = 7
• Column 2: 3 + 6 = 9
• Result: [5, 7, 9] — shape (3,) ✅

X.sum(axis=1) — collapse the column axis:

• Row 0: 1 + 2 + 3 = 6
• Row 1: 4 + 5 + 6 = 15
• Result: [6, 15] — shape (2,) ✅

Do this by hand once. It becomes intuitive forever.

4.2 Same rule works for higher dimensions

T = rng.normal(size=(4, 5, 6))   # 3D tensor
T.sum(axis=0).shape              # (5, 6)   — axis 0 collapsed
T.sum(axis=1).shape              # (4, 6)   — axis 1 collapsed
T.sum(axis=2).shape              # (4, 5)   — axis 2 collapsed
T.sum(axis=(0, 2)).shape         # (5,)     — two axes collapsed
T.sum().shape                    # ()       — a scalar

4.3 keepdims — the flag that saves your neural network

Sometimes you want to collapse an axis but keep it as size-1 so that broadcasting still works later:

This is used constantly in softmax:

def softmax(x):
    """Softmax along the last axis."""
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

Notice the keepdims=True — without it, the divide would broadcast wrong and you'd get a shape error. Remember this. We'll use softmax 40 times in this series.

5 · Reshaping — same data, different shape

Reshape lets you re-interpret the same underlying memory as a different shape. It's free (no copy) as long as the new shape has the same total size.

a = np.arange(12)          # shape (12,)
a.reshape(3, 4)            # shape (3, 4) — [[0,1,2,3],[4,5,6,7],[8,9,10,11]]
a.reshape(2, 2, 3)         # shape (2, 2, 3)
a.reshape(-1, 4)           # shape (3, 4) — the -1 means "figure it out"
a.reshape(4, -1)           # shape (4, 3)

-1 in reshape means "infer this dimension from the others". You'll use it constantly to flatten batches:

images = rng.normal(size=(32, 3, 28, 28))   # 32 MNIST-ish images
flat   = images.reshape(32, -1)             # shape (32, 3*28*28) = (32, 2352)

5.1 Transpose

.T swaps the last two axes (2D case: swaps rows and columns). .transpose(...) gives you full control:

X = rng.normal(size=(3, 4))
X.T.shape                         # (4, 3)
 
T = rng.normal(size=(2, 3, 4))
T.transpose(2, 0, 1).shape        # (4, 2, 3) — new order of axes

You'll use .transpose(0, 2, 1) a lot in transformer code to swap the sequence and head dimensions.

5.2 Adding and removing axes

Two idioms you'll see 1000 times:

v = np.array([1, 2, 3])           # shape (3,)
v[:, None].shape                  # (3, 1)   — turn a row into a column
v[None, :].shape                  # (1, 3)   — turn into an explicit row
v[None, :, None].shape            # (1, 3, 1)

Or the more explicit form:

v.reshape(3, 1)                   # same as v[:, None]
np.expand_dims(v, axis=0)         # same as v[None, :]
np.squeeze(v[:, None])            # shape (3,)   — remove size-1 axes

The [:, None] and [None, :] idiom is critical for broadcasting. We'll spend all of Session 003 on why.

6 · Element-wise operations and vectorization

The single biggest speed win from NumPy is: replace Python loops with array operations.

6.1 The slow way

import time
 
n = 1_000_000
a = list(range(n))
b = list(range(n))
 
t0 = time.time()
c = [ai + bi for ai, bi in zip(a, b)]
print(f"Python list: {time.time() - t0:.3f}s")
# ~0.05s on a modern laptop

6.2 The fast way

a = np.arange(n)
b = np.arange(n)
 
t0 = time.time()
c = a + b
print(f"NumPy: {time.time() - t0:.3f}s")
# ~0.002s — roughly 25× faster

The rule: any time you see a for loop over array elements, ask whether NumPy has an operation that does it in bulk. 95% of the time the answer is yes.

6.3 The vocabulary

a + b        # element-wise addition
a * b        # element-wise multiplication (NOT matrix multiplication!)
a @ b        # matrix multiplication (NOT element-wise!)
np.dot(a, b) # same as @ for 1D and 2D
a ** 2       # element-wise square
np.exp(a)    # element-wise exp
np.log(a)    # element-wise log
np.maximum(a, 0)  # element-wise max — this IS ReLU

That last line is worth pausing on: np.maximum(x, 0) is literally the ReLU activation function. When we implement neural nets in Session 007, this is the entirety of the ReLU non-linearity. One line.

7 · Reductions and stats — the daily-driver ops

a = rng.normal(size=(100,))
 
a.sum()      # total
a.mean()     # average
a.std()      # standard deviation (ddof=0 by default — NumPy uses population variance)
a.var()      # variance
a.min()
a.max()
a.argmin()   # index of the min
a.argmax()   # index of the max
a.cumsum()   # running sum: [a0, a0+a1, a0+a1+a2, ...]

The argmax one is important: that's how you extract predictions from a classifier. If your neural network's output is logits with shape (batch, num_classes), then logits.argmax(axis=-1) gives you the predicted class per example. Remember it.

8 · A tiny worked example — normalize a batch of vectors

Let's put it all together. Say we have a batch of 5 embedding vectors, each of dimension 3, and we want to L2-normalize each one (make each vector's length equal to 1). This exact operation appears in CLIP, in RAG, in every embedding model on Earth.

The equivalent one-liner (more idiomatic):

normalized = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

Read every line of that one-liner and make sure you can predict the shape at every step. If you can, you have earned the next 79 sessions.

8b · A 2025–2026 field report — where these NumPy patterns actually show up

Before we do war stories, take three minutes and see where the moves you just learned live in code you'll read this year. This is not filler. When you skim a modern research repo — nanoGPT, TRL, vLLM, DeepSeek-V3's reference kernels — you are looking at ~80% of the operations we just covered, dressed in a different framework. Recognize the moves and you can read any paper's code the same day it drops.

8b.1 · The attention core, in NumPy shapes

Open Karpathy's nanoGPT (github.com/karpathy/nanoGPT, ~300 lines of model code, ~2M stars of collective attention). The heart of the transformer block is these four lines (paraphrased):

# q, k, v all have shape (B, nh, T, hd)
#   B  = batch,  nh = num heads,  T = sequence length,  hd = head dim
att = (q @ k.transpose(-2, -1)) / math.sqrt(hd)   # (B, nh, T, T)
att = att.masked_fill(mask == 0, float('-inf'))
att = softmax(att, axis=-1)
y   = att @ v                                     # (B, nh, T, hd)

Every single operator there is one we just met:

• q @ k.transpose(-2, -1) — matmul over the last two axes, batched over the first two. Same rule as §6, just with two leading axes instead of zero.
• / math.sqrt(hd) — a scalar broadcast against a 4D tensor. Broadcasting rule from §5.2, unchanged.
• .masked_fill(mask == 0, -inf) — boolean indexing, §4.
• softmax(..., axis=-1) — reduce along the last axis, §4 again, with a keepdims=True under the hood so the subtraction of the row max broadcasts cleanly.

If you can read those four lines and mentally track the shape at each step — you already understand attention arithmetically. We haven't earned the meaning yet (that's S036), but the shapes are just NumPy.

8b.2 · FlashAttention-3 (Shah et al., 2024) — same shapes, different memory story

FlashAttention-3 (July 2024) hits 75% of H100 peak FLOPs on FP16 and up to 1.2 PFLOPs/s on FP8 — a 1.5–2.0× speed-up over FlashAttention-2. What changed? Not the math — the four lines above are unchanged. What changed is how the intermediate (B, nh, T, T) attention matrix is materialized. FA-2 stored it in HBM (slow, off-chip); FA-3 tiles the computation so that att never exists as a whole tensor — it's assembled in on-chip SRAM in blocks, streamed through the softmax with online-updated running max and sum, and immediately consumed by the att @ v matmul.

Why this matters for you today: the logical NumPy expression is the specification. The GPU kernel is one implementation of that spec. When you learn to think in NumPy shapes, you can read any paper — even a paper whose whole contribution is a new implementation — and know what it's computing.

Further reading:

• Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision" (2024).
• Dao. "FlashAttention-2" (2023).
• Karpathy. nanoGPT source.

8b.3 · DeepSeek-V3's multi-latent attention (2024) — a reshape you can already read

DeepSeek-V3 (December 2024, tech report) trained a 671B-parameter MoE model on 14.8T tokens for roughly $5.6M — a headline number that made the whole industry pause. One of its two key architectural moves is Multi-Latent Attention (MLA): instead of storing full K and V tensors of shape (B, nh, T, hd) in the KV cache (which is what dominates inference memory), MLA compresses them into a low-rank latent of shape (B, T, d_c) where d_c is much smaller than nh * hd, then reconstructs K and V on the fly via a learned up-projection.

In NumPy, the shape story of that compression is exactly:

# Normal KV cache:  memory = 2 * B * nh * T * hd   (K and V, per layer)
# MLA latent:        memory = B * T * d_c          (one shared latent)
# With d_c ≈ hd (roughly), MLA is ~14× smaller for a 128-head model.
 
latent   = x @ W_dc          # (B, T, d) @ (d, d_c) -> (B, T, d_c)  — compress
k        = latent @ W_uk     # (B, T, d_c) @ (d_c, nh*hd) -> (B, T, nh*hd)  — reconstruct K
k        = k.reshape(B, T, nh, hd).transpose(0, 2, 1, 3)   # (B, nh, T, hd)

Read that block. Every operation is a matmul (§6) or a reshape+transpose (§5). If you can hold those shapes in your head, you can read the MLA paper without any deep-learning background — because MLA is just a clever choice of reshapes. The neural network stuff comes later; the shapes are today.

8b.4 · Long-context reshapes — Llama 4, YaRN, LongRoPE

Llama 4 (April 2025) shipped with a 10M-token context window. Gemma 3 (March 2025) went to 128K. The technique that made this possible — pioneered by YaRN (Peng et al., 2023) and refined by LongRoPE (Ding et al., 2024) — is fundamentally a per-frequency rescaling of positional embeddings. In shape terms:

That [:, None] * [None, :] outer-product idiom is the beating heart of every long-context extension paper in 2024–2025. When someone says "we scaled context 32× by rescaling RoPE frequencies", they mean: they multiplied freqs by a schedule-dependent scalar and rebuilt this (T, hd/2) grid. It's a broadcasting trick.

8b.5 · Quantization — dtype fluency pays off

Remember §2 where we harped on dtype? In 2025 the dtype conversation exploded:

• FP8 e4m3 / e5m2 — Hopper (H100) and Blackwell (B200) natively support FP8. Llama 3.1 405B inference on H100 in FP8 halves memory vs BF16.
• INT4 (GPTQ, AWQ, QuIP#) — 4-bit weights are now standard for inference on consumer GPUs.
• 1.58-bit BitNet (Wang et al., 2024) — weights restricted to 1. A 3B-parameter BitNet matches an FP16 LLaMA of the same size at a fraction of the inference cost.

When you see a HuggingFace model card that says torch_dtype=torch.bfloat16 or load_in_4bit=True, you're seeing the direct descendant of np.array([...], dtype=np.int64). Same idea — pick the smallest number type that doesn't destroy your accuracy — enormously bigger stakes.

Further reading:

• Micikevicius et al. "FP8 Formats for Deep Learning" (2022, but the reason your 2025 GPU has FP8 units).
• Wang, Ma, Dong, Huang, Zhang, Wang, Wu, Xiong, Cao, Wei. "BitNet: Scaling 1-bit Transformers" (2024).
• Tseng et al. "QuIP#: Even Better LLM Quantization" (2024).

9 · War stories — bugs you will absolutely hit

predictions = logits.argmax(axis=-1)       # shape (100,)
labels      = np.array([[0], [1], [2], ...])  # shape (100, 1)
accuracy    = (predictions == labels).mean()

for i in range(len(dataset)):
    img = dataset[i]                       # shape (3, 224, 224)
    crop = img[:, 16:240, 16:240]          # shape (3, 224, 224) — a view!
    crop[0, 0, 0] = 255                    # "just marking one pixel for debugging"

a = np.array([10, 20, 30], dtype=np.int64)
b = np.array([3, 7, 4], dtype=np.int64)
a / b       # [3.33 2.86 7.5]  — Python 3 semantics
a // b      # [3 2 7]           — floor division

10 · Diagram — the shape mental model

Here's the picture I want burned into your brain by the end of this session:

Every transformer input tensor in this series will have shape (batch, seq_len, embed_dim). Remember the picture. When you get a shape error, redraw it.

11 · Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-do the L2-normalization worked example (§8) from memory in 24 hours. Revisit §5 (broadcasting) and §6 (matmul) on day 7.

Next session (S002): we give arrays geometric meaning — a vector is an arrow, a matrix is a linear map, and the dot product is the reason q @ k.T shows up in every transformer on Earth.

Sticky note (keep on your desk): X / np.linalg.norm(X, axis=1, keepdims=True) — L2-normalize a batch. If you can read this and see the shapes, you can read 30% of any modern DL codebase.

Recall — answer from memory (no scrolling)

Stretch — for one extra hour of practice

Take a 2D array X of shape (1000, 50) representing 1000 embedding vectors of dimension 50. Without using any Python for loops, compute:

1. The mean-subtracted version (subtract the per-column mean from each row).
2. The cosine similarity matrix between every pair of rows (should be shape (1000, 1000)).
3. The top-5 most similar row indices for row 0.

If you can do all three with only NumPy operations and no loops, you're already at 60% of the operator fluency you need for the whole series.

In your own words

Explain to a friend, in one sentence, why NumPy is fast:

(Write it in a notebook. Come back tomorrow and see if it still makes sense.)

Spaced-review pointer

This session laid the groundwork for every future session. Specifically:

• Session 002 (vectors & matrices) uses the reshaping and matmul from §5 and §6.
• Session 003 (broadcasting) is entirely about the [:, None] / [None, :] idiom from §5.2.
• Session 004 (derivatives + chain rule) will use the axis semantics from §4 to compute gradients.

Before advancing to Session 002, re-do the "Try this" block in §3.2 from memory. If you can predict all 5 shapes without running the code, go.

Next-session teaser

Now that arrays are muscle memory, we're going to give them meaning. In Session 002 we ask: what is a vector, geometrically? What does the dot product actually compute — and why does it appear in every attention head, every embedding model, every recommender system on Earth? We'll get there by drawing arrows on paper, then computing dot products by hand, then re-deriving them in NumPy — and by the end you'll read q @ k.T in a transformer paper and just see what it means.

What to bring back tomorrow — sticky note

• Diagram: the (batch, seq_len, embed_dim) rectangle.
• Equation: X.sum(axis=k) collapses axis k and returns everything else.
• Snippet: normalized = X / np.linalg.norm(X, axis=1, keepdims=True) — L2-normalize a batch of vectors.

Previous: ← Series hub · Next: DL S002 · Vectors, Matrices, and the Geometry of Data →