DL S003 · Broadcasting — The Rule That Runs the World

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 3 / 80 · Module M01 · ~2 hours

The story

I'm going to make an unpopular claim. Broadcasting is the single most underrated concept in numerical computing. More important than matrix multiplication. More consequential than autograd. Here's why: matrix multiplication has one rule, (m, k) @ (k, n) → (m, n), and if you get it wrong Python screams at you immediately. Broadcasting has three rules, and if you get it wrong Python happily runs your code, silently produces a wrong-shape result, and your model trains on garbage until you notice weeks later.

The scary part isn't the rules — they're short. It's the silence when things go wrong. A (3,) array added to a (3, 3) matrix will work. A (3,) array added to a (3, 1) matrix will work. A (1, 3) added to a (3, 1) will work — and give you back a (3, 3), which is almost never what you wanted. Broadcasting is powerful and dangerous in the same breath.

By the end of this session, you'll be able to look at any NumPy or PyTorch line and know, without running it, what shape falls out. That skill alone will save you hundreds of hours over the next 77 sessions.

Prerequisites

• Session 001 — you need .shape, .reshape, [:, None], axis= fluency.
• Session 002 — matrix multiplication and the shape rule.

1 · What is broadcasting, in one sentence?

Broadcasting is NumPy's rule for making two arrays of different shapes 'compatible' for elementwise operations, by pretending the smaller one is repeated along missing or size-1 axes — without actually copying it in memory.

That's the whole thing. Read it twice.

Let's see it in action:

import numpy as np
X = np.array([[1, 2, 3],
              [4, 5, 6]])                # shape (2, 3)
b = np.array([10, 20, 30])                # shape (3,)
 
X + b
# array([[11, 22, 33],
#        [14, 25, 36]])

Notice: X has shape (2, 3), b has shape (3,). They're not the same shape. But NumPy quietly stretched b to look like (2, 3) — i.e., pretended b was [[10, 20, 30], [10, 20, 30]] — and then added element by element. It didn't actually allocate a new array; it just walked through the addition as if b were duplicated.

Why we care: without broadcasting, you'd have to write

b_repeated = np.tile(b, (2, 1))     # actually allocates a (2, 3) array
X + b_repeated

which wastes memory and is slower. Broadcasting is what makes X @ W1 + b1 legal without any manual reshaping.

2 · The three rules

NumPy's broadcasting algorithm has exactly three rules. Learn them cold.

If at any axis the sizes are neither equal nor include a 1, you get a shape error. That's it. That's every rule.

2.1 Walking through Rule 1 (right-alignment)

Say we're adding shapes (3,) and (2, 3). Not the same length? Pad the shorter ((3,)) with 1s on the left until it has the same rank:

Now they have the same rank. Move to Rule 2.

2.2 Walking through Rule 2 (compatibility)

Axis-by-axis check:

2.3 Walking through Rule 3 (stretching)

Any axis where one side is 1 gets stretched to match the other:

That's the mental algorithm. Let's practice.

3 · Worked examples — do these on paper

For each pair of shapes, apply Rules 1–3 and predict the output shape.

3.1 Adding a scalar to a matrix

Scalars broadcast against literally anything.

np.zeros((3, 4)) + 5     # shape (3, 4), all fives

3.2 Row vector + matrix

3.3 Column vector + row vector = matrix (the WEIRD one)

This one shocks beginners. A column vector plus a row vector gives you an outer product-like matrix:

a = np.array([[1], [2], [3]])         # shape (3, 1)
b = np.array([[10, 20, 30, 40]])      # shape (1, 4)
a + b
# [[11, 21, 31, 41],
#  [12, 22, 32, 42],
#  [13, 23, 33, 43]]

This is powerful — and dangerous. It's the fingerprint of one of the most common bugs in ML code, which we'll cover in §7.

3.4 The one that FAILS

np.zeros((3, 4)) + np.zeros((2, 4))
# ValueError: operands could not be broadcast together with shapes (3,4) (2,4)

Good. NumPy is protecting you.

4 · Constructing broadcasts on purpose — the [:, None] idiom

You saw this in Session 001 §5.2. Now let's use it deliberately.

4.1 The problem

Say you have a batch of 5 vectors, each of dimension 3, and you want to compute the pairwise distances between every pair of vectors. You want a (5, 5) distance matrix.

Naive Python:

distances = np.zeros((5, 5))
for i in range(5):
    for j in range(5):
        distances[i, j] = np.linalg.norm(X[i] - X[j])

Two nested for-loops. Slow. Ugly. But correct.

4.2 The broadcasting solution

Insight: to compute X[i] - X[j] for every pair, we want an array of shape (5, 5, 3) where entry (i, j) is X[i] - X[j]. That's a subtraction between two arrays of shape (5, 1, 3) and (1, 5, 3) — which broadcast to (5, 5, 3).

X = np.random.randn(5, 3)             # (5, 3)
A = X[:, None, :]                     # (5, 1, 3)   — X reshaped as "5 rows, each a mini-batch of 1"
B = X[None, :, :]                     # (1, 5, 3)   — X reshaped as "1 row of 5 vectors"
diffs = A - B                         # (5, 5, 3) — every pairwise difference
distances = np.linalg.norm(diffs, axis=-1)   # (5, 5) — norms along the last axis

Five lines. No Python loop. On big arrays this is ~100× faster than the loop version.

This exact pattern — "insert a size-1 axis so the arrays broadcast against each other in a specific direction" — is one of the two or three most useful moves in all of NumPy. Once you internalize it, you'll use it every day.

4.3 Even shorter, with np.newaxis

np.newaxis is literally an alias for None:

X[:, np.newaxis, :]       # exactly the same as X[:, None, :]

Some people prefer np.newaxis for readability. I use None because it's shorter. Both are correct.

5 · How broadcasting makes neural nets legal

Let's revisit the little forward pass from Session 002 §6.3:

X  = np.random.randn(32, 784)     # 32 flattened MNIST images
W1 = np.random.randn(784, 128)    # hidden layer weights
b1 = np.random.randn(128)         # hidden layer biases

Compute X @ W1 + b1:

• X @ W1 has shape (32, 128).
• b1 has shape (128,).

Apply the broadcasting rules:

One bias vector, added independently to every example in the batch, with zero manual reshaping. That's what broadcasting is buying us.

Every neural network you'll ever write uses this pattern. Every single one.

6 · Broadcasting is zero-copy (this is why it's fast)

I said earlier that NumPy "pretends" the smaller array is duplicated. It doesn't actually allocate.

Let's prove it:

import sys
b = np.arange(1000, dtype=np.float64)          # shape (1000,), ~8 KB
big = np.zeros((1000, 1000), dtype=np.float64) # shape (1000, 1000), ~8 MB
 
result = big + b                               # broadcast add

If broadcasting were done by making a (1000, 1000) copy of b and then adding, we'd allocate an extra 8 MB temporarily. It doesn't. NumPy walks through the addition with a stride trick: for each (i, j) cell, it looks up big[i, j] and adds b[j], using strides to index into b as if it were 2D. Zero copy.

Implication: you can broadcast a small vector against a giant matrix with no memory penalty. This is why we can add a bias vector to every row of a batch of 10,000 examples without breaking a sweat.

7 · War stories — the bugs broadcasting cost me

predictions = logits.argmax(axis=-1)             # shape (100,)
labels      = np.array([[0], [1], [2], ...])     # shape (100, 1)  — someone else's data
accuracy    = (predictions == labels).mean()

loss_per_example = (predictions - targets) ** 2      # I wanted shape (batch,)
loss = loss_per_example.mean()

img = ...                          # shape (3, 224, 224)  — CHW format
means = img.mean(axis=(1, 2))      # shape (3,)
normalized = img / means           # I wanted (3, 224, 224)

means = img.mean(axis=(1, 2))[:, None, None]   # shape (3, 1, 1) — clearly per-channel
normalized = img / means                        # (3, 224, 224) ✅

Common thread across all three war stories: the bug wasn't Python being wrong, it was Python being too helpful. Broadcasting silently made the "wrong" op work. The fix in every case is the same: be explicit about shapes at boundaries. .squeeze(), .reshape(), [:, None], and defensive asserts.

8 · The one weird rule for matmul

@ (matmul) has its own broadcasting behaviour for high-rank tensors, and it's slightly different from elementwise ops. Worth knowing now so you don't get burned in Session 040.

The rule: for A @ B, the LAST TWO axes are treated as the matrix dimensions; everything before them is broadcast batchwise.

A = np.random.randn(32, 5, 3)     # 32 matrices, each (5, 3)
B = np.random.randn(32, 3, 7)     # 32 matrices, each (3, 7)
C = A @ B                          # shape (32, 5, 7) — matmul per batch

You can also broadcast the batch axes:

A = np.random.randn(32, 5, 3)     # 32 matrices, each (5, 3)
B = np.random.randn(3, 7)         # single (3, 7) matrix
C = A @ B                          # shape (32, 5, 7) — B broadcast across the batch

Every transformer attention op uses this. The Q, K, V tensors have shape (batch, heads, seq, d_head); the Q @ K.T op treats the last two axes as the matrix and broadcasts across batch and heads. That's why you can write attention as a two-liner regardless of batch size or head count.

8b · Broadcasting in 2025 transformer code — five patterns you'll meet weekly

Every time a modern LLM computes attention, applies a bias, or does layer normalization, broadcasting is happening under the hood. Here are the five idioms you'll see over and over in nanoGPT, Llama-style code, and HuggingFace transformers — read them until each one triggers an instant “ah, that's a broadcast”.

8b.1 · Adding positional bias to attention scores

# scores has shape (B, nh, T, T)  — batch, heads, query pos, key pos
# alibi_bias has shape (nh, T, T)  — per-head positional bias
scores = scores + alibi_bias        # broadcasts nh, T, T; batch dim stretches

This is ALiBi (Press, Smith & Lewis, 2022) — a linear bias on attention scores that extrapolates to longer sequences than seen at training time. The broadcast is invisible but critical: adding a shape-(nh, T, T) tensor to a shape-(B, nh, T, T) tensor works because axis 0 pads to 1 and stretches. If you accidentally shape the bias as (T, T) instead of (nh, T, T), it still broadcasts — same result across all heads. That's a subtle model change hidden behind a shape choice.

8b.2 · Causal masking

# mask has shape (T, T), lower-triangular of ones, zeros above
# scores has shape (B, nh, T, T)
scores = scores.masked_fill(mask == 0, float("-inf"))

The (T, T) mask broadcasts against (B, nh, T, T). One 4KB mask, applied to a 4GB score tensor. Zero copy. This is the reason a transformer's causal attention is cheap to enforce.

8b.3 · RMSNorm scale

# x has shape (B, T, d)
# gamma has shape (d,)  — learned per-feature scale
rms = torch.sqrt((x ** 2).mean(-1, keepdim=True) + eps)   # (B, T, 1)
x   = x / rms * gamma                                     # (B, T, d)

Two broadcasts in three lines. x / rms: shape-(B, T, d) divided by (B, T, 1). * gamma: shape-(B, T, d) times (d,) — gamma pads to (1, 1, d), stretches. This is RMSNorm (Zhang & Sennrich, 2019), used in Llama, Gemma, Qwen, DeepSeek — essentially every 2024–2025 LLM. LayerNorm has a nearly identical broadcast shape story with an extra mean subtraction and bias.

8b.4 · RoPE rotation

# q has shape (B, nh, T, hd)
# cos, sin have shape (T, hd)  — precomputed rotation table
cos = cos[None, None, :, :]        # (1, 1, T, hd)
sin = sin[None, None, :, :]        # (1, 1, T, hd)
q_rot = q * cos + rotate_half(q) * sin

The None, None axes are added explicitly so the broadcast is unambiguous. Without them, PyTorch would still broadcast (padding on the left), but writing it explicitly is a discipline that saves you when the tensor gains a fifth axis (e.g., for speculative decoding). This is RoPE (Su et al., 2021), refined by YaRN (2023) and LongRoPE (2024) for 128K–10M context windows.

8b.5 · Grouped-Query Attention repeat_interleave

Llama 3, Gemma 2/3, Qwen 2 all use GQA: fewer KV heads than Q heads. To make the shapes match for the Q @ K.T matmul:

# q has shape (B, nh_q, T, hd)   e.g. nh_q = 32
# k has shape (B, nh_kv, T, hd)  e.g. nh_kv = 8
# We want k to broadcast against q's head dim.
k = k.repeat_interleave(nh_q // nh_kv, dim=1)   # (B, nh_q, T, hd)

This is not a pure broadcast — repeat_interleave really allocates. In performance-critical code (vLLM, TensorRT-LLM), this is replaced by a fused kernel that does the effective broadcast without materializing. Same math, different memory story. GQA (Ainslie et al., 2023) trades a tiny quality loss for a huge KV-cache memory saving — typically 4–8×.

Further reading:

• Press, Smith, Lewis. "Train Short, Test Long: ALiBi" (2022).
• Ainslie et al. "GQA: Training Generalized Multi-Query Transformer Models" (2023).
• Su et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021).
• Karpathy's nanoGPT — read model.py and try to spot every broadcast. There are ~15.

9 · The full mental model — a decision tree

When you see any operation A op B, run this mental checklist:

That's the whole game. Master this decision tree and shape errors become 10× easier.

10 · Diagram — the "stretch" mental model

Draw this on paper once. It burns in.

import numpy as np
row = np.arange(4)                      # shape (4,)
b = np.broadcast_to(row, (1_000_000, 4))
print(b.shape, b.strides)               # (1000000, 4) (0, 8)  <- stride 0
print(b.nbytes, row.nbytes)             # 32000000 vs 32 (nbytes LIES; no alloc)
print(b.base is row)                    # True -> same memory
b[0, 0] = 1                             # ValueError: read-only
# and the expensive case, for contrast:
x = np.zeros((1000, 8)); c = np.zeros((256, 8))
print((x[:, None, :] - c[None, :, :]).nbytes)   # 16 MB, really allocated

11 · Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: Redo one worked example from §3 tomorrow. Revisit §8b (the five 2025 patterns) on day 7.

Next session (S004): Derivatives + chain rule + computation graphs — the mathematical machinery that lets losses flow backwards through the tensors you just learned to shape.

Sticky note: the three rules — (1) align on the right and left-pad with 1s, (2) axes compatible if equal or one is 1, (3) stretch is conceptual (zero-copy strides).

Recall — from memory

Stretch — for one extra hour

Write a NumPy function pairwise_cosine_similarity(A, B) where A has shape (m, d) and B has shape (n, d), and the output has shape (m, n) giving the cosine similarity between every row of A and every row of B. No Python loops. Use broadcasting + matmul.

Then benchmark it against a nested-for-loop version on m = n = 1000, d = 128. Report the speedup. (Spoiler: 100–500×.)

In your own words

Explain broadcasting in one sentence:

Spaced-review pointer

• From S001 — you're using .shape, [:, None], and axis semantics constantly.
• From S002 — the shape rules for @ combine with today's broadcasting for the full transformer attention op.

Session 004 will use these skills to visualize computation graphs — where every node is a tensor of a specific shape, every edge is an op, and gradients flow through the graph following, yes, exactly the broadcasting rules in reverse.

Next-session teaser

You've mastered making shapes agree. Now we ask the deeper question: how does one number in a big graph of numbers depend on another? Session 004 introduces derivatives, the chain rule, and computation graphs — the mathematical machinery that makes learning possible. If today felt like syntax, tomorrow is where things start to move. We'll compute ∂L/∂w by hand on a 3-node computation graph, then generalize it to any graph, then implement it in ~30 lines of NumPy at the end. That's the seed of every autograd engine ever written, including PyTorch's.

What to bring back tomorrow — sticky note

• Diagram: the "stretch a small array up to a big array" picture from §10.
• Equation (well, rules): the three broadcasting rules.
• Snippet: A - B where A = X[:, None, :] and B = X[None, :, :] gives you all pairwise differences.

Previous: ← DL S002 · Vectors, Matrices, and the Geometry of Data · Next: DL S004 · Derivatives, the Chain Rule, and Computation Graphs →