DL S005 · Probability and Information — Surprise, Entropy, Cross-Entropy

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 5 / 80 · Module M01 · ~2 hours

The story

Here's the puzzle: every time a modern neural network is trained on any classification task — MNIST, ImageNet, GPT pre-training, LLaMA fine-tuning, image-to-text, text-to-image — the loss function is called "cross-entropy". Same words. Same three lines of code. Why? What's so special about this particular loss?

The short answer: cross-entropy is not really a loss function at all. It is the negative log-likelihood of your data under your model. Minimising it means maximising the probability your model assigns to what actually happened. That framing — "make what happened more likely under your model" — is the philosophical bedrock of the whole field. Every language model, every classifier, every diffusion sampler, every RLHF reward model, all of it, is playing exactly this game.

To get there we need three ideas, in order: (1) surprise — how astonished should you be to see event x? (2) entropy — the average surprise of a distribution; the "how random is this really" number. (3) cross-entropy — the average surprise of the true distribution measured with your model's beliefs. That third one is the loss. When your model is perfect, cross-entropy equals entropy (a floor you can't get below). When it's wrong, cross-entropy is higher — and the gap is called the KL divergence, which measures how wrong you are.

None of this requires more than high-school algebra. It does require slowing down. Read every equation twice.

Prerequisites

• Session 004 — you'll need the chain rule to derive the softmax gradient in §7.
• Comfort with log and exp. That's it.

We do NOT assume prior probability theory. We rebuild it from four rules.

1 · Probability in four rules

To do everything in this session, we need four facts about probabilities.

That's it. If you can hold those four facts, you have all the probability we need for this session.

1.1 The tiny worked example we'll use all session

A biased 3-sided die (yes, imagine a triangular pyramid) with outcomes \{cat, dog, fish\}:

p(cat)  = 0.7
p(dog)  = 0.2
p(fish) = 0.1
--------------
       Σ = 1.0 ✅

Simple. We'll compute entropy of this, then design a "model" q with different probabilities, and see the cross-entropy story unfold.

2 · Surprise — how astonished should I be?

Suppose I tell you the die came up fish. You should be more surprised than if I said cat, because fish is less likely. How much more?

Shannon proposed we quantify surprise as:

I(x) = -log p(x)         (in bits if log₂; in nats if ln)

Why -log?

• If p(x) = 1 (certain), then I = 0. No surprise. ✅
• If p(x) → 0 (nearly impossible), then I → ∞. Infinite surprise. ✅
• If two independent events both happen, surprise ADDS (because -log(p * q) = -log p + -log q). ✅ (Independent events have independent surprises — this is exactly the property we want.)
• It's the ONLY function (up to units) with all three properties. Shannon proved it.

Units. If you use log₂, surprise is in bits. If you use ln, it's in nats. In ML we use ln (base-e) because exp is smooth and easier to differentiate. Either is fine — it's a constant multiplier.

2.1 Compute the surprise for each outcome

Using ln:

I(cat)  = -ln(0.7)  ≈ 0.357 nats
I(dog)  = -ln(0.2)  ≈ 1.609 nats
I(fish) = -ln(0.1)  ≈ 2.303 nats

fish is much more surprising than cat. Feels right.

import numpy as np
p = np.array([0.7, 0.2, 0.1])
print(-np.log(p))
# [0.35667494 1.60943791 2.30258509]

3 · Entropy — the average surprise of a distribution

Entropy H(p) is the expected surprise you feel when sampling from p:

H(p) = Σ  p(x) * (-log p(x))
       x
     = -Σ p(x) log p(x)

It's the average number of "surprise units" per sample. Equivalently — and this is Shannon's magic — it's the minimum average code length needed to transmit samples from p losslessly. High-entropy distributions are hard to compress; low-entropy ones easy.

3.1 Compute H(p) for our die

H(p) = 0.7 * 0.357 + 0.2 * 1.609 + 0.1 * 2.303
     = 0.250      + 0.322       + 0.230
     ≈ 0.802 nats

H = -(p * np.log(p)).sum()
print(H)                 # 0.8018185525433372

3.2 Two useful extreme cases

Maximum entropy: uniform distribution. For n outcomes, H = ln n. For our 3-class die, max entropy would be ln 3 ≈ 1.099 — happening when p = [1/3, 1/3, 1/3]. Uniform means "least informative"; there's no structure to compress.

Minimum entropy: certainty. If p = [1, 0, 0], entropy is 0. There's no surprise at all — you already know what will happen.

Our biased die sits in between (0.802 nats), closer to certainty than to uniform, which matches the intuition that cat is likely.

3.3 Entropy as "how much randomness"

Entropy is a scalar-valued randomness meter. A fair coin: H = ln 2 ≈ 0.693 nats. A weighted coin favouring heads 90%: H ≈ 0.325 nats. A cheating coin that always lands heads: H = 0. Entropy is the answer to "how much randomness is there really?" — a question that seems fuzzy until you write the formula down.

4 · Cross-entropy — the average surprise UNDER YOUR MODEL

Here's the setup. There's a true distribution p (nature). You (the ML engineer) have proposed a model distribution q — your best guess at p. Now nature draws samples from p. You react to each sample with your model's surprise level.

Cross-entropy is the average surprise, according to q, of samples drawn from p:

H(p, q) = -Σ p(x) log q(x)

Note the asymmetry. The outer weighting is p (because that's what nature actually samples), but the log is log q (because that's what YOU believe about how surprising the outcome is).

4.1 Worked example

Say your model q is:

q(cat)  = 0.5
q(dog)  = 0.3
q(fish) = 0.2

Compute:

H(p, q) = -[ 0.7 * ln(0.5) + 0.2 * ln(0.3) + 0.1 * ln(0.2) ]
        = -[ 0.7 * (-0.693) + 0.2 * (-1.204) + 0.1 * (-1.609) ]
        = -[ -0.485 - 0.241 - 0.161 ]
        =  0.887 nats

q = np.array([0.5, 0.3, 0.2])
Hpq = -(p * np.log(q)).sum()
print(Hpq)               # 0.88736

Compare to H(p) = 0.802. The cross-entropy is larger than the entropy. This is not an accident — it is a theorem.

4.2 The theorem — cross-entropy is always at least entropy

Gibbs' inequality: For any two distributions p and q over the same outcomes,

H(p, q) ≥ H(p)

with equality iff p = q. Proof sketch: use ln(x) ≤ x - 1 on the ratio q/p. Do it once on paper; not needed daily.

Interpretation: Your model q costs you extra surprise, on average, compared to knowing the true p. That extra cost is a distance measure.

4.3 The gap has a name — KL divergence

KL(p || q) = H(p, q) - H(p) = Σ p(x) log(p(x) / q(x))

KL(p || q) ≥ 0, and = 0 iff p = q. It's "how many extra nats of surprise you pay per sample by using your model q instead of the true p."

For our example:

KL(p || q) = 0.887 - 0.802 = 0.085 nats

KL is asymmetric. KL(p||q) ≠ KL(q||p) in general. This matters: minimising KL(p||q) (forward KL) pushes q to cover all of p's support; minimising KL(q||p) (reverse KL) makes q collapse onto p's modes. Session 049 (variational autoencoders) and Session 058 (RLHF) both hinge on which direction of KL you pick.

5 · Cross-entropy = negative log-likelihood

This is the punchline. It is why cross-entropy is used everywhere in ML.

Setup. You have a dataset of N samples x₁, ..., x_N, drawn i.i.d. from the true (unknown) distribution p. Your model gives probabilities q_θ(x) parameterised by weights θ. You want to pick θ to make the data likely.

The likelihood of the whole dataset under your model:

L(θ) = Π  q_θ(x_i)
       i

We take the log (turns product into sum, keeps the argmax) and negate (turns "maximise" into "minimise" for gradient descent):

NLL(θ) = -Σ log q_θ(x_i)
          i

Divide by N (empirical average):

average_NLL = -(1/N) Σ log q_θ(x_i)

Now here's the crucial observation. The empirical distribution p̂ puts mass 1/N on each observed data point. So:

average_NLL = -Σ p̂(x) log q_θ(x)  =  H(p̂, q_θ)
              x

Cross-entropy of the empirical data distribution against your model IS the average negative log-likelihood. Minimising cross-entropy loss is literally maximum likelihood estimation. This is why we do it.

5.1 For one-hot classification labels

In classification, each example has a single true class y ∈ {1, ..., K}. That means p̂ for one example is [0, ..., 0, 1, 0, ..., 0] with the 1 in position y. Then:

H(p̂, q) = -Σ p̂_k log q_k = -log q_y

Cross-entropy loss on one example = -log(predicted probability of the correct class). That's it. That's the whole formula. When your model gives probability 1 to the correct class, loss = 0. When it gives probability 0, loss = ∞ (you were infinitely surprised). Beautiful, extreme, exactly what you want.

6 · Softmax — turning logits into probabilities

Networks output arbitrary real numbers ("logits"), not probabilities. We convert with the softmax:

softmax(z)_k = exp(z_k) / Σ_j exp(z_j)

Properties:

• Output is nonnegative (because exp > 0).
• Sums to 1 (denominator is exactly the sum of numerators).
• Preserves order (bigger z_k → bigger softmax_k).
• Shift-invariant: softmax(z) = softmax(z + c) for any constant c. Useful for numerical stability.

6.1 Worked example

Logits z = [2.0, 1.0, 0.1]:

exp(z) = [7.389, 2.718, 1.105]
sum    = 11.212
softmax = [0.659, 0.242, 0.098]

z = np.array([2.0, 1.0, 0.1])
p = np.exp(z) / np.exp(z).sum()
print(p)          # [0.65900114 0.24243297 0.09856589]

6.2 The log-sum-exp trick (why your loss doesn't overflow)

If z_k = 1000, exp(z_k) overflows to inf. Since softmax is shift-invariant, we subtract max(z) before exponentiating:

def softmax_stable(z):
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

Every DL framework does this internally. If you write a custom softmax in a kernel and skip the shift, you will get NaNs at scale. It's the #1 cause of "loss suddenly became NaN at epoch 3" — Session 006 covers this in detail.

7 · The softmax + cross-entropy gradient — the punchline

Combine everything: given logits z and true class y, the loss is:

L = -log(softmax(z)_y) = -z_y + log Σ_j exp(z_j)

We want ∂L/∂z_k. It's shorter than you'd think.

Let s_k = softmax(z)_k. Then:
 
∂L/∂z_k =  s_k - 1[k == y]

The gradient is just predicted_probs - one_hot_target. Three lines of code:

def softmax_ce_grad(z, y):
    s = softmax_stable(z)
    s[y] -= 1.0
    return s

That's the entire gradient of softmax + cross-entropy. All the logs, all the exps, they cancel. It's the cleanest gradient in deep learning, and it's why softmax + cross-entropy is the default classification head everywhere.

7.1 Derivation (do this once on paper)

Do that derivation once. Ever. Then use the three-line implementation forever. This is the gradient every classification network uses on the last layer.

7.2 Numeric check

z = np.array([2.0, 1.0, 0.1])
y = 0
 
def L(z_):
    s = softmax_stable(z_)
    return -np.log(s[y])
 
# analytic
g_analytic = softmax_ce_grad(z, y)
 
# numeric
h = 1e-6
g_num = np.zeros_like(z)
for k in range(len(z)):
    z_p = z.copy(); z_p[k] += h
    z_m = z.copy(); z_m[k] -= h
    g_num[k] = (L(z_p) - L(z_m)) / (2*h)
 
print("analytic:", g_analytic)
print("numeric :", g_num)
# analytic: [-0.34099886  0.24243297  0.09856589]
# numeric : [-0.34099886  0.24243296  0.09856589]  ✅

Match to 8 decimal places. Beautiful.

8 · Where this all shows up in the series

Every future session with a loss function uses this machinery. A partial map:

• S011 (MNIST in numpy) — you'll implement softmax_ce_grad in exactly the three-line form above.
• S014 (PyTorch autograd) — F.cross_entropy(logits, targets) does all of this in one call, including the log-sum-exp trick.
• S041 (training your GPT on Tiny Shakespeare) — the LM loss IS cross-entropy over vocabulary logits at every token position. Millions of softmaxes per batch.
• S051 (SFT) — same cross-entropy, on instruction data.
• S058 (RLHF) — KL divergence (§4.3) appears as a penalty keeping the policy close to a reference model. Same math.

The point: you learn probability once. You use it forever.

8b · Cross-entropy in the wild — next-token prediction and 2024–2025 alignment

Everything you just learned about cross-entropy is the loss that trains every language model on Earth. Let's connect the dots — and then meet the four 2023–2025 successors that reshape the objective without leaving log-probability land.

8b.1 · The one loss every LLM is trained on

A language model, unrolled, computes:

$P(\text{token}_t \mid \text{token}_1, \dots, \text{token}_{t-1}) = \text{softmax}(z_t)$

where $z_t \in \mathbb{R}^V$ is the logit vector over a vocabulary of size $V \approx 32{,}000$ to $256{,}000$. The training loss for a batch of sequences is literally the average cross-entropy:

logits = model(input_ids)          # (B, T, V)
targets = input_ids[:, 1:]         # shift by one
loss = F.cross_entropy(
    logits[:, :-1].reshape(-1, V),
    targets.reshape(-1),
)

That's it. Llama 3.1 405B, GPT-4, Gemini 2, Claude 3.5, DeepSeek-V3 — they were all pretrained by minimizing exactly this expression over trillions of tokens. When you see "loss = 1.8 nats per token" in a training log, that is $-\log q_y$ averaged — the number you learned to compute today, at scale.

Per-token perplexity is $e^L$ where $L$ is the mean cross-entropy in nats. A model at loss 1.8 nats has perplexity $\approx 6.05$: on average, when it's predicting a token, it's about as uncertain as if it were picking uniformly from 6 candidates. Chinchilla-optimal 70B checkpoints hit ~1.9 nats on English. GPT-4 is estimated (from public leaks/eval) around ~1.5. Every 0.1 nat drop is a headline result.

8b.2 · DeepSeek-V3's multi-token prediction (Dec 2024)

DeepSeek-V3 added a small auxiliary head that predicts two tokens ahead in addition to the standard next-token cross-entropy. The training loss becomes:

$L = L_{\text{main}}(\text{token}_{t+1}) + \lambda \, L_{\text{aux}}(\text{token}_{t+2})$

Both are cross-entropies. The auxiliary loss pushes the model to encode information about not just the next token but the one after — which at inference time enables speculative decoding without a separate draft model. Every serious 2025 pretraining recipe is looking at this.

8b.3 · KL, DPO, KTO, ORPO, SimPO — the 2023–2025 preference tuning line

RLHF used to be a hairy PPO loop with a separate reward model. Then DPO (Rafailov et al., 2023) showed the entire objective can be rewritten as a simple classification loss over pairs of (preferred, dispreferred) responses:

$L_{\text{DPO}} = -\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$

Read through the tokens: this is (a) two log-probability ratios (log of two softmax outputs, i.e., cross-entropy-like quantities you already know), (b) a difference ("preferred should be higher than dispreferred"), (c) a sigmoid + log — which is binary cross-entropy, which is a special case of §5. So DPO is fundamentally cross-entropy applied to a ratio of log-probs. Everything you learned today generalizes directly.

Successors, all using the same log-prob primitives:

• KTO (Ethayarajh et al., 2024) — replaces the pairwise term with a Kahneman-Tversky value function; works on unpaired data.
• ORPO (Hong et al., 2024) — combines SFT and preference in one loss, no reference model.
• SimPO (Meng et al., 2024) — length-normalized log-probs, drops the reference model entirely.

All are variants of "push up log-prob of good, push down log-prob of bad". If you can read cross-entropy, you can read any of them.

8b.4 · Load-balancing loss in MoE (Mixtral, DeepSeek-V3)

Mixture-of-Experts models add an auxiliary loss to keep experts equally used:

$L_{\text{aux}} = \alpha \sum_i f_i \cdot P_i$

where $f_i$ is the fraction of tokens routed to expert $i$ and $P_i$ is the average router probability for that expert. If routing is imbalanced (all tokens go to one expert), the product explodes; balanced routing minimizes it. Even MoE architecture has cross-entropy-adjacent auxiliaries under the hood.

8b.5 · Label smoothing — the tiny CE tweak with big effects

Instead of a one-hot target, use [0.1/K, ..., 0.9 + 0.1/K, ..., 0.1/K] — give a little probability mass to every class. Introduced by Szegedy et al., 2016 for Inception-v3, still standard practice in 2025 image models and machine translation. Effect: prevents the model from becoming over-confident (softmax outputs saturating at 1.0), which improves calibration and often improves generalization by 0.3–0.7%. In the CE formula §4 you just replace the delta target $\delta_{ky}$ with a smoothed target $t_k$ — the loss is now $-\sum_k t_k \log q_k$. Same shape, softer target.

Further reading:

• Rafailov et al. "Direct Preference Optimization" (2023).
• Meng et al. "SimPO: Simple Preference Optimization with a Reference-Free Reward" (2024).
• DeepSeek "DeepSeek-V3 Technical Report" (Dec 2024).
• Sebastian Raschka's "LLM Training: RLHF and Its Alternatives" blog series.

9 · War stories

10 · Diagram — the whole picture in one figure

Read the arrow "true class = 0 → gradient is q - one_hot" until it feels obvious. That's the atom of every classifier ever built.

11 · Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: Re-derive the softmax+CE gradient tomorrow from scratch. Revisit §8b (LLM CE + DPO) on day 7.

Next session (S006): SGD, momentum, Adam, and the numerical-stability tricks (log-sum-exp, gradient clipping, fp16 loss scaling) that keep production training runs alive.

Sticky note: s = softmax_stable(z); s[y] -= 1 — three lines, that's the whole softmax+CE gradient.

Recall — from memory

Stretch — for one extra hour

Implement softmax_stable, ce_loss, and softmax_ce_grad from scratch. On a random batch (z: (32, 10), y: (32,)), verify your analytic gradient against a numerical gradient to 6 decimal places using central differences. Then compute the KL divergence between your model's average predictions and the true class marginals — a great sanity check on classifiers.

In your own words

Explain why we use cross-entropy loss for classification, in one sentence:

Spaced-review pointer

• From S004 — the chain rule is what we used to derive the softmax + CE gradient in §7.1. Redo that derivation without looking if you can.
• From S002 — the shape rules will bite the moment you generalise §7 to batches; s.shape == y_onehot.shape is your friend.

Next-session teaser

You now know (a) how to compute gradients (S004) and (b) what loss function to minimise (S005). Next session: the algorithm that actually turns those gradients into learning. We'll cover SGD, momentum, Adam, learning-rate intuition, and the numerical-stability tricks (log-sum-exp, fp16 clipping) that keep training from exploding at scale. After Session 006 you'll have the complete "chain rule + cross-entropy + gradient descent" trinity that underlies every trained neural network on earth.

What to bring back tomorrow — sticky note

• Diagram: the "logits → softmax → probs → -log at true class" flow from §10.
• Equation: ∂L/∂z_k = s_k - δ_{ky} — the entire gradient of softmax + CE, in one line.
• Snippet: s = softmax_stable(z); s[y] -= 1 — done. That's the gradient. Three lines.

Previous: ← DL S004 · Derivatives, the Chain Rule, and Computation Graphs · Next: DL S006 · Optimization Intuition + Numerical Stability →