Probability, cross-entropy, and stable logits

Course map · DL005 · Probability and information

The story

Cross-entropy asks how surprised a model is by outcomes drawn from another distribution. For categorical observations, its empirical version is average negative log-likelihood. That connection explains a common classification objective; it does not mean every diffusion model, reward model or learning algorithm uses the same loss.

We will rebuild probability rules, compute the original cat/dog/fish example, prove the entropy inequality, derive the logit gradient and inspect the modern objectives that reuse some—but not all—of these ingredients. A low loss is neither a truth guarantee nor a calibration certificate.

Prerequisites

DL004's chain rule, natural logarithms, exponentials and DL003's reduction axes. Here p denotes the target distribution and q the model unless a local code variable is explicitly named otherwise.

1 · Probability in four rules

For a finite discrete sample space, probabilities lie in [0,1] and sum to1 over mutually exclusive exhaustive outcomes. Independent events multiply: $P(A\cap B)=P(A)P(B)$. In general use $P(A\cap B)=P(A\mid B)P(B)$, with conditional probability defined when $P(B)>0$. Summing joint probabilities over one variable gives the other variable's marginal. Bayes' rule follows by writing the joint in both orders.

Probability zero means impossible in this finite model. Continuous distributions need a different qualification: a particular real-valued point can have probability zero without being logically impossible. We use finite categorical distributions throughout the entropy calculations below.

1.1 The tiny worked example we'll use all session

Imagine a categorical draw, not a geometrically dubious three-faced die: cat has probability0.7, dog0.2, fish0.1. Their sum is1. For two independent draws, cat then fish has probability0.07. If the second draw depends on the first, replace0.1 by the corresponding conditional probability. That distinction is essential for language sequences, whose tokens are not independent.

2 · Surprise — how astonished should I be?

The self-information of a possible event is $I(x)=-\log p(x)$. A certain event costs zero, rare events cost more, and independence makes surprises add because logarithms turn products into sums. With regularity assumptions such as continuity, the additive functional equation motivates a logarithm up to a positive scale. The listed properties alone should not be presented as an assumption-free uniqueness proof.

Natural logs give nats; base-two logs give bits. Divide nats by $\ln2$ to convert to bits. Both bases are differentiable; natural logs simply avoid an extra constant in common calculus formulas. Think of surprise as a coding bill: an unexpected outcome costs more under a code designed for the model's beliefs.

2.1 Compute the surprise for each outcome

Cat: $-\ln.7=0.35667494$ nats. Dog:1.60943791. Fish:2.30258509. Cat-then-fish under independence costs the sum,2.65926004 nats, equal to $-\ln.07$.

3 · Entropy — the average surprise of a distribution

Use $0\log0=0$ by a limit. Entropy is an expectation over draws from p; it is not the surprise of whichever event happens to be most common. In lossless coding, entropy is an asymptotic lower bound on average bits per symbol under the specified source model. Integer codeword lengths, finite blocks and overhead can prevent attaining it exactly for one symbol.

3.1 Compute H(p) for our die

The contributions are $0.7(0.35667494)=0.24967246$, $0.2(1.60943791)=0.32188758$, and $0.1(2.30258509)=0.23025851$. Sum: 0.80181855 nats, or about1.15678 bits. The frequent cat contributes less than dog here because surprise and frequency both matter.

3.2 Two useful extreme cases

A point mass has entropy0. For K outcomes the uniform distribution has entropy ln K. To prove it is maximal, take uniform u in the KL identity below: $D_{KL}(p\|u)=\ln K-H(p)\ge0$. Our three-class entropy0.802 is below ln3≈1.099; numerically it is closer to the uniform endpoint than to zero, correcting the original opposite statement.

3.3 Entropy as "how much randomness"

A fair coin has ln2≈0.693 nats; a0.9/0.1 coin has about0.325. Entropy depends on the variables and conditioning chosen. Conditional label entropy at fixed inputs is different from entropy of class marginals. A finite training set can be memorized even when the population has noisy labels, so a nonzero population entropy does not impose the same lower bound on an overfitted empirical training loss.

4 · Cross-entropy — the average surprise UNDER YOUR MODEL

Nature supplies the weighting p; the model q supplies the coding costs. Both must be normalized on the same outcome space. If p assigns positive mass where q assigns zero, cross-entropy is infinite. Terms with p=0 contribute zero, not a runtime 0 * log(0) NaN.

4.1 Worked example

Let q=(0.5,0.3,0.2). The contributions are $0.7\ln2=0.48520303$, $-0.2\ln.3=0.24079456$, and $-0.1\ln.2=0.16094379$. Their sum is 0.88694138 nats. The older0.88736 printed result was wrong. Subtract entropy to obtain0.08512283 nats of excess average cost.

For logits $(\ln3,0)$, q=(.75,.25). Observing class0 costs0.287682 nats; observing class1 costs1.386294. A confident error is expensive even though both outputs sum to1.

4.2 The theorem — cross-entropy is always at least entropy

On p's positive support, apply $-\ln r\ge1-r$ to r=q/p:

If q vanishes on positive p support, the left side is infinite and the inequality still holds. Equality requires q=p on that support and no q mass elsewhere. This supplies the missing conditions behind Gibbs' inequality; it is not merely a numerical observation on our example.

4.3 The gap has a name — KL divergence

KL is directional and lacks the symmetry and triangle inequality of a metric. “Mode-covering” forward KL and “mode-seeking” reverse KL describe tendencies under restricted model families and particular distributions, not laws that reverse KL always collapses. If the family contains p, both orientations have the same optimum q=p. In optimization p must be fixed for minimizing H(p,q) to be equivalent to minimizing that KL.

5 · Cross-entropy = negative log-likelihood

For independent observations, likelihood is the product of their model probabilities. Taking a log gives a sum; negating turns maximization into minimization. Group repeated observations by their empirical frequency $\hat p(x)=\mathrm{count}(x)/N$:

For supervised classification, use conditional probabilities $q_\theta(y_n\mid x_n)$. The empirical conditional objective averages the losses at the observed inputs; it is not merely fitting the unconditional class histogram.

5.1 For one-hot classification labels

A one-hot target has a1 at y and zeros elsewhere, so the sum reduces to $-\log q_y$. At q_y=1 the loss is0; at q_y=0 it is infinite. With zero logits in binary classification the loss is ln2 regardless of class balance, provided there is no weighting or other modified reduction. Random nonzero logits need not start there.

Squared error on probabilities is also a valid scoring objective (related to the Brier score). It is not forbidden for classification. It differs from categorical maximum likelihood and has different gradient behaviour; choose deliberately rather than declaring all alternatives mathematically invalid.

6 · Softmax — turning logits into probabilities

For finite real logits the exact probabilities are positive and sum to1. Softmax preserves score order and is invariant to adding a common constant. Floating-point tails can round to zero. Normalize over classes, not examples.

6.1 Worked example

For z=(2,1,.1), exponentials are approximately(7.389056,2.718282,1.105171), with sum11.212509. The probabilities are approximately(.659001,.242433,.098566). For y=0 the loss is about0.41703 nats, not the value computed from the coarsely rounded0.66.

6.2 The log-sum-exp trick (why your loss doesn't overflow)

With m=max z, subtract m before exponentiation. Compute log-probabilities as $(z_i-m)-\log\sum_j e^{z_j-m}$, and loss as its negative at y. This avoids forming a tiny probability only to log it, and avoids subtracting two huge unshifted logit-scale quantities. It does not repair NaN inputs or an all-negative-infinity masked row; define an explicit policy for such rows.

import numpy as np
 
def loss_grad(z, y):
    z = np.asarray(z, dtype=np.float64)
    shifted = z - z.max()
    e = np.exp(shifted)
    p = e / e.sum()
    loss = np.log(e.sum()) - shifted[y]
    g = p.copy()
    g[y] -= 1
    return float(loss), g
 
z = np.array([np.log(3), 0.])
loss, g = loss_grad(z, 0)
np.testing.assert_allclose(g, [-.25, .25])
assert abs(loss + np.log(.75)) < 1e-12
assert abs(loss_grad(z + 10000, 0)[0] - loss) < 1e-10
for i in range(2):
    delta = np.zeros(2); delta[i] = 1e-5
    fd = (loss_grad(z+delta, 0)[0] - loss_grad(z-delta, 0)[0]) / 2e-5
    assert abs(fd - g[i]) < 1e-8
print('NLL', round(loss, 6), 'gradient', g.tolist())

The exact retained program checks the two-class example, shift invariance and finite differences. Notice its local name p denotes predicted probabilities; in this lesson's mathematical convention those are q. The code's semantics are unchanged.

7 · The softmax + cross-entropy gradient — the punchline

7.1 Derivation (do this once on paper)

Write $L=-z_y+\log\sum_j e^{z_j}$. The first term contributes $-1$ at y; the second contributes $e^{z_i}/\sum_j e^{z_j}=q_i$. Therefore $\partial L/\partial z_i=q_i-\mathbf1[i=y]$. For normalized soft targets t, the same derivation gives q−t. For unnormalized t it is $(\sum t)q-t$, which explains why validating targets matters.

The full softmax Jacobian is $J=\operatorname{diag}(q)-qq^T$. For squared probability error $\sum_i(q_i-t_i)^2$, the logit gradient is $2J(q-t)$, not the elementwise expression $2(q-t)q(1-q)$ in the old lesson. The off-diagonal terms are essential. CE produces a large output-logit correction for a confidently wrong class; it can still vanish for a correct prediction or disappear farther upstream through saturated hidden layers or cancellation.

7.2 Numeric check

The following checks the original three-class arithmetic, both possible extreme true classes, a batch mean, soft targets, the full MSE Jacobian and a wrong-Jacobian negative control. These are small deterministic fixtures, not a trained classifier.

import numpy as np
 
def entropy(p):
    p = np.asarray(p, dtype=float)
    assert np.all(p >= 0) and np.isclose(p.sum(), 1)
    return float(-(p[p > 0]*np.log(p[p > 0])).sum())
def cross_entropy(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    assert p.shape == q.shape and np.all(p >= 0) and np.all(q >= 0)
    assert np.isclose(p.sum(), 1) and np.isclose(q.sum(), 1)
    keep = p > 0
    if np.any(q[keep] == 0): return float('inf')
    return float(-(p[keep]*np.log(q[keep])).sum())
def batch_ce(z, target):
    z = np.asarray(z, float); target = np.asarray(target, float)
    assert z.ndim == 2 and z.shape == target.shape and np.isfinite(z).all()
    assert (target >= 0).all() and np.allclose(target.sum(-1), 1)
    s = z-z.max(-1, keepdims=True)
    logq = s-np.log(np.exp(s).sum(-1, keepdims=True))
    q = np.exp(logq)
    return float(-(target*logq).sum()/len(z)), (q-target)/len(z)
 
p = np.array([.7,.2,.1]); q = np.array([.5,.3,.2])
assert abs(entropy(p)-.8018185525433372) < 1e-12
assert abs(cross_entropy(p,q)-.8869413785005589) < 1e-12
assert abs(cross_entropy(p,q)-entropy(p)-.0851228259572217) < 1e-12
assert entropy([1,0,0]) == 0 and np.isinf(cross_entropy([1,0],[0,1]))
assert not np.isclose(cross_entropy(p,q)-entropy(p), cross_entropy(q,p)-entropy(q))
z = np.array([2.,1.,.1])
for y in (0,2):
    target = np.eye(3)[[y]]; value, g = batch_ce(z[None,:], target)
    for j in range(3):
        d = np.zeros(3); d[j] = 1e-5
        num = (batch_ce((z+d)[None,:],target)[0]-batch_ce((z-d)[None,:],target)[0])/2e-5
        assert abs(num-g[0,j]) < 1e-8
rng = np.random.default_rng(5)
z = rng.normal(size=(32,10)); y = rng.integers(10,size=32); t = np.eye(10)[y]
value,g = batch_ce(z,t)
for idx in np.ndindex(z.shape):
    d = np.zeros_like(z); d[idx] = 1e-5
    assert abs((batch_ce(z+d,t)[0]-batch_ce(z-d,t)[0])/2e-5-g[idx]) < 1e-8
q = np.exp(z-z.max(-1,keepdims=True)); q /= q.sum(-1,keepdims=True)
true_marginal = t.mean(0); model_marginal = q.mean(0)
assert cross_entropy(true_marginal,model_marginal) >= entropy(true_marginal)-1e-12
# Marginals alone cannot diagnose per-example quality.
assert np.allclose(q.mean(0), q[::-1].mean(0))
z0 = np.array([2.,1.,.1]); prob = np.exp(z0-z0.max()); prob /= prob.sum(); target=np.array([1.,0.,0.])
J = np.diag(prob)-np.outer(prob,prob)
good = 2*J@(prob-target); bad = 2*(prob-target)*prob*(1-prob)
def mse(z):
    q = np.exp(z-z.max()); q /= q.sum()
    return float(((q-target)**2).sum())
num = np.array([(mse(z0+np.eye(3)[j]*1e-5)-mse(z0-np.eye(3)[j]*1e-5))/2e-5 for j in range(3)])
np.testing.assert_allclose(good,num,atol=1e-9); assert not np.allclose(bad,num)
assert abs(good.sum()) < 1e-12
# Confidently wrong but finite logit-space loss.
assert abs(batch_ce(np.array([[1000.,-1000.]]),np.array([[0.,1.]]))[0]-2000) < 1e-12
print('entropy, KL, all-coordinate CE and full softmax-MSE Jacobian passed')

8 · Where this all shows up in the series

Digit classification uses a categorical output; language modeling uses conditional vocabulary predictions; supervised instruction tuning can mask prompt positions; preference objectives compare response probabilities or rewards. These share probability tools but not necessarily the same reduction, target distribution or optimization problem. DL007 specializes to a Bernoulli output and DL006 updates parameters.

8b · Cross-entropy in language modeling and preference learning

8b.1 · Next-token prediction and perplexity

An autoregressive model factorizes a sequence probability into conditional factors. For [A,B,C,D], inputs [A,B,C] predict [B,C,D]. A causal mask prevents future-target access; shifting alone is not a causal-attention implementation. Flattening (B,T,V) logits to class rows is legitimate only after aligning and masking targets.

Perplexity is $e^\ell$ for mean NLL in nats. Loss1.8 gives6.04965;1.9 gives6.68589. It is the geometric mean inverse probability assigned to the observed tokens, not a claim the model literally considers six equally likely options. Compare only with matched tokenizer, text split, context and masking. The old private-model loss estimates have no reliable primary evidence and are not benchmarks.

For batches containing2 and8 valid tokens with mean losses1 and3, aggregate to $(2+24)/10=2.6$, not2. Averaging perplexities is also wrong. Ignore padding in numerator and denominator; reject an empty supervised set.

8b.2 · DeepSeek-V3's multi-token prediction

The V3 report defines sequential multi-token-prediction modules that use a previous-depth representation and the embedding of the next token, with shared embedding/output components and their own Transformer block. This is more specific than “add one cheap head.” Its depth-one auxiliary objective predicts the second future token; the general construction can extend to D depths, averaging their CE terms with a coefficient. Auxiliary modules can be discarded for ordinary inference or used in speculative decoding. This is a description of the reported architecture, not a reproduced quality gain or a rule for every2025 model.

8b.3 · KL, DPO, KTO, ORPO, SimPO — distinct objectives

For preferred and rejected responses, let

Sequence log-probabilities sum conditional token log-probabilities. The derivative with respect to u is $\sigma(u)-1$. At u=0, it is−0.5: preferred log-probability receives coefficient−β/2, rejected+β/2 before differentiating those sequence terms. DPO's derivation uses a specified KL-regularized preference formulation; it is not an algebraic replacement for every online RL problem.

KTO uses desirable/undesirable labels without requiring paired examples and a prospect-theoretic value construction; it is not ordinary per-example BCE. ORPO combines a supervised likelihood term with an odds-ratio preference penalty without a reference model. SimPO uses length-normalized average response log-probabilities and a target margin, also reference-free. Only DPO above uses the displayed reference ratios. These methods differ in supervision, normalization and reference costs; saying all are “CE on ratios” erases their main distinctions.

8b.4 · Load-balancing loss in MoE

Switch Transformers' top-one balancing objective is $\alpha E\sum_i f_iP_i$, where E is expert count, f is observed routing fraction and P mean router probability. Uniform f and P yield α; routing and probability both collapsed to one expert yield αE. This is finite, not an “explosion,” and the hard routing fractions are not generally differentiated as smooth probabilities.

DeepSeek-V3 mainly adjusts routing with auxiliary-loss-free bias balancing and retains a small sequence-wise complementary loss with its own definitions and normalization. Do not conflate these with Switch or categorical CE. The purpose is load control, which can compete with task loss, not measuring target-label surprise.

8b.5 · Label smoothing and calibration choices

For K classes and ε=.1, smoothing toward uniform gives $t_y=.9+.1/K$ and other entries .1/K. At K=3 these are .93333,.03333,.03333. The gradient is q−t. A fixed soft target has a finite optimum logit gap of $\ln(t_y/t_j)=\ln28$ in this example; the common logit offset is unconstrained. It does not impose a hard confidence cap on every prediction during training or guarantee a generalization gain.

Hard-label CE fits empirical conditional likelihood; smoothing changes the targets and can alter calibration and distillation behaviour. Positive temperature scaling divides logits by T, preserving argmax in exact arithmetic while fitting confidence on a held-out calibration set. It cannot repair every class-specific error or distribution shift. Compare NLL, reliability curves, task costs and calibration metrics; a binned ECE alone is not a proof and loss is not irrelevant.

import numpy as np
import torch
import torch.nn.functional as F
 
# Shift once, mask once, and compare exact CPU framework reduction.
logits = torch.arange(2*4*5,dtype=torch.float64).reshape(2,4,5)/10
ids = torch.tensor([[0,1,2,3],[2,1,0,4]])
target = ids[:,1:].clone(); target[1,-1] = -100
selected = logits[:,:-1].reshape(-1,5)
losses = F.cross_entropy(selected,target.reshape(-1),reduction='none',ignore_index=-100)
mask = target.reshape(-1) != -100
joint = losses[mask].sum()/mask.sum()
assert torch.isfinite(joint) and mask.sum() == 5
# Soft floating targets are supported; integer index targets have different shape.
z = torch.tensor([[2.,1.,.1]],dtype=torch.float64,requires_grad=True)
t = torch.tensor([[.9333333333333333,.0333333333333333,.0333333333333333]],dtype=torch.float64)
l = F.cross_entropy(z,t); l.backward()
torch.testing.assert_close(z.grad,torch.softmax(z.detach(),-1)-t)
assert not torch.allclose(F.cross_entropy(z,t), F.cross_entropy(torch.softmax(z,-1),t))
u = torch.tensor(0.,dtype=torch.float64,requires_grad=True)
(-F.logsigmoid(u)).backward(); assert u.grad == -.5
assert np.isclose(np.exp(1.9)/np.exp(1.8),np.exp(.1))
assert np.isclose((2*1+8*3)/10,2.6)
E=4; alpha=.01
assert np.isclose(alpha*E*np.dot(np.ones(E)/E,np.ones(E)/E),alpha)
assert np.isclose(alpha*E*np.dot([1,0,0,0],[1,0,0,0]),alpha*E)
print('CPU label modes, causal target shift, masking, DPO scalar and balancing arithmetic passed')

9 · Failure cases worth testing

PyTorch2.9 CE accepts either integer class indices or same-shaped floating probability targets; it does not silently interpret a valid floating one-hot matrix as indices. It does not rigorously validate every probability constraint for soft targets, so check normalization and nonnegativity yourself. Do not automatically argmax soft labels: that discards their information.

Passing softmax(logits) into a logits API changes the objective. A value around1e−40 has a finite natural logarithm if represented; it is not already zero, and a large finite loss does not automatically poison all future training. The real issue is underflow to zero, invalid arithmetic and the backward calculation. Clipping cannot repair an invalid forward graph. A model predicting one class may reflect imbalance, labels, masking, initialization or optimization; inspect those hypotheses rather than repeating an invented incident.

11 · Retention scaffold

1. Why negative log? Independence gives additive surprise: $-\ln(.7\times.1)=-\ln.7-\ln.1$. Units change by a constant when changing the base.
2. Uniform entropy and its maximum? Each of K events contributes $(1/K)\ln K$, summing to ln K. Nonnegative KL to uniform proves every other distribution has no larger entropy.
3. Overflow versus log zero? Subtracting a finite maximum prevents exponential overflow; computing log-probabilities directly avoids logging underflowed probabilities. All-masked rows and incoming NaNs still need explicit handling.
4. Loss and gradient at one label? $-\log q_y$, with logit gradient q−onehot. For z=(ln3,0),y=0 these are0.287682 and(−.25,.25); the gradient sums to0 because common shifts do nothing.
5. When not ordinary categorical CE? Real-valued regression may use Gaussian NLL, squared error or Huber under an appropriate model. Ordinal, censored or structured outcomes need their ordering/observation assumptions modeled; classification is not the only task.
6. Perplexity change1.9→1.8? 6.68589→6.04965, a ratio $e^{-.1}\approx.90484$. That is about9.52% lower perplexity under the same evaluation protocol. There is no universal basis to say it matters more than a percentage point on an unrelated accuracy benchmark.

Recall — from memory

Recover surprise, entropy, cross-entropy, KL nonnegativity and the one-hot gradient. Explain the support condition and why q=0 on a p=0 outcome is not itself an infinite-cost event. Derive the soft-target version before using label smoothing.

Stretch — for one extra hour

The batch fixture checks all320 logits for32 examples and10 classes, dividing exactly once by32. Rebuild it, then deliberately use a sum loss with a mean gradient to make the checker fail. Compare true and predicted class marginals, but also permute predictions among examples: the marginals stay unchanged while per-example quality can collapse. Marginal KL is a useful diagnostic, not a complete classifier audit.

In your own words

Explain cross-entropy as a coding bill under the model, and maximum likelihood as making the observed labels more probable under the specified conditional model.

Spaced-review pointer

Tomorrow derive q−t again; next week reconstruct the support-aware KL proof and the full softmax-MSE Jacobian. Revisit DPO only after being able to name its sequence log-probabilities and reference model.

Next-session teaser

DL006 turns a checked gradient into an optimizer update and separates numerical safeguards from convergence claims.

What to bring back tomorrow — sticky note

Logits into a logits loss; validate targets; count valid tokens; fit calibration on held-out data. Perplexity is exp of the aggregate mean NLL, not the average of batch perplexities.

Primary references and boundary

Mathematics for Machine Learning supplies probability and information definitions. PyTorch2.9 CE, DPOv3, KTO, ORPO, SimPO, DeepSeek-V3v2, Switch Transformers, Guo et al.2017 and label-smoothing analysis bound the modern comparisons. CPU fixtures test arithmetic and APIs, not preference-tuned models, MoE training, calibration gains or proprietary-model losses.

The SciPy1.15.3 entropy reference explicitly states the Shannon/relative-entropy and cross-entropy identities; the lesson implements its own support policy rather than relying on automatic renormalization.