R23 · Week 23 Recall & Drill

Assessment scope

A transformer design has separate choices for normalization, visibility, objectives, decoding and execution kernels. A mask alone does not determine retrieval quality, a token/parameter ratio is not a universal optimum, and exact attention does not require materializing its full score matrix. This revision supplies complete worked examples for each distinction without claiming a trained-model or accelerator benchmark.

Sessions covered

• S111 — Full Transformer Architecture — Encoder + Decoder
• S112 — Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each
• S113 — LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature
• S114 — Scaling Laws — Chinchilla, Compute-Optimal Training
• S115 — Efficient Attention — Flash, Sparse, Linear

90-min structure

The pacing table is not an estimate of model-training or GPU runtime.

Blank-page reconstruction · 30 min

S111 · Full Transformer

1. List a block's components in order with an explicit normalization convention.
2. What does the feed-forward branch add?
3. Explain residual normalization placement and weight tying.

Solutions. In the original post-norm encoder, input token embeddings receive position information; each block performs self-attention, dropout/residual addition followed by LayerNorm, then a position-wise feed-forward branch followed by another residual/LayerNorm. The original decoder adds a masked self-attention sublayer and an encoder cross-attention sublayer before its feed-forward sublayer, each with its own residual/normalization. A pre-norm branch instead takes the form x <- x + Attention(LN(x)), then x <- x + FFN(LN(x)), with a possible final normalization. State which architecture is being discussed rather than treating the order as universal.

Attention mixes positions and is linear in V holding its routing weights fixed; Q/K-dependent softmax makes it generally nonlinear in input. FFN applies a learned nonlinear channel transformation separately at each position, for example W2*phi(W1*x+b1)+b2 in column-vector notation. It can expand/contract channels and compute token-local features. Removing it changes capacity, but does not reduce the entire transformer to a globally linear averaging machine.

For y=x+F(LN(x)), the Jacobian contains an identity term. For y=LN(x+F(x)), the normalization Jacobian multiplies the sum. Xiong et al. analyze initialization and show different warmup behavior in their experiments. This supports an optimization distinction, not “post-norm cannot train” or “pre-norm always trains.” Initialization, depth, scaling and schedules also matter; the original trained transformer itself used post-norm.

If input embedding E has shape (V,D), an output projection can reuse E through hidden @ E.T, optionally with a separate bias. Tying saves V*D weights compared with two independent matrices, subject to compatible vocabulary/hidden width and architecture. For V=32,000 and D=512 that is 16,384,000 weights; tying constrains the model as well as saving memory, and not every release uses it.

S112 · Model Families

1. Draw encoder, decoder and cross-attention visibility.
2. Relate BERT, GPT and T5 objectives to their use cases.
3. Correct “encoders cannot generate” and “decoders are always worse embeddings.”

Solutions. An encoder's ordinary self-attention sees all nonpadding input positions. A causal decoder sees its current/past prefix, not future target tokens. An encoder-decoder combines a bidirectional source encoder, causal target self-attention and cross-attention from each target query to permitted source keys. Padding and task-specific restrictions qualify all three. Shift targets by one so the visible current token predicts the next token rather than revealing its own answer.

BERT's bidirectional masked-language objective suits contextual representations and supervised understanding; a standard BERT checkpoint is not directly an ordinary left-to-right next-token generator. That does not prove a bidirectional architecture can never generate text: masked/iterative generation, added decoders and alternative training procedures exist. GPT-style causal modeling directly parameterizes the autoregressive factorization of sequence probability. T5 uses an encoder-decoder text-to-text setup, including denoising pretraining and conditional generation tasks such as translation/summarization. Masks are one part of these designs; objectives, data and heads also matter.

Full context at every encoder token is useful for representations, but a causal model's final nonpadding token can already see the full prefix. Pooling, contrastive fine-tuning, training data and the retrieval task can dominate architecture labels. Compare retrieval recall/ranking quality, throughput and latency on the intended corpus; a visibility grid cannot establish a universal equal-size or per-compute embedding ranking.

S113 · Sampling

1. Define greedy, beam, temperature, top-k and top-p.
2. Explain repetition risks and filter order.
3. Give structured-output and creative-generation policies.

Solutions. Greedy picks a largest next-token logit with a declared tie rule; it is not a global maximum-probability sequence search. Beam search keeps several high-scoring partial sequences, accumulating log probabilities and often applying length normalization; it approximates search under a chosen score and does not certify truth or diversity. Both can repeat or produce bland text depending on model/context/objective; repetition is not inevitable under greedy decoding.

For temperature T>0, compute softmax(logits/T): lower T sharpens, higher T flattens while preserving rank. T=0 is not division by zero; APIs may give it a special greedy meaning, which must be checked. Top-k retains a specified number of ranked candidates (ties need a convention). Top-p retains the smallest highest-probability prefix whose cumulative mass reaches p, then renormalizes. It adapts candidate count to concentration, but is not always the best application policy.

If combining filters, define the order and normalization. Here it is temperature → exact-k with index tie-break → renormalize → nucleus → renormalize. Applying top-p to an unnormalized top-k remainder can accidentally retain all k because the remaining mass never reaches p. Rather than “prefer whichever happens first,” inspect the serving implementation and evaluate configurations while holding other factors fixed.

For structured output, start with a grammar/schema-constrained decoder where supported, a bounded token budget, explicit stop/validation rules and a low-variance/greedy policy; valid JSON still needs semantic validation and authorization. For creative drafting, a tested positive temperature and nucleus threshold (for example T=0.8, p=0.9 as starting values) can permit variety, with content/task evaluation. Those values are illustrative, not provider-independent settings or guarantees. Record model/version, seed when available and actual filter order for reproducibility.

S114 · Scaling Laws

1. State the dense training-compute approximation and its limits.
2. Solve a hypothetical fixed-compute allocation with a token/parameter heuristic.
3. Explain inference-aware overtraining without calling the heuristic a law.

Solutions. A commonly used dense-transformer training approximation is C ≈ 6*N*D, where N is parameter count and D training-token count, omitting some attention, embedding, optimizer and hardware effects. It is not an exact wall-clock or monetary-cost formula and needs adjustment for different architectures, sequence lengths and sparsity.

The Chinchilla study fit empirical loss/compute relationships and found model size and training tokens should scale approximately together in its studied regime. A roughly twenty-training-tokens-per-parameter reference reflects its 70B/1.4T example and associated heuristic, not a universal constant. If we assume D=rN with fixed r and C=6ND, algebra gives N=sqrt(C/(6r)), D=rN; increasing C by 100 multiplies each by 10 under this assumption. Loss fits, tokenizer/data quality, repeated tokens and context regime may change the preferred ratio. Ratios alone cannot diagnose a release as wrongly trained.

The training-compute optimum concerns predictive loss at a training budget. Lifetime cost includes repeated inference, latency and memory. A smaller model trained on more tokens may be economical at high serving volume if quality meets the requirement. That is a break-even calculation, not a blanket assertion that all smaller overtrained models are cheaper or that all extra data is useful. Licensing, data quality, latency constraints and hardware utilization also enter the decision.

S115 · Efficient Attention

1. Separate dense attention arithmetic, storage and IO bottlenecks.
2. Explain exact tiled softmax and backward recomputation.
3. Compare sparse and linear alternatives with workloads.

Solutions. For T positions and head width d, dense score and value products use order T²*d arithmetic. Naively storing scores/probabilities costs order T² elements per head/batch, whereas Q/K/V/output cost order T*d. Runtime can be memory/IO-bound or compute-bound depending on shapes and hardware; “the bottleneck is never arithmetic” is false.

FlashAttention is an IO-aware implementation of exact dense attention (within floating-point reassociation differences), not a dropped-pair approximation. It tiles Q/K/V into faster memory, maintains running softmax maxima/normalizers and output accumulators, and avoids materializing the full attention matrix in HBM. Its backward pass can recompute intermediates rather than save that matrix. Reordering/recomputation means executed arithmetic need not be identical in count; the mathematical operator is the same. A CPU streaming implementation below verifies the algebra, not FlashAttention's GPU kernel or speedups.

Sparse attention restricts an attention graph, e.g. a local window plus selected global tokens for long documents; it can miss interactions outside that pattern. It may be exact for its specified sparse operator but differs from dense full attention. Linear-attention methods often replace softmax with a feature-map/kernel structure so associativity reduces sequence scaling; some approximate a softmax kernel, others define a different operator. They may suit streaming/long sequences but trade representational behavior and numerical issues. “All efficient methods are approximate” and “all sparse/linear methods have the same accuracy cost” are both wrong.

Hands-on drill · 30 min

The programs run CPU-only. They print all three visibility maps, normalized decoding distributions, hypothetical compute allocations and an actual exact-attention comparison. No GPU kernels, LLM outputs or cloud service measurements are produced.

Step 1 — the mask is the family (7 min)

The retained heading is a shorthand, not an architecture theorem: visibility is only one family distinction; the objective, position scheme, layers and output head also matter.

import numpy as np
T,S=4,6
full=np.ones((S,S),dtype=bool)
causal=np.tril(np.ones((T,T),dtype=bool))
cross=np.ones((T,S),dtype=bool)
source_valid=np.array([True,True,True,True,False,False])
cross_padded=cross&source_valid[None,:]
def show(name,matrix):
    print(name)
    for row in matrix:print(' '.join('#' if x else '.' for x in row))
for label,matrix in [('encoder',full),('decoder causal',causal),('cross with padding',cross_padded)]:show(label,matrix)
assert full.sum()==36 and causal.sum()==10 and cross_padded.sum()==16
assert all(causal[i,j]==(j<=i) for i in range(T) for j in range(T))
# Teacher-forced inputs/targets: each visible input prefix excludes its own next-token target position.
tokens=['BOS','a','b','c','EOS']
inputs,targets=tokens[:-1],tokens[1:]
for i in range(T):print('visible input',inputs[:i+1],'predict',targets[i])
print('embedding weights saved by tying',32000*512)

The cross-attention map is rectangular: four target queries by six source keys, with two padding keys excluded. The target sequence needs its own padding/loss mask in variable-length batches. The masks demonstrate allowed information flow; they are not proof of better embeddings, fluent generation or task accuracy.

Step 2 — decoding knobs on a real distribution (8 min)

# A concrete synthetic distribution, not logits captured from a model.
def softmax(z):
    z=np.asarray(z,float);e=np.exp(z-z.max());return e/e.sum()
def distribution(logits,temperature=1.,top_k=None,top_p=None):
    logits=np.asarray(logits,float)
    if logits.ndim!=1 or not len(logits) or not np.isfinite(logits).all():raise ValueError('finite vector required')
    if temperature<=0:raise ValueError('use separate greedy policy for zero temperature')
    if top_k is not None and (not isinstance(top_k,int) or not 1<=top_k<=len(logits)):raise ValueError('invalid k')
    if top_p is not None and not 0<top_p<=1:raise ValueError('invalid p')
    p=softmax(logits/temperature)
    order=np.lexsort((np.arange(len(p)),-p))
    if top_k is not None:
        keep=order[:top_k];mask=np.zeros(len(p),bool);mask[keep]=True
        p=np.where(mask,p,0);p=p/p.sum()
    if top_p is not None:
        order=np.lexsort((np.arange(len(p)),-p))
        size=min(len(p),int(np.searchsorted(np.cumsum(p[order]),top_p,side='left'))+1)
        keep=order[:size];mask=np.zeros(len(p),bool);mask[keep]=True
        p=np.where(mask,p,0);p=p/p.sum()
    return p
logits=np.log([.4,.3,.2,.1])
for label,kw in [('baseline',{}),('cold',{'temperature':.5}),('hot',{'temperature':2.}),
                 ('k2',{'top_k':2}),('p.6',{'top_p':.6}),('k2 then p.6',{'top_k':2,'top_p':.6})]:
    p=distribution(logits,**kw);assert np.isclose(p.sum(),1)
    entropy=-sum(v*np.log(v) for v in p if v>0)
    print(label,p,'entropy',entropy)
assert np.allclose(distribution(logits,top_k=2),[4/7,3/7,0,0])
assert np.allclose(distribution(logits,top_k=2,top_p=.55),[1,0,0,0])
assert np.allclose(distribution([0,0,0],top_k=2),[.5,.5,0])
assert np.allclose(distribution(logits,top_p=1),[.4,.3,.2,.1])
# Greedy local choice need not maximize a two-step sequence.
paths={'AA':.6*.51,'AB':.6*.49,'BA':.4*.99,'BB':.4*.01}
assert max(paths,key=paths.get)=='BA' and paths['BA']>paths['AA']
print('greedy AA',paths['AA'],'best joint BA',paths['BA'])

At k=2 the probabilities become 4/7 and 3/7 before nucleus filtering. With p=0.55 only the first remains; applying p to the unnormalized 0.4/0.3 would incorrectly keep both. Equal-logit ties retain the lowest token indices here; a different library may include all boundary ties. The greedy example explains why beam search can improve a sequence likelihood objective without making that objective identical to human quality.

Step 3 — a compute budget (7 min)

from math import sqrt,isclose
def training_compute(parameters,tokens):return 6*parameters*tokens
def heuristic_allocation(compute,ratio=20):
    if compute<=0 or ratio<=0:raise ValueError('positive budget/ratio required')
    parameters=sqrt(compute/(6*ratio));return parameters,ratio*parameters
for budget in [1e21,1e23,1e24,1e25]:
    n,d=heuristic_allocation(budget)
    assert isclose(training_compute(n,d),budget,rel_tol=1e-12)
    print('assumed budget',budget,'parameters',n,'tokens',d,'ratio',d/n)
n1,d1=heuristic_allocation(1e21);n2,d2=heuristic_allocation(1e23)
assert isclose(n2/n1,10) and isclose(d2/d1,10)
for n,d in [(175e9,300e9),(70e9,1.4e12),(8e9,15e12)]:print('illustrative N,D,ratio',n,d,d/n)
# Hypothetical monetary costs, not provider pricing or measured quality parity.
large_train,small_train=100.,160.
large_per_million,small_per_million=4.,1.
break_even=(small_train-large_train)/(large_per_million-small_per_million)
assert break_even==20
for millions in [10,20,100]:
    print('million output units',millions,'large lifetime',large_train+4*millions,'small lifetime',small_train+millions)

The break-even is 20 million units given equal acceptable task quality and the invented costs. At 10 million, the extra training spend has not paid back; at 100 million it has. The model/data ratio examples are descriptive arithmetic, not verdicts that all ratios below 20 are mistakes or all ratios above it are economically optimal.

Step 4 — the bottleneck is traffic, not arithmetic (8 min)

The retained heading names the IO bottleneck targeted by tiling, not every attention workload: the dense quadratic arithmetic remains, and shapes/hardware can make it dominant.

rng=np.random.default_rng(17)
Q=rng.normal(size=(7,4));K=rng.normal(size=(9,4));V=rng.normal(size=(9,3))
def dense_attention(Q,K,V):
    scores=Q@K.T/np.sqrt(Q.shape[1]);scores-=scores.max(1,keepdims=True)
    weights=np.exp(scores);weights/=weights.sum(1,keepdims=True)
    return weights@V
def streamed_attention(Q,K,V,block):
    if block<1 or len(K)==0:raise ValueError('nonempty keys and positive block required')
    rows=[]
    for q in Q:
        maximum=-np.inf;normalizer=0.;numerator=np.zeros(V.shape[1])
        for start in range(0,len(K),block):
            scores=K[start:start+block]@q/np.sqrt(len(q))
            new_max=max(maximum,float(scores.max()))
            old_scale=np.exp(maximum-new_max)
            weights=np.exp(scores-new_max)
            numerator=old_scale*numerator+weights@V[start:start+block]
            normalizer=old_scale*normalizer+weights.sum()
            maximum=new_max
        rows.append(numerator/normalizer)
    return np.array(rows)
reference=dense_attention(Q,K,V)
for block in [1,2,4,9,12]:
    result=streamed_attention(Q,K,V,block)
    assert np.allclose(result,reference,rtol=1e-12,atol=1e-12)
    print('key block',block,'max exact-operator delta',float(abs(result-reference).max()))
# Storage/minimum-transfer accounting, not total kernel traffic.
for length in [512,2048,8192,32768]:
    d=128;element_bytes=2
    score_bytes=length*length*element_bytes
    qkv_output_bytes=4*length*d*element_bytes
    avoided_score_write_read=2*score_bytes
    print('length',length,'one score matrix bytes',score_bytes,'QKV+output bytes',qkv_output_bytes,
          'one avoided score write+read bytes',avoided_score_write_read)
assert 32768**2*2==2147483648

The running state represents a maximum m, sum sum exp(score-m) and value-weighted numerator. When a tile raises m, rescale both old accumulators by exp(old_m-new_m) before adding the new tile. This invariant gives the same final normalized weighted sum as dense attention; different floating-point association explains small deltas. It is not a performant FlashAttention kernel: this Python loop repeatedly reads tiles and runs on CPU.

Worked accumulator retest. Take one query whose two key scores are 0 and log(3), with scalar values 2 and 10, one key per tile. After tile one, (m, normalizer, numerator)=(0,1,2). Tile two raises m to log(3), so the old state is multiplied by 1/3: the new normalizer is 1/3+1=4/3 and numerator is 2/3+10=32/3. The output is 8, matching the dense weighted mean (1*2+3*10)/4. Rescaling only the denominator would give 9; rescaling only the numerator would give 16/3. Both accumulators are essential. The displayed helpers assume compatible, nonempty finite arrays and finite positive temperature/budgets; they are teaching calculations, not complete validators for arbitrary inputs or extreme overflow.

A single 32768×32768 FP16 score matrix is 2,147,483,648 bytes (2 GiB), not automatically larger than every device. Batch/head/layer storage and saved intermediates multiply pressure. The table counts a removed materialization term and minimal QKV/output storage, not total tiled IO or a speedup ratio; repeated tile reads, SRAM size, recomputation and hardware scheduling are deliberately not modeled. Dense attention's quadratic arithmetic remains.

Answer constraints and corrections

Misconception: “FlashAttention is faster because it drops attention pairs.” It reorganizes the exact dense operator with tiling and online normalization, saving HBM materialization and recomputing intermediates where useful. Sparse/linear designs change the operator or allowed interactions in different ways; evaluate their task quality and execution costs separately.

Quiz answers: (1) FFN adds position-wise nonlinear channel computation; attention is only linear in values when scores are fixed. (2) Pre/post-norm alter Jacobian paths and initialization behavior, not a universal trainability binary. (3) Bidirectionality helps expose full context at each token, but retrieval quality also depends on objective, pooling and training; causal final-token embeddings can use the whole prefix. (4) Top-k fixes candidate count under a tie policy, top-p fixes retained mass under an ordering; normalize between combined filters and evaluate rather than declaring one universally best. (5) A fixed-compute allocation jointly chooses N and D using empirical loss behavior. The illustrative D≈20N and C≈6ND equations give a useful arithmetic exercise, not a constant across architectures/data. Add inference volume, latency, memory and quality constraints before choosing a smaller more-trained model, and refit/validate the empirical relationships when the regime changes.

Gap analysis + next week preview · 10 min

Retest top-k followed by top-p at a tied boundary, a rectangular cross-attention mask, or a streaming-softmax tile with a much larger new maximum. Explain which invariant protects the result. Next week's prompting, chunking, hybrid retrieval, vector indexing and agents use these models as components; source retrieval and tool permissions introduce additional boundaries beyond fluent generation.

Corrective source notes

Primary snapshots checked 2026-10-07: Attention Is All You Need v7, BERT, T5, Xiong et al. normalization placement, Transformers 4.57.1 generation controls, Chinchilla, and FlashAttention. The exact tiled operator is CPU-verified here; GPU IO, training curves, embedding quality, generated-text quality and monetary examples are not measured service results.

The compute approximation and 70B/1.4T example are checked against Chinchilla v1 full text; tiling/normalizer/recomputation claims against FlashAttention v2 full text.

Part of the 6-month evergreen learning plan.