Transformer Blocks — Full Architectures and an Analytic Decoder
Attention mixes information across token positions. A position-wise feed-forward network transforms features with the **same weights at each position**. Normalization sets a branch's input or output scale, and residual addition carries an explicit sk
Original publication date unrecordedUpdated
Lesson preparation & details
Level: advanced
By the end, you should be able to
- Wire encoder-decoder and causal blocks with explicit normalization
- Execute an exact analytic NumPy decoder and independent derivative checks
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 9 sections
The block is more than attention
Attention mixes information across token positions. A position-wise feed-forward network transforms features with the same weights at each position. Normalization sets a branch's input or output scale, and residual addition carries an explicit skip path. Position information and masks determine what ordered information may be mixed. Stacking these operations is a Transformer, not simply repeating weighted averages.
This lesson restores both the original encoder-decoder and GPT-style decoder-only architectures while retaining the complete, exact NumPy teaching decoder and analytic backward pass. That small decoder deliberately has one head, one block, ReLU width 2d, untied output weights, no linear biases, no dropout and no final norm. It is an inspectable teaching variant, not GPT-2 or Llama. The PyTorch examples use a different, explicitly declared architecture; agreement is never assumed across these differences.
Prerequisites: MHA, positions, MLP, normalization and chain-rule/residual differentiation. Outcomes: wire encoder/decoder blocks, derive pre/post-norm gradients, enforce causality, count every parameter, understand RMSNorm/SwiGLU/GQA variants and compare complete local conditional-generation recipes.
Pre-norm, post-norm and the residual derivative
The 2017 Transformer uses post-norm: LN(x+F(x)). A common GPT-style block uses pre-norm: x+F(LN(x)), repeated for attention then FFN, followed by a final norm before the vocabulary head. The original FFN used ReLU; GELU below is a declared variant. LayerNorm subtracts feature mean and divides by square root of feature variance plus epsilon, then applies learned affine parameters. It does not set “unit mean,” and learned affine parameters/epsilon mean output variance is not exactly one.
Pre-norm exposes an identity term through the residual stream, not a guarantee that the total gradient cannot cancel, explode or be transformed by the final norm. Xiong et al. analyze large expected parameter gradients near the output of post-LN at initialization; the old claim about necessarily larger input gradients reversed their finding. Warmup and stability depend on initialization, depth, optimizer and data. Pre-norm can work without warmup in studied settings; “every deep model needs none” is not justified. DeepNorm supplies a counterexample to claims that post-norm cannot scale deeply.
Do not use sum(LayerNorm(x)) as the only gradient-flow probe: at unit gain/zero bias its sum is zero up to numerical error, so a zero gradient can be an artifact of the chosen loss. Use a nonconstant weighted probe and compare matched branches. Zeroing both entire branch functions, including biases, makes a pre-norm block the identity. Zeroing only FFN weights leaves attention; zeroing normalization gains need not remove branch biases.
import math, copy
import torch
from torch import nn
from torch.nn import functional as F
from mha_lab import MultiHeadAttention
from positions_lab import LearnedPE
class TransformerBlock(nn.Module):
def __init__(self,width=16,heads=4,ff_width=64,pre=True,dropout=0.):
super().__init__();self.pre=pre
self.attn=MultiHeadAttention(width,heads,dropout)
self.ffn=nn.Sequential(nn.Linear(width,ff_width),nn.GELU(),nn.Linear(ff_width,width),nn.Dropout(dropout))
self.ln1=nn.LayerNorm(width);self.ln2=nn.LayerNorm(width)
def forward(self,x,allowed=None):
if self.pre:
x=x+self.attn(self.ln1(x),allowed)
return x+self.ffn(self.ln2(x))
x=self.ln1(x+self.attn(x,allowed))
return self.ln2(x+self.ffn(x))
torch.manual_seed(39)
x=torch.randn(2,5,16);mask=torch.ones(5,5,dtype=torch.bool).tril()
pre=TransformerBlock();post=TransformerBlock(pre=False)
post.load_state_dict(pre.state_dict())
assert pre(x,mask).shape==x.shape==post(x,mask).shape
for block in (pre,post):
with torch.no_grad():
for p in block.attn.parameters():p.zero_()
for p in block.ffn.parameters():p.zero_()
assert torch.equal(pre(x,mask),x)
assert not torch.allclose(post(x,mask),x)
for depth in (2,6,12):
for pre_norm in (True,False):
torch.manual_seed(9)
blocks=nn.ModuleList([TransformerBlock(pre=pre_norm) for _ in range(depth)])
inp=x.clone().requires_grad_();z=inp
for b in blocks:z=b(z,mask)
probe=torch.linspace(-1,1,16)
(z*probe).sum().backward()
assert torch.isfinite(inp.grad).all()
print('matched initialization gradient diagnostic',depth,pre_norm,inp.grad.norm().item())Encoder, decoder and cross-attention
An encoder block uses bidirectional source self-attention and an FFN. An encoder-decoder decoder has three learned branches: causal target self-attention, cross-attention and FFN, each wrapped in normalization/residual operations. Cross-attention queries come from the target stream; keys/values come from encoder output. Its scores have shape (B,h,T_target,T_source). Source padding must be excluded; a source causal triangle is not normally wanted for an offline translation encoder.
Figure text equivalent — labels and relationships
- Node “Source plus positions” (square).
- Node “Encoder blocks: source self-attention and FFN” (square).
- Node “BOS plus previous targets and positions” (square).
- Node “Decoder: causal self-attention” (square).
- Node “Cross-attention: Q target, K/V source” (square).
- Node “Position-wise FFN” (square).
- Node “Norm and vocabulary logits” (square).
- “Source plus positions” leads to “Encoder blocks: source self-attention and FFN”. Line: normal; ends: arrow point.
- “BOS plus previous targets and positions” leads to “Decoder: causal self-attention”. Line: normal; ends: arrow point.
- “Encoder blocks: source self-attention and FFN” leads to “Cross-attention: Q target, K/V source”. Line: normal; ends: arrow point.
- “Decoder: causal self-attention” leads to “Cross-attention: Q target, K/V source”. Line: normal; ends: arrow point.
- “Cross-attention: Q target, K/V source” leads to “Position-wise FFN”. Line: normal; ends: arrow point.
- “Position-wise FFN” leads to “Norm and vocabulary logits”. Line: normal; ends: arrow point.
The “meeting” analogy is useful: discuss across positions, then transform each participant's features separately. Residual writes may cancel or replace information algebraically; “nobody ever overwrites” is not a memory-preservation theorem. Self-attention is itself nonlinear through input-dependent softmax; omitting an FFN does not make a whole attention stack collapse into one linear map.
The next complete local encoder-decoder uses PyTorch's built-in Transformer with explicit pre-norm, padding masks and target shift. Save it with the previous code as blocks_lab.py. The public MHA module uses True=allowed, whereas nn.Transformer uses True=blocked. This example deliberately uses the latter convention.
from seq2seq_lab import PAD,BOS,EOS,make_batch,make_vocab,reverse_pairs,token_nll
class TranslationTransformer(nn.Module):
def __init__(self,vocab,width=16,heads=4,layers=2,max_len=64):
super().__init__();self.embed=nn.Embedding(vocab,width,padding_idx=PAD)
self.position=LearnedPE(max_len,width)
self.core=nn.Transformer(d_model=width,nhead=heads,num_encoder_layers=layers,
num_decoder_layers=layers,dim_feedforward=4*width,dropout=0.,batch_first=True,norm_first=True)
self.head=nn.Linear(width,vocab)
def forward(self,src,previous):
blocked=torch.ones(previous.size(1),previous.size(1),dtype=torch.bool,device=src.device).triu(1)
s=self.position(self.embed(src));t=self.position(self.embed(previous))
z=self.core(s,t,tgt_mask=blocked,src_key_padding_mask=src.eq(PAD),
tgt_key_padding_mask=previous.eq(PAD),memory_key_padding_mask=src.eq(PAD))
return self.head(z)
@torch.no_grad()
def translation_greedy(model,src,max_new=20):
mode=model.training;model.eval()
try:
ids=torch.full((src.size(0),1),BOS,dtype=torch.long);done=torch.zeros(src.size(0),dtype=torch.bool)
for _ in range(max_new):
logits=model(src,ids)[:,-1];logits[:,[PAD,BOS]]=-torch.inf
nxt=logits.argmax(-1);nxt=torch.where(done,torch.full_like(nxt,EOS),nxt)
ids=torch.cat([ids,nxt[:,None]],1);done|=nxt.eq(EOS)
if done.all():break
return ids,done
finally:model.train(mode)
def train_translation(train,val,stoi,layers=2,epochs=2):
if set(train)&set(val):raise ValueError('split overlap')
model=TranslationTransformer(len(stoi),layers=layers);opt=torch.optim.AdamW(model.parameters(),lr=.003)
best=math.inf;snapshot=None;history=[]
for epoch in range(epochs):
model.train()
for start in range(0,len(train),8):
x,_,y=make_batch(train[start:start+8],stoi);opt.zero_grad(set_to_none=True)
loss,_=token_nll(model(x,y[:,:-1]),y);loss.backward()
nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True);opt.step()
model.eval();total=0.;count=0
with torch.no_grad():
for start in range(0,len(val),8):
x,_,y=make_batch(val[start:start+8],stoi);loss,n=token_nll(model(x,y[:,:-1]),y)
total+=loss.item()*n;count+=n
score=total/count;history.append(score)
if score<best:best=score;snapshot=copy.deepcopy(model.state_dict())
model.load_state_dict(snapshot)
return model,history
if __name__=='__main__':
pairs=reverse_pairs(40);train,val=pairs[:30],pairs[30:];itos,stoi=make_vocab(train)
transformer,history=train_translation(train,val,stoi)
src,_,target=make_batch(val[:2],stoi)
assert all(math.isfinite(v) for v in history)
a=transformer(src,target[:,:-1]);changed=target[:,:-1].clone();changed[:,-1]=4
b=transformer(src,changed)
assert torch.allclose(a[:,:-1],b[:,:-1],atol=1e-6)
print('synthetic encoder-decoder training',history,translation_greedy(transformer,src)[1].tolist())For the original six-block English–German exercise, provide already-local training/validation/test TSVs via S034's loader, use train vocabulary, set layers=6 and choose a sufficient position limit. Evaluate token-weighted NLL and decoded corpus BLEU with recorded tokenization/references, not the synthetic exact-match metric. Compare a decoder-only model trained on source+BOS+target with source labels ignored and target labels shifted correctly. Match parameters and consumed source/target compute; do not promise “within a couple BLEU.” No Multi30k download or corpus run occurred here.
Decoder-only, the FFN budget and exact counting
A decoder-only model keeps causal self-attention and FFN branches but removes the separate source encoder/cross-attention. Token/position embeddings create a residual stream; pre-norm blocks update it; final LN and a vocabulary projection produce logits. The full implementation in S040 supplies that module and sampling. A tied output projection shares the input embedding parameter, not just equal initial values. It saves Vd parameters; it does not halve the whole network except in special parameter budgets.
For a bias-free attention and two biased FFN linears with hidden width m, a block has 4d²+2dm+m+d+4d parameters: four matrices, two FFN biases and two LN gain/bias pairs. A tied-head stack adds Vd+Pd+2d for token table, position table and final LN. At d=512,m=2048,L=12,V=50257,P=1024 the exact total under these conventions is 64,060,928. GPT-2-compatible d=768,m=3072,L=12 with attention biases adds another 4d per block and totals 124,439,808. The historical 117M label is not reconciled by “weight tying makes 124M become117M”; tied counting already occurred.
Vanilla m=4d gives about 8d² FFN matrix parameters versus 4d² attention parameters: two-thirds of block matrices, not necessarily two-thirds of the entire model including embeddings. The ratio is empirical, not a mathematical minimum. SwiGLU uses three matrices, so m≈8d/3 matches the 8d² matrix budget before integer rounding. Width, activation, gating and sparsity vary by model; private GPT-4 dimensions are not established.
Exact NumPy decoder and analytic backward pass
The protected module below remains byte-for-byte intact. Its input IDs are (B,T); embedding E is (V,d) and position table P is (context,d). A pre-normalized one-head attention branch writes into the stream, then a width-2d ReLU FFN writes another residual. Its untied output head has no final normalization. This omission is a declared variant, not an accidental claim to reproduce GPT.
For V=5,d=8,context=8 the count is 40+64+256+256+40+32=688. Embedding backward must scatter-add repeated IDs; assignment loses repeated-token contributions. Residual backward distributes the upstream derivative down both branches. For unit-normalized features u and incoming gradient already multiplied by gain g, LN backward is inv_std times [g−mean(g)−u mean(gu)], with means over features per token. Normalizing across time could leak future information even with a correct attention mask.
"""Inspectable one-head, one-block NumPy decoder. No downloads, files or GPU.
Copy as tiny_transformer.py for the course training and serving labs.
Architecture deliberately omits dropout, final norm, tied weights and multi-head.
"""
import numpy as np
def softmax(x):
z = x - x.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
def norm(x, gain, bias):
centered = x - x.mean(axis=-1, keepdims=True)
inv = 1.0 / np.sqrt((centered ** 2).mean(axis=-1, keepdims=True) + 1e-5)
unit = centered * inv
return unit * gain + bias, (unit, inv, gain)
def norm_backward(dy, cache):
unit, inv, gain = cache
g = dy * gain
dx = inv * (g - g.mean(axis=-1, keepdims=True)
- unit * (g * unit).mean(axis=-1, keepdims=True))
return dx, (dy * unit).sum(axis=(0, 1)), dy.sum(axis=(0, 1))
class TinyDecoder:
def __init__(self, vocab=5, width=8, context=8, seed=7):
rng = np.random.default_rng(seed)
d = width
self.context, self.width = context, d
self.p = {"E": rng.normal(0, .1, (vocab, d)),
"P": rng.normal(0, .1, (context, d))}
for name in ("Q", "K", "V", "O"):
self.p[name] = rng.normal(0, 1 / np.sqrt(d), (d, d))
self.p["F"] = rng.normal(0, 1 / np.sqrt(d), (d, 2*d))
self.p["G"] = rng.normal(0, 1 / np.sqrt(2*d), (2*d, d))
self.p["U"] = rng.normal(0, .1, (d, vocab))
for n in ("1", "2"):
self.p["g" + n] = np.ones(d)
self.p["b" + n] = np.zeros(d)
def forward(self, ids):
p = self.p
ids = np.asarray(ids)
if ids.ndim != 2 or not np.issubdtype(ids.dtype, np.integer):
raise ValueError('IDs must be an integer batch-by-time array')
B, T = ids.shape
if B == 0 or not 0 < T <= self.context:
raise ValueError('empty batch or invalid context length')
if np.any(ids < 0) or np.any(ids >= len(p['E'])):
raise ValueError('token ID outside vocabulary')
x = p["E"][ids] + p["P"][:T]
n, c1 = norm(x, p["g1"], p["b1"])
q, k, v = [n @ p[name] for name in ("Q", "K", "V")]
scores = q @ k.transpose(0, 2, 1) / np.sqrt(self.width)
mask = np.tril(np.ones((T, T), dtype=bool))
a = softmax(np.where(mask, scores, -np.inf))
av = a @ v
h = x + av @ p["O"]
m, c2 = norm(h, p["g2"], p["b2"])
f = m @ p["F"]
r = np.maximum(f, 0)
z = h + r @ p["G"]
logits = z @ p["U"]
cache = (ids, n, c1, q, k, v, a, av, h, m, c2, f, r, z)
return logits, cache
def loss_grad(self, ids, targets):
logits, c = self.forward(ids)
targets = np.asarray(targets)
if targets.shape != logits.shape[:2] or not np.issubdtype(targets.dtype, np.integer):
raise ValueError('targets must match input shape and use integer IDs')
if np.any(targets < 0) or np.any(targets >= logits.shape[-1]):
raise ValueError('target ID outside vocabulary')
ids, n, c1, q, k, v, a, av, h, m, c2, f, r, z = c
B, T = ids.shape
flat = logits.reshape(-1, logits.shape[-1])
y = targets.reshape(-1)
shifted = flat - flat.max(axis=-1, keepdims=True)
loss = (np.log(np.exp(shifted).sum(axis=-1))
- shifted[np.arange(B*T), y]).mean()
dl = softmax(flat)
dl[np.arange(B*T), y] -= 1
dl = dl.reshape(logits.shape) / (B*T)
p, g = self.p, {}
def weight_grad(x, dy):
return x.reshape(-1, x.shape[-1]).T @ dy.reshape(-1, dy.shape[-1])
g["U"] = weight_grad(z, dl)
dz = dl @ p["U"].T
g["G"] = weight_grad(r, dz)
df = (dz @ p["G"].T) * (f > 0)
g["F"] = weight_grad(m, df)
dh_norm, g["g2"], g["b2"] = norm_backward(df @ p["F"].T, c2)
dh = dz + dh_norm
g["O"] = weight_grad(av, dh)
dav = dh @ p["O"].T
da = dav @ v.transpose(0, 2, 1)
dv = a.transpose(0, 2, 1) @ dav
ds = a * (da - (da * a).sum(axis=-1, keepdims=True))
dq = ds @ k / np.sqrt(self.width)
dk = ds.transpose(0, 2, 1) @ q / np.sqrt(self.width)
dn = np.zeros_like(n)
for name, upstream in (("Q", dq), ("K", dk), ("V", dv)):
g[name] = weight_grad(n, upstream)
dn += upstream @ p[name].T
dx_norm, g["g1"], g["b1"] = norm_backward(dn, c1)
dx = dh + dx_norm
g["P"] = np.zeros_like(p["P"])
g["P"][:T] = dx.sum(axis=0)
g["E"] = np.zeros_like(p["E"])
np.add.at(g["E"], ids, dx) # repeated IDs accumulate, not overwrite
return float(loss), g
if __name__ == "__main__":
model = TinyDecoder()
x = np.array([[0, 1, 2, 0]])
loss, grads = model.loss_grad(x, np.array([[1, 2, 0, 1]]))
assert np.isfinite(loss)
assert set(grads) == set(model.p)
print("parameters", sum(p.size for p in model.p.values()), "loss", round(loss, 6))The module's own smoke check is weaker than the separate all-coordinate finite-difference oracle. That oracle perturbs every parameter of a small configuration, tests repeated-token accumulation and compares loss gradients against centered differences. Avoid perturbation points at ReLU zero. Optimizer updates stay outside loss_grad, permitting checks, accumulation and clipping on one frozen parameter snapshot. No derivative through discrete token IDs is claimed.
Modern alternatives with explicit equations
RMSNorm rescales by root mean square without centering; standard versions learn gain and omit bias. LN and RMSNorm are not numerically identical, and no fixed speedup or “one less failure mode” follows. SwiGLU computes W_out[SiLU(W_gate x)⊙W_value x]. Omitting linear biases, using RoPE and grouping KV heads are independent model choices. Llama 3's public model is a useful concrete reference, not the architecture of every open-weight or proprietary model.
from positions_lab import rope_angles,apply_rope
class RMSNorm(nn.Module):
def __init__(self,width,eps=1e-6):
super().__init__();self.g=nn.Parameter(torch.ones(width));self.eps=eps
def forward(self,x):
work=x.float() if x.dtype in (torch.float16,torch.bfloat16) else x
normalized=work*torch.rsqrt(work.square().mean(-1,keepdim=True)+self.eps)
return normalized.to(x.dtype)*self.g
class SwiGLU(nn.Module):
def __init__(self,width,hidden):
super().__init__();self.gate=nn.Linear(width,hidden,bias=False)
self.value=nn.Linear(width,hidden,bias=False);self.out=nn.Linear(hidden,width,bias=False)
def forward(self,x):return self.out(F.silu(self.gate(x))*self.value(x))
class GQARoPE(nn.Module):
"""Full-sequence correctness reference; no inference cache or fused GQA promise."""
def __init__(self,width,heads,kv_heads):
super().__init__()
if width%heads or heads%kv_heads or (width//heads)%2:raise ValueError('GQA divisibility/even head width')
self.heads=heads;self.kv_heads=kv_heads;self.d=width//heads
self.q=nn.Linear(width,width,bias=False);self.k=nn.Linear(width,kv_heads*self.d,bias=False)
self.v=nn.Linear(width,kv_heads*self.d,bias=False);self.out=nn.Linear(width,width,bias=False)
def forward(self,x):
B,T,C=x.shape
q=self.q(x).reshape(B,T,self.heads,self.d).transpose(1,2)
k=self.k(x).reshape(B,T,self.kv_heads,self.d).transpose(1,2)
v=self.v(x).reshape(B,T,self.kv_heads,self.d).transpose(1,2)
cos,sin=rope_angles(torch.arange(T,device=x.device),self.d)
q=apply_rope(q,cos,sin);k=apply_rope(k,cos,sin)
repeat=self.heads//self.kv_heads
k=k.repeat_interleave(repeat,1);v=v.repeat_interleave(repeat,1)
z=F.scaled_dot_product_attention(q,k,v,is_causal=True,dropout_p=0.)
return self.out(z.transpose(1,2).reshape(B,T,C))
class ModernBlock(nn.Module):
def __init__(self,width=16,heads=4,kv_heads=2,hidden=40):
super().__init__();self.n1=RMSNorm(width);self.n2=RMSNorm(width)
self.attn=GQARoPE(width,heads,kv_heads);self.ffn=SwiGLU(width,hidden)
def forward(self,x):
x=x+self.attn(self.n1(x));return x+self.ffn(self.n2(x))
modern=ModernBlock().double();probe=torch.randn(1,4,16,dtype=torch.float64,requires_grad=True)
y=modern(probe);y.square().mean().backward()
assert y.shape==probe.shape and torch.isfinite(probe.grad).all()
assert sum(p.numel() for p in SwiGLU(12,32).parameters())==8*12*12
print('modern full-sequence block and SwiGLU budget: pass')This expands grouped keys/values as a clarity reference; it is not a memory-saving serving kernel and has no incremental cache. QK-normalization is another variant: the original QKNorm study uses L2-normalized queries/keys with a learned scale, not universally RMSNorm. Logit soft-capping uses cap×tanh(logit/cap), bounding magnitude but not proving saturated softmax or training failures impossible. Gemma 2 also combines local/global attention and GQA. DeepNorm changes residual scaling and initialization to stabilize deep models. Value residuals and U-shaped skip streams in modded-nanoGPT change which layers/values are combined; they must be read at a dated repository revision, not treated as standard Transformer requirements or a universal loss gain.
Debugging and architecture tradeoffs
Create masks on the input device, register reusable fixed masks as buffers where appropriate, and verify orientation/boolean meaning. A final LN belongs in the specified GPT-style model; not every valid Transformer variant requires that exact final operation. Misnesting LN changes pre-norm to post-norm even when shapes match. Test causality by changing only future inputs and comparing earlier logits/gradients.
Decoder-only uses one stream and a natural next-token objective. Encoder-decoder can pretrain on unlabeled text through denoising/corruption, as T5 demonstrates; it does not inherently need manually paired labels. Both support conditional generation and in-context prompts. Translation FLOPs-per-BLEU, general quality and scaling are empirical choices, not universal architecture rankings.
For new experiments, pre-norm LN or RMSNorm can be reasonable baselines. Keep checkpoint compatibility when reproducing a trained model. Post-norm controls stream scale but changes gradient paths; pre-norm has additive paths but residual variance can grow. RMSNorm trades centering invariance for a simpler reduction. Validate optimization and task outcomes rather than choosing from a slogan about “every frontier LM.”
Practice and worked answers
- Ordered components. A GPT-style block reads LN→causal MHA→residual add, then LN→position-wise FFN→residual add. The surrounding model adds embeddings/positions, a final norm and vocabulary head. The original post-LN places each norm after its add.
- Residual math. For x+F(x), the Jacobian is I+J_F. Zero complete branches yield identity; removing skips changes optimization but does not prove every model above four layers fails. The total derivative can still cancel or grow.
- Why pre-norm? Its identity term improves initialization/optimization behavior in studied settings. Xiong's post-LN concern is large parameter gradients near the output. Neither no-warmup nor stable arbitrary depth is guaranteed.
- FFN width. A 4d two-matrix FFN has 8d² matrix parameters. SwiGLU has three matrices; m=8d/3 matches that budget before rounding. Attention remains nonlinear itself; the FFN adds a distinct position-wise transformation.
- Why decoder-only? Simplicity and one causal next-token objective make it a useful language-model architecture. Encoder-decoder still offers bidirectional source processing and explicit cross-attention, and can pretrain from unlabeled data. No universal quality or FLOP victory follows.
- Compare Multi30k properly. Load already-local disjoint corpora, train the complete conditional models, select on validation, score test once with identical tokenization/references, and compare matched parameter/token/computation budgets. The executed synthetic comparison does not supply a BLEU result.
Additional original recall answers: the causal allowed mask is a boolean lower triangle on the correct device; a decoder in an encoder-decoder has self-attention, cross-attention and FFN; LN/FFN/attention shapes remain (B,T,d) around residuals. The NumPy exercises retain repeated-ID scatter-add, square-root head scaling, differentiation separate from updates and feature-only normalization. Review those tomorrow and sketch both complete architectures next week.
Sources, optional reading and execution boundary
Primary: Transformer v7, Xiong et al., LayerNorm, RMSNorm, GLU variants, DeepNorm, Llama3 model source, T5. Optional paper walkthrough, Karpathy, Annotated Transformer remain optional reading suggestions.
Executed scope: exact protected NumPy program and independent derivatives, PyTorch block/causality/gradient tests, complete small synthetic encoder-decoder training and full-sequence modern-block contracts. No pretrained weights, Multi30k corpus, GPU/distributed benchmark, app build or release. Previous · Next: complete GPT. The earlier local-lab course map and BPE continuation are optional next steps.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.