S111 · Full Transformer Architecture — Encoder + Decoder

Module M13: NLP & Transformers · Session 111 of 130 · Track: LLM 🏗️

What you'll be able to do after this session

• Explain Full Transformer Architecture to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S109 — Multi-Head Attention — Parallel Views

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Illustrated Transformer walkthrough (Jay Alammar / StatQuest) — StatQuest builds the whole architecture from zero — best 30-min intuition.
• 🎥 Deep dive (30–60 min) · Let's build GPT from scratch — Andrej Karpathy — Karpathy codes a decoder-only Transformer live; the reference deep dive.
• 🎥 Hands-on demo (10–20 min) · Transformers explained visually — 3Blue1Brown — 5 minutes of the cleanest visual math you'll ever see.
• 📖 Canonical article · Attention Is All You Need (2017) — The original paper — Figure 1 is worth 1000 lectures.
• 📖 Book chapter / tutorial · The Illustrated Transformer — Jay Alammar's diagrams are the internet's canonical explainer.
• 📖 Engineering blog · D2L.ai — The Transformer Architecture — Textbook chapter with runnable code and shape annotations.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Putting it all together.

Imagine you are translating an English sentence into French. You do it in two passes. First you read the whole English sentence, thinking about every word in the context of every other word — that's the encoder. Then you write French one word at a time, and for each new French word you (a) look back at what you've already written and (b) glance at the English sentence to find the right piece to translate — that's the decoder. The full Transformer is exactly this two-tower shape, wired together.

Before Transformers, we did the same job with RNNs one word at a time, which was slow and forgot the beginning by the time it reached the end. Vaswani et al. asked: what if we throw away recurrence entirely and let every word attend to every other word in parallel? The answer was a stack of identical blocks — each block is attention → add & norm → feed-forward → add & norm — on both sides. Six blocks in the encoder, six blocks in the decoder in the original paper. Same LEGO piece, stacked.

The gotcha to carry forever: encoder attention is bidirectional (every token sees every other token), but decoder self-attention is masked (a token can only see tokens to its left). Without that mask you'd be cheating — peeking at the answer while training. This one line of code is the reason GPT can generate text but BERT cannot.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The full Transformer is two stacks that talk through cross-attention. Here's the wiring:

Worked example — translate "I love cats" → "J'aime les chats". The encoder reads all three English tokens at once, produces a 3×d matrix of contextual vectors (let's say d=512). The decoder starts with a special <bos> token, runs masked self-attention (trivial with one token), then cross-attention: it forms a query vector from <bos>, dot-products it against the 3 encoder key vectors, softmaxes to get attention weights like [0.7, 0.2, 0.1], and pulls a weighted sum of encoder values. That becomes the hidden state that predicts "J'". Next step, decoder input is <bos> J', masked self-attn lets "J'" see <bos> (not future tokens), cross-attn pulls from encoder again — this time perhaps weighting "love" heavily to emit "aime". Repeat until <eos>.

Two knobs to remember: N (number of blocks — usually 6, 12, 24, 96) and d_model (usually 512, 768, 1024, 12288). Bigger both = smarter and slower.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

A minimal full Transformer forward pass in PyTorch — 60 lines, no external tokenizer:

import torch, torch.nn as nn
 
class Block(nn.Module):
    def __init__(self, d, heads, cross=False):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.cross = cross
        if cross:
            self.cross_attn = nn.MultiheadAttention(d, heads, batch_first=True)
            self.ln_c = nn.LayerNorm(d)
        self.ff = nn.Sequential(nn.Linear(d, 4*d), nn.GELU(), nn.Linear(4*d, d))
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
 
    def forward(self, x, mem=None, causal=False):
        L = x.size(1)
        mask = torch.triu(torch.ones(L, L), 1).bool() if causal else None
        a, _ = self.self_attn(x, x, x, attn_mask=mask)
        x = self.ln1(x + a)
        if self.cross:
            c, _ = self.cross_attn(x, mem, mem)
            x = self.ln_c(x + c)
        return self.ln2(x + self.ff(x))
 
class Transformer(nn.Module):
    def __init__(self, vocab=1000, d=128, heads=4, n_enc=2, n_dec=2, max_len=32):
        super().__init__()
        self.tok = nn.Embedding(vocab, d)
        self.pos = nn.Embedding(max_len, d)
        self.enc = nn.ModuleList([Block(d, heads) for _ in range(n_enc)])
        self.dec = nn.ModuleList([Block(d, heads, cross=True) for _ in range(n_dec)])
        self.head = nn.Linear(d, vocab)
 
    def embed(self, ids):
        p = torch.arange(ids.size(1))
        return self.tok(ids) + self.pos(p)
 
    def forward(self, src, tgt):
        h = self.embed(src)
        for b in self.enc: h = b(h)
        y = self.embed(tgt)
        for b in self.dec: y = b(y, mem=h, causal=True)
        return self.head(y)
 
model = Transformer()
src = torch.randint(0, 1000, (1, 5))   # "I love cats and dogs"
tgt = torch.randint(0, 1000, (1, 4))   # "<bos> J' aime les"
logits = model(src, tgt)
print(logits.shape)                     # torch.Size([1, 4, 1000])
print("params:", sum(p.numel() for p in model.parameters()))

Checklist while running:

1. Output shape is [batch, tgt_len, vocab] — one distribution per output position.
2. Total param count printed at bottom (~1M for these tiny hyperparams).
3. No NaN in logits — if you see them, positional embeddings out of range.
4. Cross-attention layer only appears in decoder blocks (check model.dec[0].cross).
5. Causal mask is upper-triangular — position i can only see 0..i.

Try this: flip causal=False in the decoder and re-run — training loss will collapse to zero because the model can peek at future tokens. That's why the mask exists.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Nobody in 2024 trains a full encoder–decoder Transformer from scratch for a generic task — the industry has largely split into three streams (encoder-only for embeddings, decoder-only for generation, enc-dec for translation/summarization). But the pieces above are still the atomic units, so bugs from Vaswani's 2017 paper still ship in production every day.

Classic war stories. (1) Silently swapped Q and K — model still trains, loss goes down, but attention patterns are garbage. Caught only by visualizing attention weights. (2) LayerNorm before vs after — the original paper puts LN after the residual add ("Post-LN"). GPT-2 and every modern model use Pre-LN because Post-LN diverges without careful warmup. If you copy-paste from Vaswani, you'll hit exploding gradients around step 500. (3) Positional embedding overflow — the model trained on max_len=512 and someone feeds a 2048-token doc at inference; the position embedding table has no entry, so PyTorch either errors or silently indexes zero. Fix: RoPE or ALiBi (S110). (4) Cross-attention key/value cached wrongly during beam search — the encoder output is fixed once per input, but junior devs recompute it every decoder step, burning 10× GPU. Always cache enc_out outside the decoding loop.

How top teams handle it. Meta's Llama, Google's Gemini, and Anthropic's Claude all use decoder-only stacks (no encoder tower), Pre-LN, RMSNorm instead of LayerNorm, SwiGLU FFN, and RoPE. The "full" enc-dec shape from the original paper survives mostly in translation (Google's NMT), speech (Whisper), and pixel/image-to-text tasks. When you see BART, T5, mBART, or Whisper — that's the classic 2017 diagram, still shipping billions of requests a day.

(e) Quiz + exercise · 10 min

1. In one sentence, what does the decoder's cross-attention layer do?
2. Why is the decoder's self-attention masked but the encoder's is not?
3. What are the two sub-layers inside every encoder block (in order)?
4. If encoder outputs shape [B, S_enc, d] and decoder input shape [B, S_dec, d], what shape is the cross-attention matrix?
5. Name two things that changed between "Vaswani 2017" and "modern GPT-style" architectures.

Stretch (connects to S109 — Multi-Head Attention): In cross-attention, whose tensor becomes Q and whose becomes K/V — and why does swapping them break translation? Trace one worked example on paper.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Full Transformer Architecture? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S112): Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each
• Previous (S110): Positional Encoding — Sinusoidal, Learned, RoPE
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.