S112 · Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each
The three families of Transformer architectures — what each is optimised for, why BERT never generates, why GPT never fills in blanks well, and when T5's encoder-decoder still wins in 2026.
🎯 Pick the right Transformer family (encoder / decoder / enc-dec) for a task in under 30 seconds, and explain why.
Why this session exists
"Just use GPT" is a fine 2023 answer and a bad 2026 answer. Half the production NLP systems you'll ever build are not text-generation problems — they're classification, embeddings, retrieval, entity extraction, or seq2seq translation. Each of those has an architecture family that is 10× cheaper and often more accurate than an autoregressive decoder-only LLM. This session teaches you the three families, the shape of their attention masks, and the one-question decision tree that picks between them.
- Draw the attention mask for BERT, GPT, and T5 from memory and explain what each cell means.
- Pick the right family (encoder / decoder / enc-dec) for a new NLP task in under 30 seconds.
- Explain why BERT can't generate fluent text and why GPT is a worse embedding model than BERT.
- Load a HuggingFace model of each family, run inference, and read its output tensor shape correctly.
- Justify to a PM why 'use an LLM for this' is often the wrong call vs a 110M-param encoder.
Prerequisites
- S110 · Attention & Transformers — the base architecture we're now specialising.
- S111 · Tokenization — BPE/WordPiece/SentencePiece; all three families share this input layer.
- S102 · CNNs helps only for intuition on "pretrained backbone + task head" style transfer.
(a) Intuition · 5 min
Give three interns the same 500-word paragraph. Intern A (BERT) is asked to read the whole thing, then answer questions like "which word here is the subject?" — she can look forwards and backwards freely, so she builds a rich understanding but never writes anything new.
Intern B (GPT) is only allowed to see words up to a cursor and asked to write the next word, then move the cursor forward and repeat. He becomes a fantastic writer and a mediocre analyst.
Intern C (T5) does both jobs sequentially: first he reads the input like Intern A (bidirectional), then he writes the output like Intern B (autoregressive). He's slower and needs both skills, but he's the one you send when the task is "translate this paragraph into French."
All three are the same Transformer block underneath — the only difference is the attention mask and the training objective. Encoder = full bidirectional mask + masked-LM loss. Decoder = causal (lower-triangular) mask + next-token loss. Enc-dec = both, glued together with cross-attention.
Which means once you understand the mask matrix and the loss, you have understood the entire family tree.
- Encoder-only (BERT, RoBERTa, DeBERTa, ModernBERT) — bidirectional attention, output is one vector per token, no generation. Best for: classification, embeddings, NER, extractive QA.
- Decoder-only (GPT, LLaMA, Mistral, Qwen, Claude, Gemini) — causal attention, output is next-token distribution, generates left-to-right. Best for: chat, code, open-ended generation, in-context learning.
- Encoder-decoder (T5, BART, FLAN-T5, mT5, Marian) — encoder reads input, decoder writes output attending to encoder via cross-attention. Best for: translation, summarisation, structured seq2seq where input and output are distinct.
A short history so today makes sense
- 2017Transformer paper'Attention Is All You Need' — Vaswani et al. Original architecture is encoder-decoder, built for translation.
- 2018BERT · GoogleEncoder-only + masked language modelling. Sweeps 11 NLP benchmarks. Kicks off the pretrain+finetune era.
- 2018GPT-1 · OpenAIDecoder-only + next-token prediction. Quieter debut, same year as BERT. Bet on scale over cleverness.
- 2019T5 · Google'Text-to-Text Transfer Transformer'. Reframes every NLP task as string → string. Enc-dec makes a comeback.
- 2020GPT-3175B params, in-context learning shows up. The 'just prompt it' era begins.
- 2024ModernBERTAnswer.AI + LightOn revive the encoder family with 8K context + Flash Attention. BERT is not dead.
(b) Visual walkthrough · 15 min
The three attention masks
The single most important picture in this session. Rows = query tokens, columns = key tokens. A filled cell means "this query can attend to that key."
The training objective is what actually shapes them
Randomly mask 15% of input tokens. Model must predict the missing token from left AND right context. Loss = cross-entropy on masked positions only.
For every position t, predict token t+1 given tokens 1..t. Loss = cross-entropy on every position. This is why decoders can generate.
Mask contiguous spans in the input. Encoder sees the corrupted input; decoder generates the missing spans as a sequence. Blends both worlds.
The stack, side by side
What lives inside each family
Choose your fighter — the 30-second decision tree
Which model wins which task
BERT · RoBERTa · DeBERTa · ModernBERT
- Classification (sentiment, toxicity, intent)
- Embeddings for search / RAG (sentence-BERT)
- Named-entity recognition
- Extractive QA (return a span)
- Very cheap: 110M–400M params run on CPU
GPT · Llama · Mistral · Claude · Gemini
- Chat, code, creative writing
- In-context learning / few-shot prompts
- Tool use + function calling
- Long open-ended generation
- Big and slow: 7B–1T+ params, need GPUs
T5 · FLAN-T5 · BART · Marian · mT5
- Machine translation (source → target)
- Abstractive summarisation
- Grammar correction / rewriting
- Data-to-text (JSON → sentence)
- Sweet spot: 60M–11B, cheaper than GPT-class
"BERT and GPT differ mainly in size and training data. Now that GPT-class models are huge, BERT is obsolete — I should use an LLM for everything, including embeddings and classification."
They differ in the masking, and that difference is structural, not incidental. BERT's bidirectional attention lets every token condition on the full sequence, which is why a 110M-parameter encoder still produces better retrieval embeddings per FLOP than a decoder LLM, and why encoders remain the workhorse of production search and reranking. A decoder's last-token representation has never seen the tokens after it.
Because the generative capability of decoder LLMs is so visible and so general that it reads as strict dominance. And a decoder LLM can do classification — just at a hundred times the cost per document, with worse latency, and with the answer arriving as text you then have to parse. The myth is expensive rather than wrong-looking, which is why it survives review.
Make the asymmetry concrete — the encoder's representation of token i depends on token i+1; the decoder's does not:
# Conceptual check, no download needed.
# Encoder (bidirectional): h_i = f(x_1..x_T) for every i
# Decoder (causal) : h_i = f(x_1..x_i) for every i
#
# So for a fill-in-the-blank task:
# 'the [MASK] sat on the mat'
# BERT conditions on 'sat on the mat'. A causal decoder cannot,
# unless you re-order the problem into a continuation.
#
# Verify on a real model:
# from transformers import pipeline
# pipeline('fill-mask', model='bert-base-uncased')('the [MASK] sat on the mat')Why does BERT mask only ~15% of tokens, and why does it replace 10% of those with a random token instead of always using [MASK]?
- 1The training signal comes only from masked positions: the loss is computed nowhere else.forced by · unmasked positions have their answer visible in the input, so predicting them is trivial and teaches nothing
- 2So mask too few and each expensive forward pass yields almost no gradient — sample efficiency collapses. Mask 1% and you pay a full sequence of compute for one prediction.forced by · cost per step is fixed by sequence length; useful signal scales with the number of masked positions
- 3But mask too many and the surviving context is too sparse to determine the answer, so the task becomes noise and the model learns the unigram distribution.forced by · reconstruction requires enough intact context to constrain the missing token
- 4Roughly 15% is where those two pressures balance for natural language — enough signal per pass, enough context to make each prediction determinate.forced by · it is an empirical optimum of the sample-efficiency/task-difficulty tradeoff, not a derived constant
- 5Now the second problem:
[MASK]appears during pretraining but never at fine-tuning or inference. If it were the only signal, the model would learn "produce a contextual representation only where I see [MASK]" — a train/serve distribution mismatch.forced by · the model can condition on the presence of the token itself, and it will, because that is the easiest discriminative feature available - 6Replacing 10% of selected positions with a random token and leaving 10% unchanged means the model can never be sure a given position is correct. It must build a contextual representation of every token, in case that one is corrupted.forced by · uncertainty about which positions are corrupted forces the useful behaviour everywhere
Therefore the 80/10/10 split is not superstition — it is a deliberate device to prevent the model from keying on the artificial [MASK] symbol, at the cost of a slightly noisier objective.
And note what this predicts: an objective with no artificial token would avoid the mismatch entirely and should do better. That is exactly ELECTRA's replaced-token detection, and also why T5 frames everything as text-to-text with sentinel spans — both are answers to the same defect this derivation exposes.
All three families are the same transformer trained under different rules about what a token is allowed to see. BERT hides scattered tokens and lets you look both ways. GPT hides everything to the right. T5 hides contiguous spans and makes you write them out as text.
The mask defines the model. Architecture is downstream of the masking scheme, not the other way round.
- BERT: bidirectional, masked-LM, encoder-only → understanding, embeddings, reranking. Cannot generate.
- GPT: causal, next-token, decoder-only → generation and, at scale, everything else via prompting.
- T5: encoder-decoder, span corruption, every task cast as text-in/text-out → clean transduction with a uniform interface.
- Cost intuition: encoder = one pass for the whole sequence. Decoder = one pass per generated token. That ratio is why encoders survive at retrieval scale.
Fire this model the moment you see: an embedding model choice · someone using a chat LLM to classify millions of rows · a fill-in-the-middle requirement · a "which model should we fine-tune" debate · latency budgets that a per-token decoder cannot meet.
You must classify tens of millions of documents per day. Fine-tune a small encoder, or prompt a hosted LLM?
Prototype with the LLM, ship with the encoder. The LLM is the fastest way to discover what your label set should actually be; the encoder is the only economical way to apply it at scale. Distillation is the bridge and is usually the right long-run answer.
The measurable trigger to switch is straightforward: compute cost-per-item × daily volume for both paths. When the LLM line crosses the fully-loaded cost of training and hosting a small model, the decision has already been made for you — and at tens of millions of documents a day it crossed long ago.
(c) Hands-on · 25 min
Load one model from each family, run inference on the same input, and inspect the output shapes so the "three families" idea becomes concrete. This is the single best 20-minute exercise for making the abstract concrete.
"""three_families.py — one input, three architectures, three output shapes.
Run once, read the printed shapes carefully. Every 'family' distinction in
this session is visible in the tensor shape of the model's output.
"""
from __future__ import annotations
import torch
from transformers import (
AutoTokenizer,
AutoModel, # encoder — returns hidden states
AutoModelForCausalLM, # decoder — returns next-token logits
AutoModelForSeq2SeqLM, # enc-dec — has generate() for translation
)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
TEXT = "The transformer architecture changed natural language processing forever."
# ---------- 1. Encoder-only (BERT) ----------
print("\n--- 1. BERT · encoder-only ---")
bert_name = "bert-base-uncased" # 110M params
bert_tok = AutoTokenizer.from_pretrained(bert_name)
bert = AutoModel.from_pretrained(bert_name).to(DEVICE).eval()
inputs = bert_tok(TEXT, return_tensors="pt").to(DEVICE)
with torch.no_grad():
out = bert(**inputs)
# Shape: (batch=1, seq_len, hidden=768). One vector per token.
print("hidden states :", out.last_hidden_state.shape)
print("[CLS] vector :", out.last_hidden_state[0, 0].shape,
"→ feed to a classifier head")
# ---------- 2. Decoder-only (GPT-2) ----------
print("\n--- 2. GPT-2 · decoder-only ---")
gpt_name = "gpt2" # 124M params
gpt_tok = AutoTokenizer.from_pretrained(gpt_name)
gpt = AutoModelForCausalLM.from_pretrained(gpt_name).to(DEVICE).eval()
inputs = gpt_tok(TEXT, return_tensors="pt").to(DEVICE)
with torch.no_grad():
out = gpt(**inputs)
# Shape: (batch=1, seq_len, vocab=50257). Distribution over next token
# at every position (that's why decoders generate).
print("logits :", out.logits.shape)
next_id = out.logits[0, -1].argmax().item()
print("next token :", repr(gpt_tok.decode([next_id])))
# ---------- 3. Encoder-decoder (T5) ----------
print("\n--- 3. T5-small · encoder-decoder ---")
t5_name = "t5-small" # 60M params
t5_tok = AutoTokenizer.from_pretrained(t5_name)
t5 = AutoModelForSeq2SeqLM.from_pretrained(t5_name).to(DEVICE).eval()
# T5 speaks 'task prefixes' — the same model does translation, summary, QA.
prompt = "translate English to German: " + TEXT
inputs = t5_tok(prompt, return_tensors="pt").to(DEVICE)
with torch.no_grad():
generated = t5.generate(**inputs, max_new_tokens=40)
print("output tokens :", generated.shape)
print("translation :", t5_tok.decode(generated[0], skip_special_tokens=True))
print("\nDone. Notice: BERT gives you *vectors*, GPT gives you *logits*, "
"T5 gives you a *sequence*. That difference is the whole session.")What each block is doing
Anatomy of the script
Try running:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForCausalLM.from_pretrained("bert-base-uncased")
inputs = tok("The transformer architecture is", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=20)[0]))Read the warning HuggingFace prints. Then compare with the same prompt on GPT-2. The difference isn't the architecture — it's the training objective.
(d) Production reality · 15 min
Pre-BERT, Google's ranker struggled with queries like "2019 brazil traveler to usa need a visa" — it kept returning results about US travellers going to Brazil (reversed intent).
The keyword-matching pipeline had no way to encode the directional meaning of "to" in context.
Google added BERT as a passage re-ranker on top of their existing retrieval. Because BERT reads the query bidirectionally, small function words like "to" and "for" now carry weight.
They shipped it to 10% of English queries in 2019 — one of the largest single-quality jumps in Search history.
Team builds a "classify support tickets into 12 categories" feature using GPT-4 via API. It works, at $0.03/ticket × 20K tickets/day = $600/day = $18K/month. Latency: 2s. Occasional refusals and format drift.
Fine-tune a DistilBERT (66M params) on 5K labelled examples. Inference on a single CPU: 20ms, cost effectively $0, accuracy went from 91% (GPT) to 94% (fine-tuned encoder) because the encoder saw the actual label distribution.
The mistake: reaching for a general-purpose LLM when you had labels the whole time.
Building a separate encoder-decoder model per language pair was N² in cost. Quality on low-resource pairs (say, Kazakh → Zulu) was poor because each pair had little parallel data.
Move to a single massively multilingual T5-style enc-dec model (mT5, later PaLM-2 for translate). One encoder learns "meaning space" across 100+ languages; one decoder generates in any target. Low-resource pairs benefit from transfer.
Where this shows up next in the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
Teach these three to a colleague without notes:
- The three attention masks and what they enable.
- Why "use GPT for everything" is often the wrong call.
- The one-question decision tree for picking a family.
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.