Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each

Why this session exists

Architecture, pretraining objective and output head are three different choices. Confusing them leads to claims such as “BERT has no vocabulary head,” “a decoder must generate to classify,” or “T5 gives a sequence but GPT gives logits.” Each of those mixes a model family with the particular API method called.

This lesson traces masks, losses and output tensors, then gives a local-only pretrained inference recipe and an executable CPU contract model. The contract model verifies shapes and information flow; it does not measure the quality of BERT, GPT-2 or T5.

Outcomes: draw all three masks; build selected-token and shifted-target losses; distinguish base models from task heads; load locally supplied checkpoints without downloads; and compare an encoder, prompted generator or distilled classifier using explicit quality/cost criteria.

Prerequisites

Session 106 covers tokenization, 110 covers position and 111 assembles the Transformer. Embedding lookup is not the same thing as a trained sentence-embedding model. Familiarity with a pretrained backbone plus task head is helpful, but CNN architecture is not required.

(a) Intuition · 5 min

One reader sees a complete document and labels it. A writer sees a prefix and extends it. A translator reads the source fully, then writes a separate target. These are visibility patterns, not claims that the reader cannot write anything or that the writer cannot analyze a document. A decoder's last token sees the entire preceding input; early tokens have less context.

A short history so today makes sense

The 2017 Transformer was encoder-decoder. GPT and BERT appeared in 2018 with different pretraining formulations. BERT's NAACL 2019 paper specifies MLM and NSP, not merely “an encoder.” T5's 2019 preprint, published in JMLR 2020, systematically compares transfer-learning choices and casts classification, translation and other tasks as text-to-text [1,2]. Later encoders such as ModernBERT show that the family continued to evolve; a model name or release date does not establish a 50× serving advantage.

The three attention masks

For four valid input tokens, an encoder allows every query to read every key. A causal decoder allows key j only when j≤i. In an encoder-decoder, source self-attention is full, target self-attention is causal and target cross-attention can read all nonpadding source tokens. With S source tokens and T target tokens, the three matrices have sizes S×S, T×T and T×S.

“Full” is not “no mask whatsoever”: padding and task-specific restrictions still matter. A fixed encoder input can be processed in parallel; a causal training sequence can also be processed in parallel because its ground-truth prefix is available. Generating unknown future tokens still requires a sequential decision process.

The training objective is what actually shapes them

BERT MLM. Select approximately 15% of eligible WordPiece positions. Of selected positions, replace 80% with MASK, 10% with a random token and leave 10% unchanged; compute cross-entropy against the original token at all selected positions [1]. Selected is not synonymous with visibly masked. In expectation that is 12% MASK, 1.5% random and 1.5% unchanged over eligible input tokens. Special tokens/padding are not prediction candidates. A random replacement may accidentally equal the original.

The 15% choice balances prediction signal and retained context in that recipe; it is not a mathematically optimal constant for every model. Leaving some selected tokens unchanged and randomly replacing others reduces the train/fine-tune mismatch associated with always seeing MASK at prediction sites. BERT also trained NSP: distinguish an actual following segment from a randomly sampled one. Later encoders changed or removed that objective. ELECTRA's replaced-token discrimination is another objective, not a proof that every non-MLM objective must outperform MLM.

GPT causal LM. Use $p(x)=\prod_t p(x_t\mid x_{<t})$. In a shifted implementation, logits at input index t predict x_(t+1), and padding or excluded prefix labels are ignored. A model trained to predict missing tokens with right context is not automatically calibrated for this causal factorization.

T5 span corruption. For “The red fox jumps over the lazy dog,” replace a contiguous span such as “red fox” with a unique sentinel. The encoder might receive The <extra_id_0> jumps over <extra_id_1> dog; the target is <extra_id_0> red fox <extra_id_1> the lazy <extra_id_2>. The decoder receives a shifted target, predicts sentinel-delimited removed spans and an end token. T5's recipe uses 15% corruption with mean span length 3 in its selected configuration [2]. A task prefix such as translate English to German: identifies supervised task formatting; it does not give an arbitrary base checkpoint every translation skill without relevant training.

The stack, side by side

BERT combines token, segment and learned absolute position embeddings and uses its documented encoder design. GPT-2 uses learned position embeddings and causal blocks. T5 uses bucketed relative attention bias and a different norm arrangement. The FFN activation, normalization, vocabulary, position and head are not identical across these named models. The shared Transformer idea is a useful starting point, not a sufficient implementation specification.

BERT's base output is (B,S,d); its MLM head outputs (B,S,V), a sequence classifier (B,C) and a token classifier (B,S,C). A GPT causal-LM forward pass outputs (B,T,V) and .generate() returns token IDs (B,generated_length). T5's forward pass also outputs vocabulary logits, on its decoder side; .generate() likewise returns IDs. Shapes expose the requested interface, not a theorem about all capabilities of that family.

Choose your fighter — the 30-second decision tree

Which model wins which task

A fixed-label classifier, NER tagger or extractive QA system is a strong use case for a small task-trained encoder. Retrieval may use a bi-encoder with a contrastive sentence objective; reranking may use a cross-encoder over query-document pairs. Vanilla BERT mean pooling is not automatically a good semantic embedding. A decoder adapted for embeddings can also be effective; bidirectionality alone does not establish a universal ranking.

Generation, code and tool proposals fit a causal language-model interface. Translation, rewriting and summarization offer clear source/target structure and are natural encoder-decoder candidates, but decoder models can perform them too. Small decoders exist and can run on CPU; parameter size and GPU requirements are not definitions of the family. A multimodal vision encoder plus language decoder need not use T5-style cross-attention; visual embeddings may instead be inserted into a decoder sequence.

Three ways to hide the answer

The most useful mental test is to ask which labels are already visible. An unmasked encoder predicting every input token can copy. A causal model with misaligned targets can also copy. MLM hides or corrupts selected positions; causal LM predicts the next unseen token; span corruption reconstructs removed spans through a separate decoder. Architecture and objective cooperate—neither can be inferred solely from the other.

Fine-tuned encoder, prompted LLM or distilled student

For 50,000 labeled tickets and eight stable classes, start with a stratified/time-appropriate held-out evaluation and an encoder baseline. Compare a prompted generator on the same labels, including malformed outputs, abstentions and rare-class recall. A high-volume serving estimate includes training, annotation, host utilization, operations and privacy, not just a per-token API rate.

For a hypothetical 20,000 tickets/day at 0.03 currency units per ticket, arithmetic gives 600/day or 18,000 per 30 days. Those are assumed prices, not a vendor quote. A self-hosted model is not free. Distillation can amortize teacher labeling, but a human-verified holdout is necessary because teacher errors and class bias propagate. “Prototype with an LLM, ship an encoder” is an option, not the only economical architecture.

(c) Hands-on · 25 min

The first code block is fully executable with the existing CPU torch environment. It demonstrates output heads, future-token isolation, selected-position corruption, loss exclusion and span sentinels. No downloaded model is hidden behind these tests.

import torch
from torch import nn
from torch.nn import functional as F
torch.set_num_threads(1);torch.manual_seed(112)
emb=nn.Embedding(32,8);head=nn.Linear(8,32)
ids=torch.tensor([[1,2,3,4]])
x=emb(ids);changed=x.clone();changed[:,3]+=torch.arange(8)
enc=lambda z:F.scaled_dot_product_attention(z[:,None],z[:,None],z[:,None],dropout_p=0.0)[:,0]
dec=lambda z:F.scaled_dot_product_attention(z[:,None],z[:,None],z[:,None],is_causal=True,dropout_p=0.0)[:,0]
assert enc(x).shape==(1,4,8) and head(enc(x)).shape==(1,4,32)
assert not torch.allclose(enc(x)[:,:3],enc(changed)[:,:3])
assert torch.allclose(dec(x)[:,:3],dec(changed)[:,:3],atol=1e-6)
assert not torch.allclose(dec(x)[:,-1],dec(changed)[:,-1])
# Cross-attention has target length 2 and source length 4.
q=torch.randn(1,1,2,8);memory=x[:,None]
cross=F.scaled_dot_product_attention(q,memory,memory,dropout_p=0.0)
assert cross.shape==(1,1,2,8)
# Exact selected 80/10/10 branch accounting, no sampling-frequency assertion.
original=torch.arange(20).remainder(16)[None]
selected=torch.zeros_like(original,dtype=torch.bool);selected[:,:10]=True
branch=torch.tensor([[0]*8+[1,2]+[3]*10])
corrupt=original.clone();corrupt[branch==0]=31;corrupt[branch==1]=7
labels=original.clone();labels[~selected]=-100
assert (corrupt==31).sum()==8 and (branch==1).sum()==1 and (branch==2).sum()==1
logits=torch.randn(1,20,32,requires_grad=True)
loss=F.cross_entropy(logits.flatten(0,1),labels.flatten(),ignore_index=-100)
manual=-logits.log_softmax(-1)[selected].gather(1,original[selected,None]).mean()
assert torch.allclose(loss,manual)
loss.backward();assert logits.grad[~selected].abs().sum()==0
assert logits.grad[selected].abs().sum()>0
# A corrupted-position-only loss would wrongly exclude the unchanged selected token.
assert selected.sum()==10 and (branch<2).sum()==9
# Span boundaries are nonoverlapping [start,end) token intervals.
def corrupt_spans(tokens,spans):
    source=[];target=[];cursor=0
    for i,(a,b) in enumerate(spans):
        if not cursor<=a<b<=len(tokens):raise ValueError('invalid spans')
        sentinel=f'<extra_id_{i}>'
        source+=tokens[cursor:a]+[sentinel];target+=[sentinel]+tokens[a:b];cursor=b
    source+=tokens[cursor:];target+=[f'<extra_id_{len(spans)}>']
    return source,target
source,target=corrupt_spans('The red fox jumps over the lazy dog'.split(),[(1,3),(5,7)])
assert source==['The','<extra_id_0>','jumps','over','<extra_id_1>','dog']
assert target==['<extra_id_0>','red','fox','<extra_id_1>','the','lazy','<extra_id_2>']
try:corrupt_spans(['a','b'],[(0,2),(1,2)])
except ValueError:pass
else:raise AssertionError('overlap not rejected')
print('base',list(enc(x).shape),'MLM logits',list(head(enc(x)).shape),'cross',list(cross.shape))
print('span source',source,'target',target)
print('family visibility, output heads, corruption labels and negative controls passed')

Local-only Hugging Face inference recipe

The following function preserves the original one-input/three-family exercise. It is defined and syntax-checked here, not called: Transformers and three local checkpoints are not required by the executed fixture. Supply trusted local directories containing configuration, tokenizer and safetensors weights for the intended architectures, and use the source-reviewed Transformers 4.57.1 interface. local_files_only=True forbids fallback downloads; trust_remote_code=False avoids checkpoint-supplied Python code. Record the exact files/revisions separately when running it.

from pathlib import Path
 
def inspect_local_families(bert_dir, gpt_dir, t5_dir, text):
    import torch
    from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM, AutoModelForSeq2SeqLM
    results = {}
    for name, directory, loader in [('bert', bert_dir, AutoModel),
                                    ('gpt', gpt_dir, AutoModelForCausalLM),
                                    ('t5', t5_dir, AutoModelForSeq2SeqLM)]:
        path = Path(directory).resolve()
        if not path.is_dir(): raise ValueError('trusted local model directory required')
        options = dict(local_files_only=True, trust_remote_code=False)
        tokenizer = AutoTokenizer.from_pretrained(str(path), **options)
        model = loader.from_pretrained(str(path), use_safetensors=True, **options).cpu().eval()
        prompt = 'translate English to German: '+text if name == 't5' else text
        inputs = tokenizer(prompt, return_tensors='pt')
        with torch.inference_mode():
            if name == 'bert':
                output = model(**inputs)
                results[name] = {'hidden': list(output.last_hidden_state.shape),
                                 'first_token': list(output.last_hidden_state[:,0].shape)}
            else:
                if name == 't5':
                    start = model.config.decoder_start_token_id
                    if start is None: raise ValueError('decoder start token required')
                    forward = model(**inputs, decoder_input_ids=torch.tensor([[start]]))
                else: forward = model(**inputs)
                ids = model.generate(**inputs, do_sample=False, max_new_tokens=20)
                results[name] = {'logits': list(forward.logits.shape), 'ids': list(ids.shape),
                                 'decoded': tokenizer.decode(ids[0], skip_special_tokens=True)}
        del model
    return results

What each block is doing

AutoModel selects the architecture's base class; task-specific auto classes select compatible task-head classes. AutoModelForSeq2SeqLM does not magically add an encoder and decoder to any checkpoint. The tagged BERT implementation contains both BertForMaskedLM and BertLMHeadModel; warning behavior depends on the configuration and loaded keys [3]. The original suggestion to switch a BERT checkpoint into a causal loader and judge gibberish is not a controlled architecture comparison.

For BERT classification, a CLS representation is an input to a trained head, not a guarantee of useful classification. For embeddings, use documented pooling and training objective, apply the padding mask and test normalization. For T5, forward computes logits and generate orchestrates repeated decoder steps; the method does not bypass forward. GPT-2's input and T5's task-prefixed input also tokenize to different lengths, so identical N is not expected across all models.

Practice: test the “BERT cannot generate” premise

Attach a vocabulary head to an encoder and observe (B,T,V) in the CPU fixture. This disproves the missing-head argument, not the objective mismatch. Fill-mask inference can predict an interior token using both sides. Iterative masked generation is possible with an explicit procedure, but it is not automatically the same distribution or quality as a causally pretrained generator. Conversely, a causal decoder can answer a fill-in-the-blank question reformatted as a continuation. Compare objective, configuration, training and evaluation rather than proclaiming an absolute family limitation.

(d) Production reality · 15 min

Treat the original search, support-ticket and multilingual-translation stories as design scenarios unless a source documents the exact deployment. Query direction words can matter to a contextual encoder, but that does not prove a particular production reranker topology. A ticket classifier may reduce costs at volume, but synthetic estimates cannot establish its accuracy or latency. Multilingual encoder-decoder sharing can transfer information across language pairs; it does not prove that Google Translate uses an mT5 checkpoint or that it dominates every low-resource pair.

Version drift changes tokenizer behavior, supported heads, default generation parameters and model quality. Store the full tokenizer/checkpoint identity, task head, label map, pooling and output contract. Reject a missing classifier head instead of treating randomly initialized task weights as pretrained evidence. Measure quality and latency under real padding and batching; a one-pass decoder classification path is not the same cost as generating a paragraph.

Where this shows up next in the plan

Session 113 samples from next-token distributions, including an encoder-decoder's target distribution. Session 114's language-model scaling fits are not automatically encoder fits. Session 117 can use a purpose-trained encoder or decoder embedding model. Fine-tuning and LoRA apply to both families; reranking is a task interface, not an exclusively BERT capability.

(e) Recall + stretch · 10 min

1. Masks? Encoder S×S full valid-key visibility; decoder T×T causal; encoder-decoder adds T×S source cross-attention. Padding remains masked.
2. Can BERT generate? It has an MLM vocabulary head in the task-specific class and can fill blanks. Ordinary BERT pretraining is not causal next-token training; generation needs an explicit compatible procedure, not just a new loader name.
3. GPT embeddings worse? Early states see prefixes, but the last sees the complete input. Pooling, training and evaluation decide embedding quality; bidirectionality is not a universal verdict.
4. 50K labeled tickets? Benchmark a small supervised encoder first, with rare-class metrics and cost accounting; compare alternatives on the same holdout rather than assume a winner.
5. Span corruption? Replace spans by distinct sentinels; generate sentinel-delimited removed spans with a shifted decoder input. BERT predicts selected positions in place.
6. Forward versus generate? Forward returns states/logits for supplied tokens. Generate repeatedly calls the model with a decoding policy. Both GPT and T5 expose logits and generated IDs through different methods.
7. Encoder-decoder use case? Source-to-target translation or rewriting is a natural candidate. A 2026 dominance claim needs a named benchmark and deployment, not the family label.
8. Auto classes? Architecture-dispatched base versus compatible causal-LM or seq2seq task classes; not arbitrary architecture conversion. Check head initialization warnings and configuration.
9. Modern encoders? Longer context and efficient kernels can improve their operating range. Neither a revival narrative nor a model announcement proves an unsourced 1/50 cost advantage.

Explain-out-loud test

Draw the rectangular cross-attention matrix; compute the expected 12/1.5/1.5 corruption percentages; explain why output shape depends on the head; and justify a family choice without a guessed speedup.

What comes next

Sampling turns vocabulary scores into decisions. Learning-plan hub.

Primary references

1. Devlin et al., BERT, NAACL 2019, §3.1.
2. Raffel et al., T5, JMLR 2020.
3. Transformers v4.57.1 BERT classes and auto-model mappings.