Seq2Seq — A Complete Length-Aware Encoder–Decoder
A sequence-to-sequence model defines a conditional distribution over a target sequence given a source. In the fixed-state recurrent version, an encoder reads the source and passes its final state to an autoregressive decoder. For an LSTM the interfac
Original publication date unrecordedUpdated
Lesson preparation & details
Level: intermediate
By the end, you should be able to
- Train a complete length-aware encoder-decoder and compare decoding policies
- Evaluate held-out length buckets without inventing quality curves
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 9 sections
What the fixed interface does and does not imply
A sequence-to-sequence model defines a conditional distribution over a target sequence given a source. In the fixed-state recurrent version, an encoder reads the source and passes its final state to an autoregressive decoder. For the unidirectional, non-projected LSTM used here, the interface is both hidden and cell states, for every layer: each has shape (layers,batch,hidden). It is fixed-size as source length varies, not literally one scalar or only the last hidden output.
Sutskever, Vinyals and Le's 2014 paper uses deep LSTMs and source reversal. Its reported 34.8 BLEU is a particular ensemble/system setting, not a universal score for the little class below. The paper explicitly reports that the reversed-source LSTM did not suffer on long sentences in its experiment. The old lesson attributed a fabricated catastrophic length curve to that paper. Bahdanau et al.'s arXiv v1 submission history records September 1, 2014 at 16:33:02 UTC, before Sutskever et al.'s September 10 submission—not a response written a month later. We can motivate a finite-state bottleneck without inventing that history.
BLEU is clipped n-gram precision with a brevity penalty and a geometric aggregate; corpus, references, tokenization and scoring implementation matter. “30 is decent, 40 excellent” across tasks is not a valid interpretation. This lesson measures exact match and explicitly defined aligned-token accuracy on synthetic transformations. Neither is BLEU or a WMT reproduction.
You will build tokenization, length-aware batching, encoder/decoder, correctly shifted training, greedy/beam generation and held-out length evaluation. Prerequisites are LSTM tuple states, embeddings, stable cross-entropy and the difference between padding positions and actual tokens.
Computation graph and capacity
Figure text equivalent — labels and relationships
- Node “Source tokens and lengths” (square).
- Node “Packed encoder LSTM” (square).
- Node “Final h and c for each layer” (square).
- Node “Decoder recurrent state” (square).
- Node “BOS then previous target token” (square).
- Node “Next-token logits” (square).
- Node “Emit token or EOS” (square).
- “Source tokens and lengths” leads to “Packed encoder LSTM”. Line: normal; ends: arrow point.
- “Packed encoder LSTM” leads to “Final h and c for each layer”. Line: normal; ends: arrow point.
- “Final h and c for each layer” leads to “Decoder recurrent state”. Line: normal; ends: arrow point.
- “BOS then previous target token” leads to “Decoder recurrent state”. Line: normal; ends: arrow point.
- “Decoder recurrent state” leads to “Next-token logits”. Line: normal; ends: arrow point.
- “Next-token logits” leads to “Emit token or EOS”. Line: normal; ends: arrow point.
- “Emit token or EOS” leads to “BOS then previous target token”. Line: normal; ends: arrow point.
The decoder models p(y given x) as a product of next-token probabilities conditioned on previous targets and the source state. Encoder and decoder parameters need not be shared. Training uses gold previous targets; inference uses generated tokens. Source and target lengths can differ.
The funnel analogy identifies a restricted interface: more source tokens do not create more stored state slots. Finite precision, model width, optimization, recurrence and training distribution all affect information retention. It is not an information-theoretic proof that a 50-token sequence can never fit into a real-valued vector, nor that increasing width must leave a residual error at every width. A reversal experiment changes computational paths, not the amount of information in the input. Attention replaces the single shared interface with query-dependent access to all encoder outputs, but each retrieved context is still a weighted summary.
Dataset, vocabulary and valid-length batching
The following complete module is shared with the attention lesson. Save all Python fences here in order as seq2seq_lab.py. Importing it defines functions without starting training. It uses only standard Python and PyTorch. It contains a deterministic reversal task and number words to digit sequence task without the old num2words installation. The latter intentionally covers integers 0–999, not the misleading million-word example under a 10,000 limit.
PAD has no target loss, BOS starts decoding, EOS ends it, and UNK represents held-out vocabulary items. These are this tokenizer's conventions, not a claim every LLM tokenizer has four interchangeable special tokens. Build vocabulary from training text, serialize its order and report unknown-token rates. A target containing UNK can have low token loss while losing task information.
import math, random, copy, json
from pathlib import Path
from collections import defaultdict
import torch
from torch import nn
from torch.nn import functional as F
from torch.nn.utils.rnn import pack_padded_sequence
PAD, BOS, EOS, UNK = 0,1,2,3
SPECIALS=['<pad>','<bos>','<eos>','<unk>']
def tokenize(text):
return text.lower().replace('-',' ').replace(',','').split()
def number_words(n):
if not 0 <= n < 1000:raise ValueError('number contract: 0..999')
units='zero one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen'.split()
tens='zero ten twenty thirty forty fifty sixty seventy eighty ninety'.split()
if n<20:return units[n]
if n<100:return tens[n//10]+(' '+units[n%10] if n%10 else '')
return units[n//100]+' hundred'+(' and '+number_words(n%100) if n%100 else '')
def reverse_pairs(count=80, low=2, high=6, seed=1):
if low<1 or high<low or count<1:raise ValueError('invalid task dimensions')
if count>sum(10**n for n in range(low,high+1)):raise ValueError('requested more distinct strings than exist')
rng=random.Random(seed);seen=set()
# All default requests are small relative to the finite task space.
while len(seen)<count:
seq=tuple(str(rng.randrange(10)) for _ in range(rng.randint(low,high)))
seen.add(seq)
return [(' '.join(s),' '.join(reversed(s))) for s in sorted(seen)]
def make_vocab(pairs):
tokens=sorted({t for pair in pairs for s in pair for t in tokenize(s)})
if any(t in SPECIALS for t in tokens):raise ValueError('reserved token in corpus')
itos=SPECIALS+tokens
return itos,{t:i for i,t in enumerate(itos)}
def make_batch(pairs, stoi, reverse_source=False):
if not pairs:raise ValueError('empty batch')
sources=[];targets=[]
for s,t in pairs:
source=[stoi.get(v,UNK) for v in tokenize(s)]
target=[stoi.get(v,UNK) for v in tokenize(t)]
if not source:raise ValueError('empty source')
sources.append(source[::-1] if reverse_source else source)
targets.append([BOS]+target+[EOS])
lengths=torch.tensor([len(s) for s in sources],dtype=torch.long)
x=torch.full((len(pairs),max(map(len,sources))),PAD,dtype=torch.long)
y=torch.full((len(pairs),max(map(len,targets))),PAD,dtype=torch.long)
for b,(s,t) in enumerate(zip(sources,targets)):
x[b,:len(s)]=torch.tensor(s);y[b,:len(t)]=torch.tensor(t)
return x,lengths,y
class Encoder(nn.Module):
def __init__(self,vocab,emb=16,hidden=24,layers=1):
super().__init__();self.embed=nn.Embedding(vocab,emb,padding_idx=PAD)
self.rnn=nn.LSTM(emb,hidden,num_layers=layers,batch_first=True)
def forward(self,src,lengths):
if (lengths<=0).any() or (lengths>src.size(1)).any():raise ValueError('bad source lengths')
packed=pack_padded_sequence(self.embed(src),lengths.cpu(),batch_first=True,enforce_sorted=False)
_,state=self.rnn(packed)
return state
class Decoder(nn.Module):
def __init__(self,vocab,emb=16,hidden=24,layers=1):
super().__init__();self.embed=nn.Embedding(vocab,emb,padding_idx=PAD)
self.rnn=nn.LSTM(emb,hidden,num_layers=layers,batch_first=True)
self.head=nn.Linear(hidden,vocab)
def step(self,previous,state):
out,state=self.rnn(self.embed(previous[:,None]),state)
return self.head(out[:,0]),state
class Seq2Seq(nn.Module):
def __init__(self,vocab,emb=16,hidden=24,layers=1):
super().__init__();self.encoder=Encoder(vocab,emb,hidden,layers)
self.decoder=Decoder(vocab,emb,hidden,layers)
def start(self,src,lengths):return self.encoder(src,lengths),None
def step(self,previous,state,memory):return self.decoder.step(previous,state)
def forward(self,src,lengths,target,teacher_ratio=1.,generator=None):
if not 0<=teacher_ratio<=1:raise ValueError('teacher ratio must be in [0,1]')
state,memory=self.start(src,lengths);previous=target[:,0];outputs=[]
for t in range(1,target.size(1)):
logits,state=self.step(previous,state,memory);outputs.append(logits)
use_gold=torch.rand(target.size(0),generator=generator)<teacher_ratio
previous=torch.where(use_gold,target[:,t],logits.argmax(-1))
return torch.stack(outputs,1)
def token_nll(logits,target):
gold=target[:,1:];valid=gold.ne(PAD).sum()
if valid==0:raise ValueError('all ignored targets')
total=F.cross_entropy(logits.reshape(-1,logits.size(-1)),gold.reshape(-1),ignore_index=PAD,reduction='sum')
return total/valid,int(valid)
assert number_words(437)=='four hundred and thirty seven'
assert number_words(0)=='zero' and number_words(999).endswith('ninety nine')Packing obtains the state after the last real token in this batched implementation; explicit masked recurrence or processing each unpadded sequence separately are alternatives. A zero padding embedding does not freeze recurrence: hidden weights and biases can still change the state. Merely gathering the final hidden output does not recover every layer's cell state. Reverse each source's valid tokens before padding; flipping a padded rectangle moves padding to the front. Targets must not be reversed when testing the original source-reversal intervention.
Teacher forcing and the complete training recipe
The gold sequence is BOS, token 1, token 2, EOS, PAD… . At decoder time 0, input BOS predicts token 1. The loss compares logits with target[:,1:] exactly once. PAD is ignored; EOS is not. Computing a mean of batch means weights short and long batches incorrectly. Evaluation below sums NLL times valid-token counts and divides once.
Teacher forcing is conditional maximum likelihood, not a defective crutch. Generation visits model-produced histories, so error accumulation is possible. Scheduled sampling mixes gold and predicted inputs; discrete selection has no ordinary derivative through the argmax/sample. It changes the objective and is not a universally consistent likelihood estimator. The critique by Huszár and the original scheduled-sampling paper should be read together. RLHF/DPO optimize preference-related objectives; neither is simply “the cure for teacher forcing.”
@torch.no_grad()
def evaluate_nll(model,pairs,stoi,batch_size=16,reverse_source=False):
if not pairs:raise ValueError('empty evaluation split')
mode=model.training;model.eval();total=0.;count=0
try:
for i in range(0,len(pairs),batch_size):
x,lens,y=make_batch(pairs[i:i+batch_size],stoi,reverse_source)
loss,n=token_nll(model(x,lens,y),y);total+=loss.item()*n;count+=n
return total/count
finally:model.train(mode)
def train_model(model,train,val,stoi,epochs=5,batch_size=16,lr=.01,teacher_ratio=1.,reverse_source=False):
if not train or not val or set(train)&set(val):raise ValueError('nonempty disjoint splits required')
opt=torch.optim.Adam(model.parameters(),lr=lr);rng=random.Random(4)
teacher_rng=torch.Generator().manual_seed(4);best=math.inf;best_state=None;history=[]
for epoch in range(epochs):
order=list(train);rng.shuffle(order);model.train();total=0.;tokens=0
for i in range(0,len(order),batch_size):
x,lens,y=make_batch(order[i:i+batch_size],stoi,reverse_source)
opt.zero_grad(set_to_none=True)
loss,n=token_nll(model(x,lens,y,teacher_ratio,teacher_rng),y)
if not torch.isfinite(loss):raise FloatingPointError('nonfinite loss')
loss.backward();nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True);opt.step()
total+=loss.item()*n;tokens+=n
vl=evaluate_nll(model,val,stoi,reverse_source=reverse_source)
history.append({'epoch':epoch,'train_nll':total/tokens,'val_nll':vl,'tokens':tokens})
if vl<best:best=vl;best_state=copy.deepcopy(model.state_dict())
if best_state is None:raise ValueError('at least one epoch required')
model.load_state_dict(best_state)
return historyTo train number words, generate all n in range(1000), pair number_words(n) with space-separated digits of n, then split unique n values with a recorded seed. For a local English–French corpus, use the explicit TSV loader below. Split by document/source identity before building vocabulary; reject overlaps and keep a final test split untouched while tuning width, teacher ratio, beam or LR. For a larger model, device transfer and GPU precision are additional engineering work—not silently used in this CPU implementation.
Greedy and beam decoding with declared length policy
Greedy chooses one next token at a time. Beam search retains several prefixes by a score; it approximates search under the model and does not repair incorrect probabilities. Scores below are cumulative log probabilities divided by generated_length**alpha, excluding BOS and including EOS. This is a declared teaching policy, not GNMT's exact length penalty, which uses ((5+length)/6)^alpha and also discusses coverage. Setting alpha=0 gives raw log probability. Raw scores often favor short outputs but do not always choose immediate EOS.
The implementation masks PAD/BOS before log-softmax, so its log_probability is the sum of log probabilities renormalized over allowed output tokens, not the unrestricted training softmax likelihood. For reference-versus-generated score comparisons, apply that same mask, EOS convention and length policy to both.
Finished hypotheses are never expanded. A length cap produces a clearly flagged truncation; it is not silently reported as a completed translation. PAD/BOS are forbidden output tokens. Larger beams can improve search and worsen task accuracy simultaneously. Comparing reference and generated scores under the same length policy can expose search error; a higher-scoring reference does not guarantee a finite beam will find it.
@torch.no_grad()
def decode_ids(model,src,lengths,beam=1,max_new=20,alpha=0.):
if src.size(0)!=1 or beam<1 or max_new<1 or alpha<0:raise ValueError('invalid search config')
mode=model.training;model.eval()
try:
state,memory=model.start(src,lengths)
beams=[(0.,[BOS],state,False)]
def score(item):return item[0]/max(1,len(item[1])-1)**alpha
for _ in range(max_new):
candidates=[]
for logp,ids,state,done in beams:
if done:candidates.append((logp,ids,state,done));continue
logits,new_state=model.step(torch.tensor([ids[-1]]),state,memory)
logits=logits.clone();logits[:,[PAD,BOS]]=-torch.inf
lp=logits.log_softmax(-1)[0]
values,indices=lp.topk(min(beam,lp.numel()-2))
for p,tok in zip(values.tolist(),indices.tolist()):
candidates.append((logp+p,ids+[tok],new_state,tok==EOS))
beams=sorted(candidates,key=score,reverse=True)[:beam]
if all(item[3] for item in beams):break
best=max(beams,key=score)
return {'ids':[i for i in best[1][1:] if i!=EOS],'ended':best[3],'score':score(best),'log_probability':best[0]}
finally:model.train(mode)
def aligned_accuracy(pred,gold):
denominator=max(len(pred),len(gold))
return sum(a==b for a,b in zip(pred,gold))/denominator if denominator else 1.
@torch.no_grad()
def evaluate_lengths(model,pairs,stoi,beam=1,reverse_source=False,max_new=30):
bins=defaultdict(list)
for pair in pairs:
x,lens,y=make_batch([pair],stoi,reverse_source)
result=decode_ids(model,x,lens,beam=beam,max_new=max_new)
gold=y[0,1:-1].tolist()
bins[int(lens[0])].append((float(result['ended'] and result['ids']==gold),aligned_accuracy(result['ids'],gold),float(not result['ended'])))
return {length:{'count':len(items),'exact':sum(x[0] for x in items)/len(items),'aligned':sum(x[1] for x in items)/len(items),'truncated':sum(x[2] for x in items)/len(items)} for length,items in sorted(bins.items())}
def load_local_tsv(path):
pairs=[]
for line in Path(path).read_text(encoding='utf8').splitlines():
columns=line.split('\t')
if len(columns)!=2 or not all(s.strip() for s in columns):raise ValueError('expected nonempty source<TAB>target')
pairs.append(tuple(columns))
if len(set(pairs))!=len(pairs):raise ValueError('duplicate pair: split/clean explicitly')
return pairs
if __name__=='__main__':
torch.manual_seed(34)
pairs=reverse_pairs();train,val,test=pairs[:56],pairs[56:68],pairs[68:]
itos,stoi=make_vocab(train);model=Seq2Seq(len(itos))
history=train_model(model,train,val,stoi,epochs=4)
# Padding-invariant final state: same valid source, wider padded rectangle.
x,lens,y=make_batch(train[:3],stoi)
a=model.encoder(x,lens);b=model.encoder(F.pad(x,(0,4),value=PAD),lens)
assert all(torch.allclose(u,v,atol=1e-6) for u,v in zip(a,b))
assert aligned_accuracy([4,5,6],[4,5])==2/3 # overlong outputs are penalized
assert y[0,0]==BOS and (y[0]==EOS).sum()==1
assert set(train).isdisjoint(val) and set(train).isdisjoint(test)
print('synthetic reversal only',history,evaluate_lengths(model,test,stoi))
print('beam probe',evaluate_lengths(model,test[:2],stoi,beam=3))Shared tensor states among sibling beams are safe here because decoder steps return new tensors without in-place state mutation. A cache implementation that mutates state would require copying or separate indexed storage. nn.LSTM requires the documented (h,c) tuple; it does not silently accept one tensor and invent the missing cell state.
Measuring width, length and reversal without inventing a curve
For the original practice, generate 20,000 distinct reversal examples of lengths 3–12, reserve disjoint validation/test inputs, and separately generate lengths 20,30,50. A larger synthetic generator should bound attempts or verify count feasibility. Compare hidden widths 64,128,256,512,1024 with matched token budget and multiple seeds. Use train_model and evaluate_lengths; record bin counts, exact accuracy, truncations and token NLL. Plot those returned values; no “typical output” is supplied as if measured.
Reversal of a monotonic source-target alignment redistributes path lengths: without reversal, source i to target i is roughly n steps; with reversal it is roughly 2i−1. The mean is similar, while early alignments become short. That is the original paper's optimization hypothesis, not proof that reversal helps every task. Reversing the input of the reversal task changes it into an identity-copy relation in encoder order, so calling it symmetric and predicting “not much” was wrong. Attention offers a direct read path; it does not prove source reversal can never affect training again.
The original width-only pseudocode omitted optimizer, task split and score definitions. The runnable functions here supply them. A remaining long-length gap can reflect extrapolation, decoding, insufficient training or capacity; it does not uniquely diagnose a dimensional bottleneck. Beam width 1 versus 5 is a measured quality/cost choice, not a guaranteed 1–2 BLEU gain or exact 5× runtime multiplier. Sampling is useful when multiple outputs are acceptable; temperature/top-k change the distribution, not the underlying model's learned likelihoods.
The following bounded runner makes the width/reversal/search exercise concrete without changing the shared module's existing helpers. It fits a fresh model for each width and source order, keeps one disjoint split fixed across conditions, and measures each search policy on the same test pairs. Equal epochs here mean equal training examples/token presentations, not equal FLOPs across widths. Use multiple seeds before drawing a quality conclusion; the default one-epoch fixture is a plumbing test.
def reversal_ablation(widths=(8,16), seeds=(7,), count=48, low=2, high=4,
epochs=1, evaluation_lengths=(6,), policies=((1,0.),(5,.7))):
rows=[]
for seed in seeds:
pairs=reverse_pairs(count,low,high,seed)
random.Random(seed).shuffle(pairs) # do not split lexicographically sorted strings
a,b=int(.7*count),int(.85*count)
train,val,test=pairs[:a],pairs[a:b],pairs[b:]
itos,stoi=make_vocab(train)
tests={'in_range':test}
for length in evaluation_lengths:
if low<=length<=high:raise ValueError('extrapolation lengths must be outside training range')
tests[f'length_{length}']=reverse_pairs(4,length,length,seed+1)
for width in widths:
for reversed_input in (False,True):
torch.manual_seed(seed)
model=Seq2Seq(len(itos),hidden=width)
history=train_model(model,train,val,stoi,epochs=epochs,
reverse_source=reversed_input)
for label,heldout in tests.items():
for beam,alpha in policies:
bins=defaultdict(list)
for pair in heldout:
x,lens,y=make_batch([pair],stoi,reversed_input)
result=decode_ids(model,x,lens,beam=beam,alpha=alpha,
max_new=int(lens[0])+2)
gold=y[0,1:-1].tolist()
bins[int(lens[0])].append((result['ended'] and result['ids']==gold,
aligned_accuracy(result['ids'],gold),not result['ended']))
metrics={n:{'count':len(v),'exact':sum(z[0] for z in v)/len(v),
'aligned':sum(z[1] for z in v)/len(v),
'truncated':sum(z[2] for z in v)/len(v)} for n,v in sorted(bins.items())}
rows.append(dict(seed=seed,width=width,reverse_source=reversed_input,
split=label,beam=beam,alpha=alpha,metrics=metrics,
train_presentations=sum(h['tokens'] for h in history),
val_nll=history[-1]['val_nll'],selected='best validation weights',
unknown_source_tokens=sum(stoi.get(t,UNK)==UNK for s,_ in heldout for t in tokenize(s))))
return rows
if __name__=='__main__':
ablation=reversal_ablation(widths=(8,),seeds=(7,),policies=((1,0.),(3,.7)))
assert len(ablation)==8
assert all(sum(v['count'] for v in r['metrics'].values())>0 for r in ablation)
print('bounded reversal ablation; no target quality promised',json.dumps(ablation))This small SVG exporter draws the returned observations for one named condition. It does not join interpolation and extrapolation as if they were a measured continuous curve. Counts remain in the JSON rows.
def length_curve_svg(rows,path):
from html import escape
parts=['<svg xmlns="http://www.w3.org/2000/svg" width="700" height="360">',
'<rect width="100%" height="100%" fill="white"/>',
'<text x="45" y="22">Synthetic reversal: exact match by source length</text>',
'<path d="M45 45 V275 H665" fill="none" stroke="black"/>']
maximum=max(n for r in rows for n in r['metrics'])
for fraction in (0.,.5,1.):
parts.append(f'<text x="10" y="{279-220*fraction}">{fraction:g}</text>')
parts.append(f'<text x="{45+600*fraction}" y="295">{maximum*fraction:g}</text>')
for i,row in enumerate(rows):
color=('blue','red','green','purple')[i%4]
points=' '.join(f"{45+600*n/maximum:.2f},{275-220*v['exact']:.2f}" for n,v in row['metrics'].items())
parts.append(f'<polyline points="{points}" stroke="{color}" fill="none"/>')
for n,v in row['metrics'].items():
parts.append(f'<circle cx="{45+600*n/maximum}" cy="{275-220*v["exact"]}" r="3" fill="{color}"/>')
label=f"{row['split']}; beam={row['beam']}; alpha={row['alpha']}"
parts.append(f'<text x="45" y="{315+15*i}" fill="{color}">{escape(label)}</text>')
Path(path).write_text(''.join(parts)+'</svg>')
if __name__=='__main__':
import os
if os.environ.get('LENGTH_FIGURE_PATH'):
length_curve_svg([r for r in ablation if not r['reverse_source'] and r['beam']==1],
os.environ['LENGTH_FIGURE_PATH'])Each metrics mapping is ready to plot with source length on the horizontal axis and exact, aligned or truncated on the vertical axis; retain the count for every point and separate interpolation from extrapolation. val_nll above is the last epoch's selection metric, while generation uses the best restored weights; do not mislabel it as a fresh NLL of that restored model. To evaluate number words, use all 1,000 unique integers, shuffle their pairs before splitting and reuse the same batching/training/search functions. No claimed 99% accuracy, length cliff or beam gain substitutes for these returned measurements.
Beyond translation and a fair historical comparison
Encoder-decoder is an architectural pattern, not proof every generative system is an LSTM seq2seq descendant. Rush et al.'s neural headline summarization uses an attention-based feed-forward encoder, while Vinyals and Le study a neural conversational sequence model. Listen, Attend and Spell combines an acoustic listener, attention and speller; Tacotron maps text to speech representations. Show and Tell uses a CNN image encoder with an LSTM caption decoder. These are different input encoders under a conditional-generation interface.
Code generation can use a source/target encoder-decoder or one causal stream. GPT-style decoder-only models concatenate prompt and continuation with a causal mask; they can support demonstrations in context without proving that encoder-decoder models cannot do so. T5 pretrains from unpaired text by making corrupted-input/reconstruction pairs. AlphaFold's Evoformer and geometric structure module are not a token-autoregressive seq2seq decoder or an “AlphaFold language head.” Diffusion with U-Nets is not created by swapping two LSTMs for two U-Nets.
Do not reconstruct the old mixed BLEU table. Phrase-based baselines, reranking, ensembles, vocabularies, years, language directions and tokenization differ. Compare one paper's rows with their conditions or rerun identical data/scoring. The useful historical lesson is the fixed-state interface, learned soft alignment and then the recurrence-free original Transformer—not an unsupported leaderboard ending in “45+.”
Practice and worked answers
- Draw the graph. Three source steps feed encoder (h,c); BOS starts four target steps plus EOS. Every target step receives its predecessor and state; only the fixed state crosses from encoder to decoder in this model. More source steps do not widen it.
- Special tokens. PAD fills rectangles and is ignored in loss, BOS supplies initial decoder input, EOS is learned and terminates generation, UNK handles unseen vocabulary. Their IDs are explicit and are not a universal tokenizer law.
- Teacher forcing. Feed the true previous target to optimize conditional likelihood. Inference histories differ; scheduled sampling is an alternative objective with caveats, not a mandatory fix, and preference alignment has a different purpose.
- Length normalization. This beam ranks log p divided by generated length to alpha; alpha=0 disables it. GNMT instead uses ((5+length)/6)^alpha. Neither is mathematically mandatory. Freeze completed beams, state conventions and token restrictions before comparing.
- Length curve. It is measured, not known in advance. Bucket held-out exact match, token accuracy and truncations by source length. The cited Sutskever reversed-source system did not exhibit the claimed catastrophic long-sentence collapse. Our finite-state interface remains a plausible limitation, not proof of a particular threshold.
- Decoder-only comparison. One causal stack can share computation across prompt and continuation and use one next-token objective. An encoder-decoder separately encodes the whole source bidirectionally and cross-attends during output. Both can learn conditional generation; compare source/target token losses, parameter count, token budget and latency rather than asserting universal superiority.
For recovery failures, inspect target shift before celebrating near-zero loss; packing before trusting final states; state tuples before switching GRU to LSTM; and search diversity before widening a beam. Revisit the graph tomorrow and run one controlled length or width ablation after a week.
Sources, optional resources and boundary
Primary: Sutskever et al., Cho et al., Bahdanau et al., BLEU, T5 span-corruption pretraining, GNMT length/coverage scoring, scheduled sampling, its critique, packed sequences. Optional original Serrano video, follow-up, and PyTorch tutorial are not substitute evidence for the claims.
Only bounded synthetic CPU reversal contracts were executed; this is not WMT, pig-Latin, full number-word accuracy or a pretrained translation benchmark. Local TSV training is a complete opt-in recipe, not an invoked data download. Optional previous/next reading: LSTM/GRU · attention.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.