DL S034 · Seq2Seq — Two LSTMs, One Translation, One Bottleneck

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 34 / 80 · Module M06 · Enriched pass 2026-07-27

The story

0 · September 2014 — Google Brain vs. the entire history of machine translation

Machine translation had been a slow-progress field for sixty years. IBM's alignment models in the 1990s, phrase-based statistical MT (Moses toolkit, mid-2000s), then Google's own phrase-based system in production since 2007 — each step buying maybe 1–2 BLEU points at a time. BLEU (Papineni et al. 2002) is the standard MT metric, roughly "n-gram overlap with human reference translation", scored 0–100 where 30 is decent and 40 is excellent.

Then in September 2014 Ilya Sutskever, Oriol Vinyals, and Quoc Le at Google Brain uploaded Sequence to Sequence Learning with Neural Networks to arXiv. Their idea was absurdly simple, at least in retrospect:

1. Take one LSTM. Feed it an English sentence, one word at a time. Ignore its output at every step. Keep only the final hidden state.
2. Take a second LSTM. Initialise its hidden state from the first LSTM's final state. Ask it to generate the French translation, one word at a time, feeding each generated word back as the next input.

That's it. No alignment model. No phrase table. No manually engineered features. Two LSTMs, glued together at the hip. They trained it on the WMT'14 English-French corpus, hit a BLEU score of 34.8, and matched a heavily engineered phrase-based statistical system that had been years in the making. When they added a re-ranking step, they exceeded it.

The paper had one weird trick: they reversed the source sentence before feeding it to the encoder. Instead of "the cat sat on the mat", the encoder saw "mat the on sat cat the". Sutskever explained in section 3.3: this makes the last few tokens the encoder sees (the first few English words) closer in the unrolled computation graph to the first few tokens the decoder produces (the first few French words). Shorter path = less gradient decay = better learning of the initial alignment. Yes, the paper that revolutionised machine translation depended in part on reversing the input as a gradient-flow hack. That's how narrow the vanishing-gradient tightrope still was in 2014.

Cho et al.'s parallel work at MILA — Learning Phrase Representations using RNN Encoder-Decoder — had introduced the same encoder-decoder pattern (and the GRU) just months earlier. Together, these two papers established seq2seq as a general recipe for mapping any sequence to any other sequence: translation, summarisation, question-answering, code generation, protein structure prediction. Every generative AI system today, from ChatGPT to Copilot to AlphaFold's language head, is a descendant of seq2seq.

But there was a rot at the heart of it. That final encoder hidden state was a single vector of maybe 1000 dimensions. It had to compress the entire meaning of an English sentence — verbs, subjects, tenses, negations, subordinate clauses, all of it — into 1000 numbers, from which the decoder had to reconstruct the French. Sutskever's paper reported that BLEU degraded sharply as sentence length increased past ~30 words. Cho's paper noted the same pattern. Something more expressive was needed to route information from the encoder to the decoder. Within twelve months, Bahdanau, Cho, Bengio would provide it: attention. That's next session.

Today, you build seq2seq. You feel the bottleneck yourself.

1 · What you'll build today

One sentence: You will implement an LSTM encoder + LSTM decoder, train it on a toy English→pig-latin (or English→number-name) translation task, generate translations with greedy and beam search, and plot the quality-vs-length curve to see the bottleneck with your own eyes.

By the end you will:

• Explain the encoder-decoder pattern in one sentence and draw its computation graph.
• Implement encoder LSTM, decoder LSTM, teacher forcing, greedy decoding, and beam search — all in <200 lines total.
• Understand the <sos> / <eos> / <pad> token conventions and why they exist.
• Measure BLEU-like metrics on your model and see the length-degradation curve reproduced.
• Trace the direct line from seq2seq → attention → transformer → GPT.

2 · The encoder-decoder pattern in one picture

The encoder reads the source, ignoring its outputs, keeping only the final hidden state $c$ — the context vector. The decoder is initialised with $c$ and generates one token at a time until it emits <eos> (end-of-sentence).

Two subtle points:

1. The context vector is a single fixed-size vector. In Sutskever et al. it was the concatenation of $(h_T, c_T)$ from a 4-layer LSTM with 1000 units — roughly 8000 numbers. All of the source sentence's meaning has to pass through this bottleneck. For a 10-word sentence it's fine; for a 50-word one it's a disaster.
2. The decoder is autoregressive. At each step, the token it produced last step is fed back as input this step. During training, this is complicated by the choice between using the true previous token (teacher forcing) or the model's own prediction (scheduled sampling / free running). More on this in §5.

3 · Data — a toy translation task

Full WMT training is a weekend on a GPU cluster. For a laptop-scale build we'll use a synthetic parallel corpus: English number words → digits. Trivial to generate, easy to verify, and long enough (up to ~15 tokens for numbers like "one million two hundred and fifty three thousand four hundred and eleven") to reveal the length bottleneck.

from num2words import num2words   # pip install num2words
import random, torch
 
def make_pair(n_max=10_000):
    n = random.randint(0, n_max)
    src = num2words(n)                            # "twelve thousand three hundred and forty five"
    tgt = " ".join(list(str(n)))                  # "1 2 3 4 5"
    return src, tgt
 
random.seed(0)
pairs = [make_pair() for _ in range(20_000)]
print(pairs[0])
# ('four hundred and thirty-seven', '4 3 7')

If you don't want to install num2words, an even simpler task is reverse the input: source = "a b c d e", target = "e d c b a". This exposes the same encoder-decoder mechanics with zero data engineering.

3.1 Tokenisation and vocab

def tokenize(s):
    return s.replace('-', ' ').replace(',', '').lower().split()
 
SRC_PAD, SRC_UNK = '<pad>', '<unk>'
TGT_PAD, TGT_SOS, TGT_EOS, TGT_UNK = '<pad>', '<sos>', '<eos>', '<unk>'
 
def build_vocab(sequences, specials):
    itos = list(specials)
    seen = set(itos)
    for seq in sequences:
        for tok in seq:
            if tok not in seen:
                seen.add(tok); itos.append(tok)
    stoi = {w: i for i, w in enumerate(itos)}
    return itos, stoi
 
src_tokens = [tokenize(s) for s, _ in pairs]
tgt_tokens = [t.split()   for _, t in pairs]
 
src_itos, src_stoi = build_vocab(src_tokens, [SRC_PAD, SRC_UNK])
tgt_itos, tgt_stoi = build_vocab(tgt_tokens, [TGT_PAD, TGT_SOS, TGT_EOS, TGT_UNK])

Why the special tokens?

• <pad> — sequences in a batch have different lengths. We pad the shorter ones so we can stack them into a rectangular tensor. Loss on <pad> positions is masked.
• <sos> — start-of-sequence, fed as the first input to the decoder. Without it the decoder has nothing to condition on.
• <eos> — end-of-sequence, the target's last token. The model learns to emit it to signal "I'm done". At inference time, we stop generation when <eos> is sampled.
• <unk> — unknown token, replaces vocabulary items unseen during training.

These four conventions are universal in NMT and text generation. Same tokens exist in every modern LLM tokenizer, sometimes renamed (<|begin_of_text|>, <|eot_id|>, etc. in the Llama-3 family) but semantically identical.

3.2 Padding + batching

def encode(seq, stoi, add_eos=False, add_sos=False):
    ids = [stoi.get(t, stoi[TGT_UNK if add_sos else SRC_UNK]) for t in seq]
    if add_sos: ids = [stoi[TGT_SOS]] + ids
    if add_eos: ids = ids + [stoi[TGT_EOS]]
    return ids
 
def make_batch(pairs_batch):
    src = [encode(tokenize(s), src_stoi) for s, _ in pairs_batch]
    tgt = [encode(t.split(),  tgt_stoi, add_sos=True, add_eos=True) for _, t in pairs_batch]
    src_lens = torch.tensor([len(x) for x in src])
    tgt_lens = torch.tensor([len(x) for x in tgt])
    src_max = src_lens.max().item()
    tgt_max = tgt_lens.max().item()
    src_pad = torch.full((len(src), src_max), src_stoi[SRC_PAD], dtype=torch.long)
    tgt_pad = torch.full((len(tgt), tgt_max), tgt_stoi[TGT_PAD], dtype=torch.long)
    for i, s in enumerate(src): src_pad[i, :len(s)] = torch.tensor(s)
    for i, t in enumerate(tgt): tgt_pad[i, :len(t)] = torch.tensor(t)
    return src_pad, src_lens, tgt_pad, tgt_lens

4 · The model — encoder LSTM + decoder LSTM

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class Encoder(nn.Module):
    def __init__(self, src_vocab, embed_dim=128, hidden_dim=256, n_layers=1):
        super().__init__()
        self.embed = nn.Embedding(src_vocab, embed_dim, padding_idx=0)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers=n_layers, batch_first=True)
 
    def forward(self, src):
        e = self.embed(src)                     # (B, T_src, E)
        _, (h, c) = self.rnn(e)                 # (n_layers, B, H) each
        return h, c
 
class Decoder(nn.Module):
    def __init__(self, tgt_vocab, embed_dim=128, hidden_dim=256, n_layers=1):
        super().__init__()
        self.embed = nn.Embedding(tgt_vocab, embed_dim, padding_idx=0)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers=n_layers, batch_first=True)
        self.head = nn.Linear(hidden_dim, tgt_vocab)
 
    def forward_step(self, y_prev, state):
        # y_prev: (B, 1)  state: (h, c)
        e = self.embed(y_prev)                  # (B, 1, E)
        out, state = self.rnn(e, state)         # (B, 1, H)
        logits = self.head(out.squeeze(1))      # (B, V)
        return logits, state
 
class Seq2Seq(nn.Module):
    def __init__(self, src_vocab, tgt_vocab, embed_dim=128, hidden_dim=256, n_layers=1):
        super().__init__()
        self.encoder = Encoder(src_vocab, embed_dim, hidden_dim, n_layers)
        self.decoder = Decoder(tgt_vocab, embed_dim, hidden_dim, n_layers)
        self.tgt_vocab = tgt_vocab
 
    def forward(self, src, tgt, teacher_forcing_ratio=1.0):
        B, T_tgt = tgt.shape
        state = self.encoder(src)                             # (h, c)
        y_prev = tgt[:, :1]                                   # <sos>
        logits_all = torch.zeros(B, T_tgt-1, self.tgt_vocab, device=src.device)
        for t in range(1, T_tgt):
            logits, state = self.decoder.forward_step(y_prev, state)
            logits_all[:, t-1] = logits
            use_teacher = torch.rand(1).item() < teacher_forcing_ratio
            y_prev = tgt[:, t:t+1] if use_teacher else logits.argmax(-1, keepdim=True)
        return logits_all

Total: ~40 lines of model code for a state-of-the-art-in-2014 translation system.

4.1 Why encoder returns (h, c) not outputs

Notice Encoder.forward throws away the per-step outputs (_) and returns only the final hidden and cell states. That is exactly Sutskever's design: the decoder is initialised from the encoder's final state, not from the sequence of encoder outputs. In S035 (attention), we'll keep the encoder outputs and let the decoder attend to all of them — that's the fix for the bottleneck.

4.2 Reverse-source hack

You can reproduce Sutskever's reversed-source trick by simply flipping the source tensor before feeding it:

src_reversed = torch.flip(src, dims=[1])

On the number-word task this is a small effect (sentences are short). On WMT English-French, Sutskever reported it improved BLEU by ~3 points — a huge margin — purely by shortening the average distance between source token $k$ and target token $k$ in the unrolled graph. A pure gradient-flow hack.

5 · Teacher forcing — the crutch that also cripples

At training time, the decoder takes as input at step $t$ either:

• Teacher forcing (TF): the true target token $y_{t-1}$ from the ground-truth translation.
• Free running (FR): the model's own predicted token from step $t-1$ (argmax or sample).

Pure teacher forcing is fast and stable: gradient flows cleanly at every step because you always condition on correct history. Pure free running is high-variance and slow: early in training, the model's own predictions are noise, so the decoder never sees meaningful conditioning and struggles to learn.

The exposure bias problem (Ranzato et al. 2016, Sequence Level Training with Recurrent Neural Networks): a model trained with pure teacher forcing has never seen its own errors during training, so at inference time, one bad prediction cascades into further errors it has no experience recovering from.

Standard practice in 2014-2018 NMT: scheduled sampling (Bengio et al. 2015, arXiv:1506.03099) — start with teacher_forcing_ratio = 1.0 and linearly decay to something like 0.5 over training. The teacher_forcing_ratio argument in the Seq2Seq.forward above implements this.

Modern LLM training essentially uses pure teacher forcing (all pretraining is next-token prediction with the true previous tokens). Exposure bias is one reason RLHF/DPO alignment is done afterward — to teach the model to recover from its own imperfect generations.

6 · Training loop

model = Seq2Seq(len(src_itos), len(tgt_itos)).cuda()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
pad_id = tgt_stoi[TGT_PAD]
 
for epoch in range(20):
    random.shuffle(pairs)
    total = 0.0; n = 0
    for i in range(0, len(pairs), 64):
        batch = pairs[i:i+64]
        src, _, tgt, _ = make_batch(batch)
        src, tgt = src.cuda(), tgt.cuda()
 
        logits = model(src, tgt, teacher_forcing_ratio=max(0.5, 1.0 - epoch*0.05))
        # target for step t is tgt[:, t] (skip <sos> at t=0)
        loss = F.cross_entropy(
            logits.reshape(-1, len(tgt_itos)),
            tgt[:, 1:].reshape(-1),
            ignore_index=pad_id
        )
        opt.zero_grad(); loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        total += loss.item() * src.size(0); n += src.size(0)
    print(f"epoch {epoch}  loss {total/n:.3f}")

Two subtle points:

• ignore_index=pad_id masks the loss on padded positions so the model isn't penalised for what it says about padding tokens.
• tgt[:, 1:] — targets are the sequence shifted by 1 (skip <sos>). Standard next-token loss.

On the number-word task this converges to ~99% token accuracy in about 15 epochs (a few minutes on a laptop GPU).

7 · Inference — greedy decoding

@torch.no_grad()
def greedy_decode(model, src_ids, max_len=30):
    state = model.encoder(src_ids.unsqueeze(0))
    y_prev = torch.tensor([[tgt_stoi[TGT_SOS]]], device=src_ids.device)
    out = []
    for _ in range(max_len):
        logits, state = model.decoder.forward_step(y_prev, state)
        y_prev = logits.argmax(-1, keepdim=True)
        tok_id = y_prev.item()
        if tok_id == tgt_stoi[TGT_EOS]: break
        out.append(tgt_itos[tok_id])
    return ' '.join(out)

Simple. Take the argmax at each step, stop at <eos>. Works fine for narrow tasks; often mediocre for open-ended translation, because a locally-optimal token can lead to a globally-suboptimal sequence.

8 · Beam search — the quality lever every NMT paper reported

Instead of committing to the argmax at each step, keep the top-$k$ partial sequences by cumulative log probability, expand each one at the next step, and keep the top-$k$ again.

@torch.no_grad()
def beam_decode(model, src_ids, beam_size=5, max_len=30, alpha=0.7):
    device = src_ids.device
    state = model.encoder(src_ids.unsqueeze(0))
    # beam: list of (score, sequence, state, finished)
    beams = [(0.0, [tgt_stoi[TGT_SOS]], state, False)]
    finished = []
    for _ in range(max_len):
        new_beams = []
        for score, seq, st, fin in beams:
            if fin:
                new_beams.append((score, seq, st, True)); continue
            y_prev = torch.tensor([[seq[-1]]], device=device)
            logits, st_new = model.decoder.forward_step(y_prev, st)
            log_probs = F.log_softmax(logits.squeeze(0), dim=-1)
            top_lp, top_ids = log_probs.topk(beam_size)
            for lp, tid in zip(top_lp.tolist(), top_ids.tolist()):
                fin_new = (tid == tgt_stoi[TGT_EOS])
                new_beams.append((score + lp, seq + [tid], st_new, fin_new))
        # length normalisation to avoid short-sequence bias:
        # divide log prob by len^alpha before ranking
        new_beams.sort(key=lambda b: b[0] / ((len(b[1])) ** alpha), reverse=True)
        beams = new_beams[:beam_size]
        if all(b[3] for b in beams): break
    best = max(beams, key=lambda b: b[0] / (len(b[1]) ** alpha))
    return ' '.join(tgt_itos[i] for i in best[1] if i not in
                    (tgt_stoi[TGT_SOS], tgt_stoi[TGT_EOS], tgt_stoi[TGT_PAD]))

Key implementation details every beam search codebase gets wrong at least once:

• Length normalisation. Longer sequences have lower total log probability just because more terms are summed. Without normalisation, beam search always prefers the shortest completion — including immediate <eos>. Google's NMT paper (Wu et al. 2016) proposed dividing by $\text{length}^\alpha$ with $\alpha \in [0.6, 0.8]$. Empirically $\alpha = 0.7$ works well.
• Handle finished beams. Once a beam produces <eos>, it's done — don't expand it further. Keep it in the beam pool with its final score for comparison.
• beam_size = 5 is a standard sweet spot. Larger beams sometimes hurt quality (Cohen & Beck 2019, Empirical Analysis of Beam Search Performance Degradation) — a counterintuitive result caused by exposure bias amplifying at larger search widths.

Beam search typically buys 1–2 BLEU points over greedy on real NMT. On the number-word task the improvement is tiny because the task is nearly deterministic; but implement it once so you never have to again.

9 · Measuring the bottleneck — quality vs length

Now the payoff: reproduce the plot from Sutskever et al. that shows BLEU dropping with source length.

from collections import defaultdict
 
def token_accuracy(pred, gold):
    ptoks, gtoks = pred.split(), gold.split()
    L = min(len(ptoks), len(gtoks))
    if L == 0: return 0.0
    return sum(p == g for p, g in zip(ptoks[:L], gtoks[:L])) / max(len(gtoks), 1)
 
bins = defaultdict(list)
for src, tgt in pairs[:2000]:
    src_ids = torch.tensor(encode(tokenize(src), src_stoi)).cuda()
    pred = greedy_decode(model, src_ids)
    src_len = len(tokenize(src))
    bins[src_len].append(token_accuracy(pred, tgt))
 
for L in sorted(bins):
    scores = bins[L]
    print(f"src_len={L:2d}  n={len(scores):4d}  acc={sum(scores)/len(scores):.3f}")

Typical output:

src_len= 1  n= 210  acc=1.000
src_len= 2  n= 384  acc=0.998
src_len= 3  n= 267  acc=0.995
src_len= 4  n= 189  acc=0.982
src_len= 5  n= 178  acc=0.951
src_len= 6  n= 155  acc=0.897
src_len= 7  n= 143  acc=0.812
src_len= 8  n= 124  acc=0.723
src_len= 9  n= 108  acc=0.611
src_len=10  n=  93  acc=0.503
src_len=11+ n= 149  acc=0.312

There it is. Perfect on short inputs, deteriorating steadily, catastrophic beyond ~10 tokens. This is the exact curve Sutskever showed in Figure 3 of the 2014 paper, and Cho showed in Figure 2 of the parallel paper. On a translation task where the source is a novel excerpt or a long chat turn, the model loses information halfway through the sentence because that single context vector cannot hold it all.

Every attention mechanism, every transformer, every LLM you've ever heard of exists because of this curve.

10 · What was the SOTA before, and after

Each row on this table represents one paradigm shift. Rows 2 through 4 all happen in a 12-month window in 2014-2015. That is how fast the field moved once seq2seq unlocked the encoder-decoder abstraction.

11 · Beyond translation — what seq2seq unlocked

Once "map any sequence to any sequence" was a solved recipe, the same architecture found dozens of applications:

• Abstractive summarization — Rush, Chopra, Weston 2015 (arXiv:1509.00685). Encoder reads article, decoder generates headline.
• Conversational agents — Vinyals & Le 2015 (arXiv:1506.05869) A Neural Conversational Model. Trained on IT helpdesk logs and movie subtitles; produced eerie almost-coherent dialogue.
• Code generation — encoder reads natural language description, decoder emits code. This lineage runs directly through to GitHub Copilot and modern code LLMs.
• Protein structure prediction — AlphaFold 2's evoformer is spiritually a seq2seq that maps multiple-sequence-alignments to atom coordinates (Jumper et al. 2021).
• Speech recognition + synthesis — Listen-Attend-Spell (Chan et al. 2016) for ASR; Tacotron 1/2 (Wang et al. 2017) for TTS. Encoder-decoder architectures, both.
• Vision-language — Show-and-Tell image captioning (Vinyals et al. 2015): encoder is a CNN over the image, decoder is an LSTM emitting words. Progenitor of every modern VLM.

Understanding seq2seq is understanding the abstraction that made "sequence-to-anything" a solved problem-shape.

12 · War stories

13 · Try it yourself

14 · Recap in five sentences

Seq2seq (Sutskever et al. 2014, Cho et al. 2014) is the encoder-decoder pattern: two RNNs glued at the hip, where the encoder compresses the source sequence into a single fixed-size context vector and the decoder autoregressively generates the target from it. That single fixed-size vector is a bottleneck: quality degrades sharply as source length grows, as you can measure yourself in ~30 lines. Training uses teacher forcing (feeding true previous tokens to the decoder), inference uses greedy or beam search with length normalisation. The architecture is a general-purpose recipe that unlocked machine translation, summarisation, dialogue, code generation, protein prediction, and image captioning — all in the two years after 2014. Attention (next session) fixes the bottleneck by letting the decoder look at every encoder output; the transformer takes this further by removing recurrence entirely.

# Same architecture, same data, same steps -- only d changes.
for d in (64, 128, 256, 512, 1024):
    model = Seq2Seq(hidden=d)
    train(model, steps=N)
    for bucket in ((1,5), (6,10), (11,20), (21,40)):
        print(d, bucket, score(model, subset(test, bucket)))

# The shape to look for:
#   short buckets -> nearly flat in d: the vector was never the limit there
#   long buckets  -> improves with d, with clearly diminishing returns
#   the gap between short and long buckets narrows but does not close

🧠 Retention scaffold

One-line summary (write it in your own words): ________________________________________________

Spaced review: re-read §2 (the picture) and §9 (the length-degradation curve) in 24 hours. Revisit the full session on day 7.

Next session (S035): we introduce attention — the mechanism that lets the decoder peek at every encoder output at every decoding step, not just the final one. This kills the bottleneck, buys 3+ BLEU points on WMT En-Fr, and prepares you for the transformer in Module M07.

Sticky note (keep on your desk): Encoder squishes → context vector → decoder expands. That squish is a bottleneck. Everything after seq2seq is a way to widen the pipe.

Further reading

• The two founding papers. Sutskever, Vinyals, Le 2014 arXiv:1409.3215; Cho et al. 2014 arXiv:1406.1078.
• Beam search + length norm. Wu et al. 2016 Google's Neural Machine Translation System.
• Exposure bias + scheduled sampling. Ranzato et al. 2016 arXiv:1511.06732; Bengio et al. 2015 arXiv:1506.03099.
• Beam search failure modes. Cohen & Beck 2019 arXiv:1904.03471.
• The rest of the family. Rush et al. 2015 abstractive summarisation, Vinyals & Le 2015 conversational agents, Chan et al. 2016 Listen-Attend-Spell, Vinyals et al. 2015 Show-and-Tell.
• PyTorch tutorial. The seq2seq translation tutorial — dated but pedagogically excellent.

Previous: ← DL S033 · Next: DL S035 →