S103 · RNNs & LSTMs — Sequences & the Vanishing Gradient

Module M12: Deep Learning · Session 103 of 130 · Track: ML 🌀

What you'll be able to do after this session

• Explain RNNs & LSTMs to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S100 — PyTorch Fundamentals — Tensors, Autograd, nn.Module

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Recurrent Neural Networks (RNNs) explained — StatQuest — Josh Starmer's clearest-possible RNN intro.
• 🎥 Deep dive (30–60 min) · Stanford CS224n — RNNs and LSTMs — Full lecture on the vanishing gradient and how LSTMs solve it.
• 🎥 Hands-on demo (10–20 min) · LSTM in PyTorch — from scratch — Aladdin Persson types out an LSTM cell live.
• 📖 Canonical article · Understanding LSTM Networks — Chris Olah — The single most-cited LSTM explainer ever written.
• 📖 Book chapter / tutorial · A Critical Review of RNNs for Sequence Learning (Lipton 2015) — Excellent survey with all the math and history.
• 📖 Engineering blog · Netflix — Deep learning for recommender systems (case study) — Sequence models in production for personalisation.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Sequences before transformers took over.

Everything you've learned about neural nets so far assumed inputs come in a fixed size (a 224×224 image, a 13-feature vector). But language, audio, stock prices, sensor data — the interesting stuff is sequential, and each item's meaning depends on what came before. A Recurrent Neural Network is the first neural architecture that natively handles this: it reads the sequence one element at a time and maintains a hidden state that carries information forward, like a running summary in your head as you read a sentence.

The catch is that "carrying information forward" turned out to be very hard. When you backpropagate a loss through 100 time steps, gradients get multiplied through 100 matrix products; unless those matrices are exquisitely well-conditioned, gradients either explode (weights blow up to NaN) or vanish (weights become tiny and the network stops learning long-range dependencies). This is the central problem in sequence modelling, and it's why plain RNNs work for length-10 sequences and fall apart at length-100.

The LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber, 1997) was the first clean fix. It splits state into a cell state (long-term memory that flows through mostly untouched) and a hidden state (short-term working memory), plus three learned "gates" — forget, input, output — that decide what to remember, what to write, and what to reveal at each step.

Gotcha to carry forever: RNNs are inherently sequential, so you cannot parallelise the time dimension across GPUs. This is exactly why transformers eventually replaced them for large-scale NLP — but LSTMs are still the right tool for small streaming problems (on-device speech, time-series forecasting, low-latency inference).

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Here's what happens inside an LSTM cell at time step t:

Reading it in English: at each step the LSTM (1) decides how much of the old memory to forget, (2) computes a candidate new memory from the input, (3) decides how much of that candidate to write into memory, (4) decides how much of the resulting memory to reveal as the hidden state.

Worked example — an RNN unrolled over time. Given a sequence x_1, x_2, x_3, initial hidden state h_0 = 0, and update rule h_t = tanh(W_h · h_prev + W_x · x_t + b):

• Step 1: h_1 = tanh(W_x · x_1 + b)
• Step 2: h_2 = tanh(W_h · h_1 + W_x · x_2 + b) — depends on x_1 through h_1
• Step 3: h_3 = tanh(W_h · h_2 + W_x · x_3 + b) — depends on x_1, x_2, x_3

To backprop the loss at step 3 to x_1, you must differentiate through tanh(W_h · tanh(W_h · tanh(...))). Each tanh has derivative ≤ 1 and each W_h multiplication can shrink or amplify — that product across many steps is what vanishes or explodes.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# lstm_names.py — classify short surnames by language of origin
import torch, torch.nn as nn, string, random
 
ALL = string.ascii_letters + " .,;'"
N_CHARS = len(ALL)
 
# Toy data — replace with data/names/*.txt from the PyTorch tutorial for real fun
data = {"English":  ["Smith", "Jones", "Taylor", "Brown"],
        "Italian":  ["Rossi", "Bianchi", "Ferrari", "Romano"],
        "Japanese": ["Sato", "Suzuki", "Takahashi", "Tanaka"]}
langs = list(data.keys())
 
def to_tensor(name):
    t = torch.zeros(len(name), 1, N_CHARS)
    for i, c in enumerate(name):
        t[i, 0, ALL.index(c)] = 1
    return t
 
class LangLSTM(nn.Module):
    def __init__(self, in_size, hidden, n_out):
        super().__init__()
        self.lstm = nn.LSTM(in_size, hidden)
        self.fc   = nn.Linear(hidden, n_out)
    def forward(self, x):
        out, _ = self.lstm(x)                  # out: [T, 1, H]
        return self.fc(out[-1])                # use LAST time step
 
model = LangLSTM(N_CHARS, 64, len(langs))
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
loss_fn = nn.CrossEntropyLoss()
 
for step in range(2000):
    lang = random.choice(langs); name = random.choice(data[lang])
    x, y = to_tensor(name), torch.tensor([langs.index(lang)])
    opt.zero_grad(); loss = loss_fn(model(x), y); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # important!
    opt.step()
    if step % 200 == 0:
        pred = langs[model(x).argmax(1).item()]
        print(f"step {step}: {name:12s} true={lang:8s} pred={pred:8s} loss={loss.item():.3f}")

Checklist while it runs:

1. After ~1000 steps predictions for the toy set should match nearly perfectly.
2. print(out.shape) inside forward — confirm it's [len(name), 1, 64].
3. Swap nn.LSTM for nn.RNN and observe accuracy drop on longer names.
4. Remove the clip_grad_norm_ line and train longer — occasionally loss spikes to NaN.
5. model.lstm.weight_ih_l0.shape is [4*hidden, input] — the 4 gates stacked.

Try this modification: Change nn.LSTM to nn.GRU. Often within 1% of LSTM on short sequences with fewer params.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

RNNs quietly power a lot of production systems even in the transformer era, because they have unbeatable properties for streaming and on-device inference: constant memory per step, no attention matrix to materialise, easy to compile to microcontrollers.

Speech recognition on Google Assistant and Apple Siri long relied on stacked LSTMs (or the closely related GRU) with CTC loss, and the on-device wake-word detector is still typically a tiny GRU running at under 1 mW. Time-series forecasting at Uber (their M4 competition entry) used LSTMs to predict ride demand five minutes ahead across every city. Recommendation systems like Netflix's session-based recommenders often use GRUs to summarise "what has this user watched in the last hour" into a vector for downstream ranking.

The classic bugs:

• Not clipping gradients. torch.nn.utils.clip_grad_norm_(params, max_norm=1.0) is essentially mandatory — otherwise an unlucky batch will send weights to NaN and you'll spend a day figuring out which step it happened on.
• Wrong batching for variable-length sequences. Padding all sequences to the longest wastes compute; the correct pattern is pack_padded_sequence / pad_packed_sequence. Skipping this can be 5× slower and can leak padding tokens into your loss.
• Truncated BPTT. For very long sequences you must chop them into windows and detach() the hidden state between windows, otherwise the compute graph grows unboundedly.
• Statefulness in production. If your model is trained "stateful" (hidden state carries across batches), you must maintain per-user hidden states in your serving layer — many teams have shipped a stateful model to a stateless serving stack and gotten mysteriously bad results.

Netflix's blog notes that even when they migrated to transformer-based recommenders, they kept an LSTM baseline in production for months because it was cheaper to serve at their QPS and gave a stable A/B comparison point.

(e) Quiz + exercise · 10 min

1. Explain in one sentence why plain RNNs fail on long sequences.
2. Name the three gates in an LSTM and what each one controls.
3. Why can't you parallelise an RNN along the time dimension?
4. What is gradient clipping and why is it essentially required for RNN training?
5. What is pack_padded_sequence and what performance problem does it solve?

Stretch (connects to S100 — PyTorch Fundamentals): In PyTorch, why does the LSTM output tensor have shape [T, B, H] by default rather than [B, T, H], and how do you switch it?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is RNNs & LSTMs? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S104): Embeddings — word2vec, GloVe, Contrastive Learning
• Previous (S102): CNNs — Convolution, Pooling, ImageNet Architectures
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.