DL S032 · RNN From Scratch — Modelling Sequences One Step at a Time

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 32 / 80 · Module M06 · Enriched pass 2026-07-27

The story

0 · Munich, 1991 — Sepp Hochreiter's diploma thesis

Technical University of Munich, spring 1991. Josef "Sepp" Hochreiter is a 24-year-old master's student, working on his diploma thesis with Jürgen Schmidhuber. His task: figure out why recurrent neural networks — a hot topic since Elman's 1990 paper Finding Structure in Time — refuse to learn dependencies more than a handful of timesteps apart.

Everyone in the field had noticed it. You could train an RNN to remember one or two words back. You could not train it to remember what happened five sentences ago, no matter how long you trained, no matter how much data you threw at it. The gradient signal from the loss just... evaporated as it flowed backward through time. Hochreiter opened his thesis by writing down the math: at each unroll step, the gradient is multiplied by the same recurrent weight matrix $W_{hh}$ (or more precisely, its Jacobian). Multiply by that matrix $T$ times and the gradient either shrinks to zero (if the eigenvalues are less than 1) or explodes to infinity (if they're greater than 1). There is no stable middle ground.

His 1991 thesis — Untersuchungen zu dynamischen neuronalen Netzen — was in German, so it took the English-speaking ML community a few years to catch on. But it named the disease that plagued every RNN of the 1990s: the vanishing gradient problem. Hochreiter and Schmidhuber's follow-up in 1997, Long Short-Term Memory, was their cure — a gated architecture designed explicitly so that gradients could flow across hundreds of timesteps without vanishing. That's next session.

Today, you build the RNN that gave Hochreiter the headache. You unroll it. You watch the gradient vanish with your own eyes. Because unless you feel why the plain RNN fails, LSTMs, attention, transformers, and every subsequent innovation will feel like arbitrary tricks. They are not arbitrary — they are answers to a specific pain, and you need to feel the pain first.

1 · What you'll build today

One sentence: You will implement a plain RNN cell in NumPy, train a character-level language model on tiny Shakespeare, watch it generate almost-English gibberish, then reproduce Hochreiter's vanishing-gradient discovery by measuring gradient norms across timesteps.

By the end you will:

• Write a 10-line RNN cell forward pass with pen and paper, correctly.
• Unroll an RNN over $T$ timesteps in PyTorch, keeping the computational graph.
• Implement backpropagation through time (BPTT) and its truncated cousin (TBPTT).
• Diagnose vanishing and exploding gradients from a single loss curve or gradient-norm plot.
• Explain gradient clipping (Pascanu, Mikolov, Bengio 2013) and why every RNN codebase uses it.
• See the specific mathematical structure that motivates the LSTM's gates — and be ready to appreciate Hochreiter's fix in S033.
• Connect the RNN → Transformer story: why we abandoned recurrence for attention in 2017, and why State Space Models (Mamba, Mamba-2, Jamba, RWKV) brought recurrence back in 2024–2025.

2 · The idea in one paragraph — keep a running summary

An MLP maps a fixed-size input to a fixed-size output in one shot. A convnet does the same over a fixed 2D grid. But a sequence — a sentence, an audio waveform, a stock-price history, a video — has variable length and its meaning depends on the order of its elements.

Recurrence gives you the simplest possible way to handle this: maintain a hidden state vector $h$, and at each timestep update it based on the previous $h$ and the current input $x_t$.

That's the entire architecture. Three weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$), two biases, one tanh. The state $h_t$ is a running summary of everything the network has seen from time 0 through time $t$.

The crucial architectural choice — the one that makes RNNs recurrent rather than merely deep — is weight sharing across time. The same $W_{xh}, W_{hh}, W_{hy}$ are used at every timestep. This is analogous to convolution's spatial weight sharing: convolutions assume the visual world is translation-equivariant (a cat is a cat regardless of pixel position), RNNs assume the temporal world is time-equivariant (the pattern "a noun follows the" is the same pattern whether it happens at word 5 or word 500).

3 · Forward pass in 10 lines of NumPy

Before touching PyTorch, do it once by hand. This is the single most important exercise in this session.

import numpy as np
 
def rnn_cell_forward(x_t, h_prev, Wxh, Whh, Why, bh, by):
    """One RNN step. x_t: (D,), h_prev: (H,). Returns h_t, y_t."""
    h_t = np.tanh(Wxh @ x_t + Whh @ h_prev + bh)   # (H,)
    y_t = Why @ h_t + by                            # (V,)
    return h_t, y_t
 
def rnn_forward(inputs, h_0, params):
    Wxh, Whh, Why, bh, by = params
    h_prev = h_0
    hs, ys = [], []
    for x_t in inputs:
        h_t, y_t = rnn_cell_forward(x_t, h_prev, Wxh, Whh, Why, bh, by)
        hs.append(h_t); ys.append(y_t)
        h_prev = h_t
    return hs, ys

Now instantiate it with tiny numbers and step through:

D, H, V = 4, 3, 5           # input dim, hidden dim, vocab size
T = 6                       # sequence length
rng = np.random.default_rng(0)
 
Wxh = rng.normal(scale=0.1, size=(H, D))
Whh = rng.normal(scale=0.1, size=(H, H))
Why = rng.normal(scale=0.1, size=(V, H))
bh  = np.zeros(H); by = np.zeros(V)
h_0 = np.zeros(H)
 
inputs = [rng.normal(size=D) for _ in range(T)]
hs, ys = rnn_forward(inputs, h_0, (Wxh, Whh, Why, bh, by))
print(np.array(hs).shape)   # (6, 3)
print(np.array(ys).shape)   # (6, 5)

Six timesteps in, six hidden states out, six per-step predictions out. Note that h_t depends on every earlier input, transitively, because $h_t = f(x_t, h_{t-1}) = f(x_t, f(x_{t-1}, h_{t-2})) = \ldots$. That transitive dependence is the whole point.

3.1 A picture in your head

Unroll the loop. The RNN "in time" is:

Every arrow labelled with W_xh uses the same $W_{xh}$. Every arrow labelled with W_{hh} uses the same $W_{hh}$. When we backprop, gradient will flow backwards along every arrow — including all the way back through the horizontal chain of $W_{hh}$ multiplications. That chain is what's about to break.

4 · Backpropagation through time (BPTT)

An unrolled RNN is just a very deep feedforward network with tied weights. If you unroll for $T$ steps, you have a network with $T$ hidden layers, each using the same $W_{hh}$ matrix. Standard backprop applies. We just have a name for it in the RNN context: backpropagation through time (BPTT), coined by Werbos in 1990.

The total loss is a sum over timesteps:

The gradient of the loss with respect to a parameter that's used at every timestep (like $W_{hh}$) accumulates over time:

Each per-step term itself unpacks:

That product-of-Jacobians in the middle — $\prod_{i} \partial h_i / \partial h_{i-1}$ — is the object at the heart of the vanishing/exploding gradient story. Each Jacobian is:

where $z_i = W_{xh} x_i + W_{hh} h_{i-1} + b_h$ is the pre-activation. This Jacobian is a product of:

1. A diagonal matrix of $\tanh'$ values, each in $(0, 1]$.
2. The recurrent weight matrix $W_{hh}$.

Both factors matter. The diagonal factor is capped at 1 (usually much less — tanh'(z) = 1 - tanh²(z) peaks at 1 when z=0 and decays fast in either direction). The recurrent matrix can have any spectral norm. Multiply this Jacobian $t - k$ times and things go badly.

5 · The vanishing / exploding gradient problem — with numbers

Let's make Hochreiter's discovery concrete. Consider a simplified RNN with no input, no bias, no non-linearity (h_t = W h_{t-1}). If $W$ has eigendecomposition $W = Q \Lambda Q^{-1}$, then:

So $h_T$ scales like $\lambda_{\max}^T$ where $\lambda_{\max}$ is the largest eigenvalue in magnitude. If $|\lambda_{\max}| < 1$, everything decays to zero. If $|\lambda_{\max}| > 1$, everything blows up. Only if $|\lambda_{\max}| = 1$ exactly do you get a stable signal — an infinitely thin knife-edge that is impossible to maintain during training.

The tanh non-linearity does not help. Its Jacobian factor is $\leq 1$, which pushes the effective operator's spectral norm down, making the vanishing case even worse.

5.1 Reproduce the vanishing gradient in 20 lines

import torch, torch.nn as nn
 
torch.manual_seed(0)
H, T = 64, 100
rnn = nn.RNN(input_size=H, hidden_size=H, batch_first=True, nonlinearity='tanh')
 
x = torch.randn(1, T, H, requires_grad=True)
out, _ = rnn(x)                # (1, T, H)
loss = out[:, -1, :].sum()     # loss depends only on final timestep
loss.backward()
 
# grad w.r.t. each input timestep
grad_norms = x.grad.norm(dim=-1).squeeze()   # (T,)
for t in [0, 20, 40, 60, 80, 99]:
    print(f"t={t:3d}  ||dL/dx_t|| = {grad_norms[t]:.3e}")

Typical output:

t=  0  ||dL/dx_t|| = 4.152e-11
t= 20  ||dL/dx_t|| = 3.418e-08
t= 40  ||dL/dx_t|| = 2.815e-05
t= 60  ||dL/dx_t|| = 2.304e-02
t= 80  ||dL/dx_t|| = 1.729e+00
t= 99  ||dL/dx_t|| = 1.201e+02

Read it: the loss depends only on the output at $t=99$. Gradient flowing backward to $t=99$ is fine (norm ≈ 100). Gradient reaching $t=0$ is eleven orders of magnitude smaller. If an important signal was in $x_0$ (say, the subject of a long sentence), the RNN has essentially no ability to update its weights to remember it. This is the vanishing gradient in one code block.

Change nonlinearity='relu' and re-run, and you'll typically see the opposite failure — the loss goes to nan within a few forward-backward passes because gradients explode instead. There's no free lunch.

5.2 Gradient clipping — the exploding side, patched

Pascanu, Mikolov, and Bengio's 2013 paper On the difficulty of training recurrent neural networks did two things: it explained the math above rigorously, and it proposed a dead-simple fix for the exploding side — gradient clipping.

If the global gradient norm exceeds a threshold, rescale the entire gradient vector to have exactly that norm:

In PyTorch, one line:

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Every RNN training loop uses this. It does not help the vanishing side (you can't "unclip" a zero gradient), but it stops training from immediately diverging when gradients explode. Modern transformer training still uses gradient clipping for a related reason — during pretraining, occasional bad batches produce huge gradients, and clipping is cheap insurance against a run-ending loss spike.

6 · A character-level RNN language model in ~60 lines

Time to build the thing Karpathy made famous.

6.1 Data prep

import torch, torch.nn as nn, torch.nn.functional as F
 
with open('tinyshakespeare.txt') as f:
    text = f.read()
 
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
V = len(chars)                             # ~65 for tinyshakespeare
 
data = torch.tensor([stoi[c] for c in text], dtype=torch.long)
n = int(0.9 * len(data))
train, val = data[:n], data[n:]
 
def get_batch(split, block_size=64, batch_size=32):
    src = train if split == 'train' else val
    ix = torch.randint(len(src) - block_size - 1, (batch_size,))
    x = torch.stack([src[i:i+block_size]     for i in ix])
    y = torch.stack([src[i+1:i+block_size+1] for i in ix])
    return x, y

Every training example is a block_size-long window and the target is the same window shifted by one — predict the next character at each position.

6.2 The model

class CharRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim=64, hidden_dim=128):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.rnn   = nn.RNN(embed_dim, hidden_dim, batch_first=True)
        self.head  = nn.Linear(hidden_dim, vocab_size)
 
    def forward(self, x, h=None):
        e = self.embed(x)                        # (B, T, E)
        out, h = self.rnn(e, h)                  # (B, T, H)
        logits = self.head(out)                  # (B, T, V)
        return logits, h
 
    @torch.no_grad()
    def generate(self, prompt_ids, max_new=200, temperature=1.0):
        h = None
        ids = prompt_ids.clone().unsqueeze(0)    # (1, T)
        for _ in range(max_new):
            logits, h = self(ids[:, -1:], h)     # feed one char at a time
            probs = F.softmax(logits[:, -1] / temperature, dim=-1)
            nxt = torch.multinomial(probs, 1)
            ids = torch.cat([ids, nxt], dim=1)
        return ids.squeeze(0)

6.3 Training loop

model = CharRNN(V).cuda()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
 
for step in range(3000):
    x, y = get_batch('train'); x, y = x.cuda(), y.cuda()
    logits, _ = model(x)                                   # (B, T, V)
    loss = F.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    if step % 200 == 0:
        print(f"step {step}  loss {loss.item():.3f}")

After ~3k steps the loss drops from log(65) ≈ 4.17 (uniform random) to about 2.0. Sample text:

Prompt: "ROMEO:"
Generated: "ROMEO:
Wille the his stall the with your me be his and the sooul,
And so the some the son the with a stalth,
And so the see the with a for my with a with a will..."

That's genuinely Shakespearean rhythm — capitalized speaker names, iambic-ish line breaks, near-English words — from a 130k-parameter model trained on 1MB of text in 90 seconds. Karpathy's blog post covers what happens when you scale this to 3M-parameter LSTMs on 100MB of C source code; you get syntactically almost-valid C with balanced braces and reasonable variable names. In 2015 this was genuinely magical.

6.4 Truncated BPTT — the reason block_size exists

Notice we chose block_size=64 — we do not train on the full Shakespeare corpus as one 1-million-character sequence. That's because BPTT requires storing every intermediate activation for backprop, so memory grows linearly with sequence length. On a laptop GPU, a 1M-token sequence would need hundreds of GB.

Truncated BPTT (TBPTT) cuts the sequence into chunks of length $T$ (here 64), runs BPTT within each chunk, then either resets the hidden state or carries it forward without propagating gradient through the boundary (.detach()). This bounds memory at $O(T)$ instead of $O(\text{full length})$.

In our get_batch implementation we sample random windows and reset the hidden state each time, which is the simplest form of TBPTT. More sophisticated implementations preserve state across sequential windows.

The cost: the model cannot learn dependencies longer than block_size. If the answer to a question depends on context 200 characters back but block_size=64, tough luck. This tradeoff — longer windows → more compute + memory + vanishing gradients; shorter windows → cheaper training but less context — was one of the central pains of the RNN era and one of the things transformers solved (partially) by making attention $O(T^2)$ in compute but $O(1)$ in gradient distance.

7 · Where RNNs win, where they lose

RNNs are still competitive when:

• Streaming inference matters (audio, sensor data, real-time trading). RNNs process one token at a time in constant memory and constant time per token, whereas naive transformer inference is $O(T)$ per token because of attention over the full history. This is why many production ASR (automatic speech recognition) systems still use LSTMs.
• Sequence lengths are very long (10k–1M tokens) and you cannot afford $O(T^2)$ attention. This is exactly the regime where State Space Models (Mamba, Mamba-2) have made recurrence fashionable again in 2024–2025.
• Compute budget is tiny (edge devices). A one-layer GRU with 128 hidden units fits in 100KB.

RNNs lose badly when:

• You need to model long-range dependencies with dense information content (natural language, code). Transformers dominate.
• You need to parallelise across sequence length during training. RNNs are sequential-by-construction — you cannot compute $h_t$ without $h_{t-1}$. On a GPU with 10,000 cores, this leaves 99.99% of the hardware idle at each step. Transformers can process every timestep in parallel. This was the single biggest reason for the 2017 shift.
• You need in-context learning (feeding examples in the prompt and having the model figure out the task). Attention's ability to directly reference any earlier token gives transformers a huge advantage here; RNNs would have to compress the examples through their hidden state bottleneck.

8 · The 2024–2025 comeback of recurrence — State Space Models

The obituary of the RNN was written prematurely. In late 2023 the Mamba paper (Gu & Dao, 2023) proposed a selective state space model — a recurrent architecture with an input-dependent transition matrix, trainable via a parallel scan algorithm that fully utilises GPUs. Mamba matched transformer language modelling quality at up to 3B parameters while scaling linearly with sequence length.

The 2024–2025 line accelerated fast:

• Mamba-2 (Dao & Gu, ICML 2024, arXiv:2405.21060) — reformulated the state space model as a special case of structured attention (they call this the "state space duality"), delivered 2–8× speedups over Mamba, hit 8B parameters.
• Jamba (AI21, March 2024, Jamba-1.5 release August 2024) — a hybrid architecture interleaving Mamba layers with transformer attention layers and MoE experts. Jamba-1.5-Large (94B active / 398B total) matched Llama-3 70B quality with 3× the effective context length (256k tokens) and much cheaper inference.
• RWKV-7 "Goose" (BlinkDL et al., 2025, rwkv.com/rwkv-7) — a pure-recurrent transformer-quality model. RWKV-7 removed the softmax attention entirely and matches Llama-3-8B on many benchmarks while running at constant memory per token.
• Griffin / RecurrentGemma (Google DeepMind, 2024, arXiv:2402.19427) — a hybrid of gated linear recurrence and local attention. Griffin-14B outperformed Llama-2 13B while inference-time-competing with Mamba.

The unifying insight: modern recurrent architectures put in the engineering work to avoid Hochreiter's vanishing gradient — usually by making the recurrence a large but well-conditioned linear operator (a state space transition) with carefully designed initialisation. The 1990s vanilla RNN failed because it had no such structure. Add structure, and recurrence is back on the menu.

9 · War stories

10 · Try it yourself

11 · Recap in five sentences

An RNN maintains a hidden-state vector that is updated at each timestep by a shared set of weights, giving it the ability to model variable-length sequences with $O(1)$ memory per step. Training uses backpropagation through time, which is standard backprop applied to the unrolled computation graph — mathematically simple, but the repeated multiplication by the recurrent Jacobian causes gradients to vanish or explode across long sequences. Gradient clipping patches the exploding side; nothing patches the vanishing side, which is why LSTMs, GRUs, attention, and transformers had to be invented. RNNs were dethroned by transformers in 2017 for language, but 2024–2025 state-space models (Mamba, Mamba-2, Jamba, RWKV-7) have made structured recurrence competitive again for long-context modelling. You now have the canonical example of "recurrence" and can meaningfully read the modern SSM literature.

import torch
W = torch.randn(64, 64) * 0.1
W.requires_grad_(True)
h = torch.randn(1, 64)

norms = []
for t in range(50):
    h = torch.tanh(h @ W)
    if (t + 1) % 10 == 0:
        g = torch.autograd.grad(h.sum(), W, retain_graph=True)[0]
        norms.append((t + 1, g.norm().item()))
for t, n in norms:
    print(t, n)   # falls by orders of magnitude as t grows

🧠 Retention scaffold

One-line summary (write it in your own words): ________________________________________________

Spaced review: re-read Section 4 (BPTT derivation) and Section 5 (vanishing gradient with numbers) in 24 hours. Revisit the full session on day 7.

Next session (S033): we fix Hochreiter's vanishing gradient with gates — the LSTM (1997) and the GRU (2014). You will see how a single "cell state" with additive updates preserves gradient flow across hundreds of timesteps, and how the same idea prefigures the transformer's residual connections.

Sticky note (keep on your desk): RNN = shared weights + hidden state. BPTT = deep-network backprop with tied weights. Vanishing gradient = the product of Jacobians shrinks to zero. Every gate, residual, and attention head since 1997 exists to fight this one product.

Further reading

• Origin. Elman 1990, Finding Structure in Time. The paper that gave us the term "simple recurrent network".
• Vanishing gradient. Hochreiter 1991 thesis; Bengio, Simard, Frasconi 1994 Learning long-term dependencies with gradient descent is difficult; Pascanu, Mikolov, Bengio 2013, arXiv:1211.5063.
• Karpathy blog. The Unreasonable Effectiveness of Recurrent Neural Networks. Still one of the best pedagogical writeups in ML.
• Modern recurrence. Mamba (Gu & Dao 2023); Mamba-2 (Dao & Gu, ICML 2024); Jamba-1.5 (AI21 2024); Griffin / RecurrentGemma (DeepMind 2024); RWKV (Peng et al. 2023) and the newer RWKV-7 "Goose" release notes on rwkv.com.
• Why transformers won (for now). Vaswani et al. 2017, Attention Is All You Need. Read the abstract and Section 4; the parallelism argument is what killed RNNs for language.

Previous: ← DL S031 · Next: DL S033 →