RNNs from scratch: complete BPTT, character generation and structured recurrence

A recurrent network repeatedly updates a fixed-width state with shared parameters. Unroll it and the forward graph looks like a deep network across time;backpropagation must sum contributions from all uses of those parameters. This lesson makes that graph and its failure modes executable rather than treating recurrence as obsolete history.

Sequence state, historical motivation and the learning problem

Elman's simple recurrent networks,Hochreiter's early gradient analysis and the later LSTM/Pascanu work are important context. The original lesson's imagined Munich scene,precise inevitable timestep limit and “no stable middle ground” story are not a substitute for the math. Some dependencies fit in a small state yet are hard to learn;other tasks genuinely require more information than a fixed finite-precision state can preserve. Capacity and gradient conditioning are distinct possible limitations.

The note-taker analogy is useful:an index card summarizes a lecture and is rewritten as new input arrives. Shared update weights specify how to rewrite it. The card is not a lossless transcript. A recurrent cell works with variable sequence lengths because it reuses the rule,not because parameters grow with T.

Outcomes and prerequisites. Starting with array backprop,autograd,logit cross-entropy and DL031's lookup tables,implement NumPy forward/BPTT,verify every parameter gradient,train a small character model on bundled text,generate from a complete prompt,and measure temporal gradients. Then distinguish TBPTT,clipping,gates,attention and structured scans. No Shakespeare dataset,weights or GPU is downloaded.

Unroll the recurrence

Using column vectors,

Shapes are x(D),h(H),Wxh(H,D),Whh(H,H),Why(V,H),bh(H),by(V). z contains logits,not probabilities. A row-batch implementation uses transposed weight products;switching conventions without transposing is a common silent derivation error.

Each horizontal edge reuses Whh. Later inputs cannot affect earlier states in this causal recurrence. Local loss1 contributes a short gradient path;loss3 also contributes paths through h2 and h1. Convolution shares across space;recurrence shares across time,but boundary states and time-varying inputs still matter.

The existing exact scalar program remains as a hand-checkable starting point. For inputs(1,0,−1),Wx.5,Wh.8,b0,its states are approximately0,.462117,.353724,−.213677. It checks the sum of recurrent-weight contributions against finite differences.

import numpy as np
 
x = np.array([1., 0., -1.])
wx, wh, bias, target = .5, .8, 0., .2
 
def forward(wh):
    h = [0.]
    for token in x:
        h.append(np.tanh(token*wx + h[-1]*wh + bias))
    return np.array(h)
 
h = forward(wh)
dh = h[-1] - target
grad_wh = 0.
for t in range(len(x), 0, -1):
    da = dh * (1-h[t]**2)
    grad_wh += da * h[t-1]
    dh = da * wh
 
def loss(w):
    return .5 * (forward(w)[-1]-target)**2
fd = (loss(wh+1e-5)-loss(wh-1e-5)) / 2e-5
assert abs(fd-grad_wh) < 1e-8
print('states', np.round(h, 6).tolist(), 'recurrent gradient', round(grad_wh, 6))

Full NumPy forward and BPTT with losses at every step

At each t,softmax-CE gives $\delta z_t=p_t-e_{y_t}$ for a summed loss. Add the future-state derivative before differentiating tanh:

Accumulate $dW_{hy}+=\delta z_t h_t^\top$, $dW_{xh}+=\delta a_t x_t^\top$, $dW_{hh}+=\delta a_t h_{t-1}^\top$,and the respective bias sums. Input derivatives are $W_{xh}^\top\delta a_t$. The returned h0 derivative distinguishes a detached boundary from a reset value.

import torch
import torch.nn as nn
import torch.nn.functional as F
 
def rnn_cell_forward(x_t,h_prev,Wxh,Whh,Why,bh,by):
    h_t=np.tanh(Wxh@x_t+Whh@h_prev+bh)
    return h_t,Why@h_t+by
 
def rnn_forward(inputs,h0,params):
    hs=[h0.copy()];logits=[]
    for xt in inputs:
        ht,zt=rnn_cell_forward(xt,hs[-1],*params)
        hs.append(ht);logits.append(zt)
    return np.array(hs),np.array(logits)
 
def rnn_loss_backward(inputs,targets,h0,params):
    Wxh,Whh,Why,bh,by=params
    hs,z=rnn_forward(inputs,h0,params)
    shifted=z-z.max(1,keepdims=True)
    logp=shifted-np.log(np.exp(shifted).sum(1,keepdims=True))
    loss=-logp[np.arange(len(targets)),targets].sum()
    dz=np.exp(logp);dz[np.arange(len(targets)),targets]-=1
    grads=[np.zeros_like(p) for p in params]
    dWx,dWh,dWy,dbh,dby=grads
    dx=np.zeros_like(inputs);future=np.zeros_like(h0)
    for t in range(len(inputs)-1,-1,-1):
        ht=hs[t+1]
        dWy+=np.outer(dz[t],ht);dby+=dz[t]
        dh=Why.T@dz[t]+future
        da=dh*(1-ht**2)
        dWx+=np.outer(da,inputs[t]);dWh+=np.outer(da,hs[t]);dbh+=da
        dx[t]=Wxh.T@da;future=Whh.T@da
    return loss,grads,dx,future
 
rng=np.random.default_rng(32);D,H,V,T=4,3,5,6
params=[rng.normal(scale=.1,size=(H,D)),rng.normal(scale=.1,size=(H,H)),
        rng.normal(scale=.1,size=(V,H)),np.zeros(H),np.zeros(V)]
inputs=rng.normal(size=(T,D));h0=np.zeros(H);targets=np.arange(T)%V
hs,z=rnn_forward(inputs,h0,params)
assert hs[1:].shape==(6,3) and z.shape==(6,5)
loss,grads,dx,dh0=rnn_loss_backward(inputs,targets,h0,params)
for arr,grad in list(zip(params,grads))+[(inputs,dx),(h0,dh0)]:
    for idx in np.ndindex(arr.shape):
        old=arr[idx];eps=1e-5
        arr[idx]=old+eps;plus=rnn_loss_backward(inputs,targets,h0,params)[0]
        arr[idx]=old-eps;minus=rnn_loss_backward(inputs,targets,h0,params)[0]
        arr[idx]=old
        assert abs((plus-minus)/(2*eps)-grad[idx])<1e-8
# Independent autograd graph using the same numerical tensors.
tp=[torch.tensor(p,requires_grad=True) for p in params]
ti=torch.tensor(inputs,requires_grad=True);th0=torch.tensor(h0,requires_grad=True)
h=th0;out=[]
for xt in ti:
    h=torch.tanh(tp[0]@xt+tp[1]@h+tp[3]);out.append(tp[2]@h+tp[4])
tl=F.cross_entropy(torch.stack(out),torch.tensor(targets),reduction='sum')
tl.backward();assert abs(tl.item()-loss)<1e-12
for a,b in zip(grads,[p.grad.numpy() for p in tp]):np.testing.assert_allclose(a,b,atol=1e-12,rtol=1e-12)
np.testing.assert_allclose(dx,ti.grad.numpy(),atol=1e-12)
np.testing.assert_allclose(dh0,th0.grad.numpy(),atol=1e-12)
changed=inputs.copy();changed[-1]+=10
assert np.array_equal(rnn_forward(changed,h0,params)[0][:-1],hs[:-1])
print('full NumPy BPTT finite differences and torch parity: pass; summed loss',loss)

For a mean loss,divide all associated derivatives by the number of contributing targets. A padding mask changes that denominator. These equations use one effective hidden bias;PyTorch nn.RNN has separate input and recurrent bias tensors whose sum enters the cell. Match both when porting weights.

Temporal Jacobians: what vanishes and what does not

The local state Jacobian is $J_t=\operatorname{diag}(1-h_t^2)W_{hh}$. A final loss reaching an earlier state traverses an ordered product of changing J_t. Its norm is bounded by the product of their operator norms. A bound below1 at every step gives contraction;one singular value greater than1 is not proof that every direction or actual trajectory explodes.

For a purely linear,constant recurrence,the matrix power is governed asymptotically by spectral structure,but nonnormal matrices can exhibit transient growth even with eigenvalues inside the unit circle. Tanh bounds the hidden values while their sensitivities can behave differently. Saturation shrinks its diagonal derivative. Orthogonal/unitary designs,initialization,gating and carefully scaled recurrence can help;stability is not mathematically impossible.

To measure temporal reach,use derivatives with respect to each input or the initial state. The original rich-component experiment differentiated h_T with respect to shared W;that gradient aggregates all timesteps and is not an isolated early-state gradient. Its expected decay was therefore not a valid universal oracle.

# Controlled scalar sensitivities: no random-model benchmark needed.
for factor in (.5,1.,1.5):
    initial=torch.tensor(1.,dtype=torch.float64,requires_grad=True);state=initial
    for _ in range(20):state=factor*state
    grad=torch.autograd.grad(state,initial)[0].item()
    assert abs(grad-factor**20)<1e-9
    print('linear factor',factor,'20-step sensitivity',grad)
# Nonnormal counterexample: eigenvalues=.5, yet one-step norm can greatly exceed1.
A=np.array([[.5,10.],[0.,.5]])
assert max(abs(np.linalg.eigvals(A)))<1 and np.linalg.norm(A,2)>1
# A seeded actual RNN reports observed gradients, not the old invented printed numbers.
torch.manual_seed(32)
rnn=nn.RNN(16,16,batch_first=True).double()
xx=torch.randn(1,50,16,dtype=torch.float64,requires_grad=True)
out,final=rnn(xx);out[:,-1].sum().backward()
norms=xx.grad.norm(dim=-1).squeeze()
assert torch.isfinite(norms).all() and norms[0]<norms[-1]
print('seeded RNN temporal norms',[(t,norms[t].item()) for t in (0,10,20,30,40,49)])

Switching tanh to ReLU does not automatically make a randomly initialized RNN explode. Measure both rather than prescribing a predetermined outcome. A single loss curve does not identify temporal conditioning;inspect gradients and compare controlled sequence lengths/inputs.

Clipping bounds a finite gradient, not every failure

Global norm clipping rescales g by min(1,τ/||g||),preserving its direction when finite. For g=(3,4),τ1,the result is(.6,.8),not elementwise(1,1). It can reduce destructive optimizer steps;it cannot recover vanished information,undo a forward NaN or guarantee convergence.

p=nn.Parameter(torch.zeros(2,dtype=torch.float64));p.grad=torch.tensor([3.,4.],dtype=torch.float64)
oldnorm=torch.nn.utils.clip_grad_norm_([p],1.,error_if_nonfinite=True)
assert abs(oldnorm.item()-5)<1e-12
assert abs(p.grad.norm().item()-1)<1e-6
assert abs((p.grad[0]/p.grad[1]).item()-.75)<1e-12
p.grad=torch.tensor([float('inf'),1.],dtype=torch.float64)
try:
    torch.nn.utils.clip_grad_norm_([p],1.,error_if_nonfinite=True)
    raise AssertionError('nonfinite gradient accepted')
except RuntimeError:
    pass

Clip after backward and before optimizer.step;with scaled mixed precision,unscale first. Stable cross-entropy operates on logits using log-sum-exp,so finite large logits do not inherently create log(0) in the loss. Inspect input outliers and the first nonfinite activation/gradient rather than blaming a confident softmax alone. Input scaling,clipping thresholds and anomaly controls are task-dependent,not blanket cures.

Character modeling: complete local training and generation

A character model uses an embedding table,a recurrent layer and a vocabulary head. At each position predict the next character. Shifted targets must never cross a split boundary. The bundled text is deliberately tiny and repetitive;its optimization behavior tests mechanics,not English fluency or a Shakespeare benchmark. An optional trusted local file can replace it explicitly—never a hidden download fallback.

from pathlib import Path
 
BUNDLED_TEXT=('The river turns beside the old stone bridge.\n'
              'A quiet bird returns to the garden.\n'
              'We read a line and write another line.\n')*35
 
def prepare_text(text):
    if len(text)<100:raise ValueError('text too short')
    # Vocabulary from training only; reject held-out unseen characters explicitly.
    cut=int(.9*len(text));training=text[:cut];validation=text[cut:]
    chars=sorted(set(training));stoi={c:i for i,c in enumerate(chars)}
    unknown=set(validation)-set(chars)
    if unknown:raise ValueError('held-out OOV characters: '+repr(sorted(unknown)))
    encode=lambda s:torch.tensor([stoi[c] for c in s],dtype=torch.long)
    return encode(training),encode(validation),stoi,chars
 
def get_batch(src,block_size=24,batch_size=8,generator=None):
    if len(src)<=block_size:raise ValueError('split shorter than one input/target window')
    # Valid start includes len(src)-block_size-1.
    starts=torch.randint(len(src)-block_size,(batch_size,),generator=generator)
    x=torch.stack([src[i:i+block_size] for i in starts])
    y=torch.stack([src[i+1:i+block_size+1] for i in starts])
    return x,y
 
class CharRNN(nn.Module):
    def __init__(self,vocab_size,embed_dim=16,hidden_dim=24):
        super().__init__()
        self.embed=nn.Embedding(vocab_size,embed_dim)
        self.rnn=nn.RNN(embed_dim,hidden_dim,batch_first=True)
        self.head=nn.Linear(hidden_dim,vocab_size)
    def forward(self,x,h=None):
        out,h=self.rnn(self.embed(x),h)
        return self.head(out),h
    @torch.no_grad()
    def generate(self,prompt_ids,max_new=100,temperature=1.,generator=None):
        if prompt_ids.ndim!=1 or len(prompt_ids)==0:raise ValueError('nonempty1D prompt required')
        if not np.isfinite(temperature) or temperature<=0:raise ValueError('temperature must be finite positive')
        if max_new<0:raise ValueError('negative generation length')
        was_training=self.training;self.eval()
        ids=prompt_ids.clone().unsqueeze(0)
        try:
            logits,h=self(ids)  # consume ALL prompt characters exactly once
            for i in range(max_new):
                probs=F.softmax(logits[:,-1]/temperature,dim=-1)
                nxt=torch.multinomial(probs,1,generator=generator)
                ids=torch.cat([ids,nxt],dim=1)
                if i+1<max_new:logits,h=self(nxt,h)
        finally:self.train(was_training)
        return ids.squeeze(0)
 
def train_char_local(text=BUNDLED_TEXT,steps=160):
    train,val,stoi,itos=prepare_text(text)
    torch.manual_seed(32);model=CharRNN(len(itos))
    opt=torch.optim.Adam(model.parameters(),lr=.01)
    generator=torch.Generator().manual_seed(32)
    fixed_x,fixed_y=get_batch(train,generator=generator)
    def fixed_loss():
        model.eval()
        with torch.no_grad():return F.cross_entropy(model(fixed_x)[0].reshape(-1,len(itos)),fixed_y.reshape(-1)).item()
    before=fixed_loss();history=[]
    for step in range(steps):
        model.train();x,y=get_batch(train,generator=generator)
        logits,_=model(x)
        loss=F.cross_entropy(logits.reshape(-1,len(itos)),y.reshape(-1))
        opt.zero_grad(set_to_none=True);loss.backward()
        norm=torch.nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True)
        opt.step()
        if step%40==0:history.append({'step':step,'loss':loss.item(),'gradient_norm_before_clip':norm.item()})
    after=fixed_loss()
    vx,vy=get_batch(val,generator=torch.Generator().manual_seed(33))
    with torch.no_grad():validation=F.cross_entropy(model(vx)[0].reshape(-1,len(itos)),vy.reshape(-1)).item()
    return model,stoi,itos,{'before':before,'after':after,'validation_batch_loss':validation,'history':history}
 
model,stoi,itos,training_report=train_char_local()
assert training_report['after']<training_report['before']
prompt=torch.tensor([stoi[c] for c in 'The '])
for temperature in (.5,1.,1.5):
    sample=model.generate(prompt,60,temperature,torch.Generator().manual_seed(34))
    assert torch.equal(sample[:len(prompt)],prompt) and len(sample)==len(prompt)+60
    print('temperature',temperature,''.join(itos[i] for i in sample.tolist()))
print('bundled character training',training_report,'parameters',sum(p.numel() for p in model.parameters()))

The original generator fed only the last prompt character on its first iteration. This version primes the state with the full prompt,then feeds each sampled token once. Lower temperature sharpens the next-token distribution,higher flattens it;“creative” is not a numerical property guaranteed by a temperature value.

For a local corpus use prepare_text(Path(local_path).read_text()) or pass that string to train_char_local. Record the file hash,split boundary,vocabulary,model shape,seed,steps and evaluation protocol. Save state_dict plus vocabulary if persisting;load only trusted compatible files. The bundled repetition means train/validation text shares patterns;its held-out batch loss is a diagnostic,not independent generalization evidence.

A NumPy character model using the same BPTT

The NumPy cell is also a complete character language model when x is a one-hot character. The columns of Wxh then act as learned lookup vectors. This wrapper uses the exact rnn_loss_backward above, divides its summed gradients by the number of targets, clips their joint norm and updates every parameter with SGD. It resets state for independent windows, rather than accidentally carrying shuffled history. The tiny repeated pattern tests optimization, not fluency.

def train_numpy_chars(text='ab cab ba\n'*40,steps=120,width=12,context=12):
    if min(steps,width,context)<1 or len(text)<10*(context+1):
        raise ValueError('positive settings and enough text required')
    cut=int(.8*len(text));chars=sorted(set(text[:cut]));lookup={c:i for i,c in enumerate(chars)}
    if set(text[cut:])-set(chars):raise ValueError('held-out OOV character')
    ids=np.array([lookup[c] for c in text]);train,val=ids[:cut],ids[cut:]
    rg=np.random.default_rng(132);v=len(chars)
    weights=[rg.normal(scale=.1,size=(width,v)),rg.normal(scale=.1,size=(width,width)),
             rg.normal(scale=.1,size=(v,width)),np.zeros(width),np.zeros(v)]
    eye=np.eye(v);zero=np.zeros(width)
    def evaluate(split):
        total=count=0
        for start in range(0,len(split)-1,context):
            end=min(start+context,len(split)-1)
            total+=rnn_loss_backward(eye[split[start:end]],split[start+1:end+1],zero,weights)[0]
            count+=end-start
        return total/count
    before=evaluate(train)
    for _ in range(steps):
        start=int(rg.integers(len(train)-context))
        _,gs,_,_=rnn_loss_backward(eye[train[start:start+context]],train[start+1:start+context+1],zero,weights)
        gs=[g/context for g in gs];norm=np.sqrt(sum((g*g).sum() for g in gs))
        scale=min(1.,1./max(norm,1e-12))
        for w,g in zip(weights,gs):w-=.3*scale*g
    return weights,lookup,chars,{'before':before,'after':evaluate(train),'validation_nll':evaluate(val)}
 
def sample_numpy_chars(weights,lookup,chars,prompt,count=40,temperature=1.):
    if not prompt or any(c not in lookup for c in prompt):raise ValueError('in-vocabulary prompt required')
    if count<0 or not np.isfinite(temperature) or temperature<=0:raise ValueError('invalid sampling settings')
    rg=np.random.default_rng(133);eye=np.eye(len(chars));state=np.zeros(weights[1].shape[0])
    for c in prompt:state,logits=rnn_cell_forward(eye[lookup[c]],state,*weights)
    result=prompt
    for _ in range(count):
        z=logits/temperature;prob=np.exp(z-z.max());prob/=prob.sum()
        nxt=int(rg.choice(len(chars),p=prob));result+=chars[nxt]
        state,logits=rnn_cell_forward(eye[nxt],state,*weights)
    return result
 
numpy_weights,numpy_lookup,numpy_chars,numpy_report=train_numpy_chars()
assert numpy_report['after']<numpy_report['before']
numpy_sample=sample_numpy_chars(numpy_weights,numpy_lookup,numpy_chars,'ab ',20)
assert numpy_sample.startswith('ab ') and len(numpy_sample)==23
print('NumPy character diagnostic',numpy_report,numpy_sample)

Both frameworks now have a complete local training/sampling path. Their optimizers, widths and initializations differ, so these losses are not a controlled NumPy-versus-PyTorch performance comparison.

Prompt, shape and state-continuity tests

model.eval()
# A full forward and tokenwise state carry must agree in eval mode.
sequence=torch.tensor([[stoi[c] for c in 'The river']])
with torch.no_grad():
    all_logits,all_h=model(sequence)
    h=None;pieces=[]
    for t in range(sequence.shape[1]):
        logits,h=model(sequence[:,t:t+1],h);pieces.append(logits)
    torch.testing.assert_close(torch.cat(pieces,dim=1),all_logits)
    torch.testing.assert_close(h,all_h)
    assert h.shape==(1,1,24)  # batch_first does NOT change hidden-state layout
    # Compare first sampled token to full-prompt reference with the same RNG.
    ref=torch.multinomial(F.softmax(model(prompt.unsqueeze(0))[0][:,-1],dim=-1),1,
                          generator=torch.Generator().manual_seed(77))
generated=model.generate(prompt,1,1.,torch.Generator().manual_seed(77))
assert generated[-1].item()==ref.item()
try:
    model.generate(prompt,1,0);raise AssertionError('zero temperature accepted')
except ValueError:
    pass
# Different prefixes with the same final character are not silently discarded.
with torch.no_grad():
    aa=model(torch.tensor([[stoi[c] for c in 'The ']]))[0][:,-1]
    bb=model(torch.tensor([[stoi[c] for c in 'A q ']]))[0][:,-1]
assert not torch.allclose(aa,bb)

With batch_first=True,inputs/outputs are(B,T,D)/(B,T,H);hidden states remain(layers×directions,B,H). Bidirectional output concatenates directions and is inappropriate for a causal next-token LM if the reverse direction sees future targets. Dropout between stacked recurrent layers is different from input dropout and has no inter-layer role for one layer.

Reproducibility needs more than torch.manual_seed: record PyTorch/NumPy versions, RNG states, data order, precision and kernel settings. The PyTorch2.9 RNN documentation warns that some cuDNN/CUDA versions have nondeterministic RNN behavior and documents version-specific environment settings. torch.use_deterministic_algorithms(True) can reject unsupported nondeterministic operations; warn_only=True only warns and is not a deterministic guarantee. A CUDA/cuDNN setting cannot prove cross-release or cross-device bitwise equality. The checks here are single-thread CPU runs; no GPU determinism or throughput test is claimed.

TBPTT: detach is not reset

Full BPTT retains activations across the unroll,with memory depending on T,batch,width,layers and saved tensors. Random independent windows usually reset state each batch. Stateful TBPTT carries numerical state across contiguous chunks of the same streams,then detaches at chosen boundaries to stop direct gradient credit crossing them. Detaching an unrelated sequence's state does not erase its information;reset or mask it instead.

def detach_state(state):
    if state is None:return None
    if isinstance(state,tuple):return tuple(s.detach() for s in state)
    return state.detach()
 
# Numerical carry survives, but cross-boundary autograd credit does not.
w=torch.tensor(.8,dtype=torch.float64,requires_grad=True)
a=torch.tensor(.2,dtype=torch.float64,requires_grad=True)
first=torch.tanh(w*a);carried=first.detach();last=torch.tanh(w*carried)
g_a=torch.autograd.grad(last,a,allow_unused=True,retain_graph=True)[0]
assert g_a is None and carried.item()==first.item()
assert torch.autograd.grad(last,w)[0] is not None
# A stateful training skeleton for already aligned contiguous streams.
def tbptt_step(model,opt,x,y,state=None):
    model.train();state=detach_state(state)
    logits,state=model(x,state)
    loss=F.cross_entropy(logits.reshape(-1,logits.shape[-1]),y.reshape(-1))
    opt.zero_grad(set_to_none=True);loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True)
    opt.step();return loss.item(),detach_state(state)
opt=torch.optim.SGD(model.parameters(),lr=.001)
stream=torch.tensor([[stoi[c] for c in 'The river turns beside the old stone bridge.']])
_,state=tbptt_step(model,opt,stream[:,:12],stream[:,1:13])
_,state=tbptt_step(model,opt,stream[:,12:24],stream[:,13:25],state)
assert state.grad_fn is None

A chunk length64 limits direct credit through that training graph;it does not prove a carried recurrent state can never represent a dependency beyond64. A random-window reset removes previous context entirely,whereas stateful detachment preserves it without full-history gradients. More activation checkpointing can trade recomputation for memory;it does not make arbitrary long-context training free.

Gates, attention and resource tradeoffs

LSTM has a cell update $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$. Along its direct path,the derivative contains forget gates;near1 can preserve signal better. Gates depend on state and other paths still exist,so LSTM does not abolish vanishing/exploding gradients. A GRU combines gated candidate/current-state paths under a specified convention. Replacing nn.RNN with LSTM changes the carried state to an(h,c) tuple and increases parameters;it is not an identical-interface,equal-capacity ablation without adjustments.

Streaming recurrence has constant-size model state at fixed width and bounded per-token work independent of elapsed history. Standard cached full attention keeps keys/values that grow with context and scans prior tokens;full training attention performs quadratic interactions. Memory-efficient attention reduces score-matrix storage without deleting that arithmetic. Transformers can parallelize known token positions in training,but autoregressive sampling still depends on earlier sampled outputs.

A128-hidden GRU's storage depends on input width and dtype. With input128,PyTorch's three gates and two bias sets give99,072 parameters,396,288FP32 bytes before embedding/head/state—not a universal100KB device model. Batching and within-cell matrix products parallelize even a vanilla RNN;the claim that99.99% of a GPU is necessarily idle is false.

For the original RNN/LSTM exercise,match parameter budget or report differing counts,train on the same local corpus and compare held-out loss,sample protocol and actual temporal derivatives. The next lesson supplies an executable three-model comparison with trained input-gradient profiles and a loss-chart exporter; use that implementation rather than treating this exercise prompt as a completed comparison. A10–20% improvement or fixed orders-of-magnitude gradient gap must be measured,not prewritten as expected output.

Structured recurrence and associative scans

A general nonlinear recurrence cannot be parallelized by merely diagonalizing Whh. But an affine recurrence $h_t=A_th_{t-1}+b_t$ can compose transitions associatively:

An associative prefix scan computes many such composed transitions in parallel. Practical efficiency depends on structured A and a manageable state;dense matrix composition can be too expensive. Coefficients must be available from the inputs/parameters rather than requiring a previous nonlinear state to define the next transformation.

def compose(later,earlier):
    A2,b2=later;A1,b1=earlier
    return A2@A1,A2@b1+b2
r=np.random.default_rng(32)
trans=[(r.normal(size=(3,3))*.1,r.normal(size=3)) for _ in range(3)]
a,b,c=trans
left=compose(c,compose(b,a));right=compose(compose(c,b),a)
np.testing.assert_allclose(left[0],right[0]);np.testing.assert_allclose(left[1],right[1])
hstart=r.normal(size=3);hseq=hstart.copy()
for A,bias in trans:hseq=A@hseq+bias
np.testing.assert_allclose(hseq,left[0]@hstart+left[1])
print('associative affine scan oracle passed')

Mamba uses selective state-space dynamics with input-dependent discretization/selection and hardware-aware execution. It is not simply tanh removed from an arbitrary dense RNN or a universal “diagonal-plus-low-rank Whh” description. Mamba2 develops structured state-space duality with attention and efficient algorithms. Their reported throughput/quality comparisons belong to their specific model sizes,baselines and hardware.

Jamba/Jamba1.5 interleave attention and Mamba with MoE components;the attention layers retain their own context-dependent cache,so the whole hybrid is not automatically constant-memory. Griffin combines gated linear recurrence with local attention;RecurrentGemma is a related released model family,not a14B model by definition. RWKV7 develops a recurrent state update with generalized delta-rule dynamics;it should not be summarized as “removed softmax from a standard transformer.” Structured recurrence remains useful without making any of these a universal benchmark winner.

These designs address memory,conditioning and parallel execution in different ways. Attention's direct path between positions can shorten dependency distance;it does not remove all depth-wise Jacobian products. The best choice depends on information retention,sequence length,streaming constraints,data and measured hardware cost—not only whether the sequence fits a context window.

Worked answers, failure checks and retention

1. **Cell equation:**h_t=tanh(Wxh x_t+Whh h_(t−1)+bh);logits=Why h_t+by. Reuse every parameter over t.
2. **Jacobian product:**diag(1−h_t²)Whh at each step,ordered across time. Uniform contraction gives a bound;changing directions,saturation and nonnormal effects defeat a simplistic largest-eigenvalue-only prediction.
3. **Training skeleton:**get correctly shifted local batch,forward,logit CE,zero gradients,backward,finite global-norm clip,optimizer step;evaluate separately and preserve tokenizer/split identity.
4. **Why clipping differs:**it bounds a large finite gradient magnitude;it cannot restore vanished credit or repair a nonfinite forward. clip_grad_norm_(parameters,1.,error_if_nonfinite=True) is the explicit tested call.
5. **Why attention became useful:**training-position parallelism,direct access to prior representations and shorter paths between distant positions. Recurrence remains attractive for bounded streaming state;modern structured forms add efficient scans and different dynamics.
6. **Parallel-scan stretch:**affine transition composition must be associative and computationally cheap under structure. Arbitrary tanh state updates cannot be made equivalent by a scan of Whh alone;the executable3×3 example shows the actual algebra.
7. **Reset or detach?**Reset between independent/shuffled sequences;detach across aligned TBPTT boundaries to keep numerical history but cut gradients. Detach does not fix cross-user/state leakage.
8. **Causality and prompt tests:**later inputs cannot change earlier forward states;full-prompt priming must agree with tokenwise state carry. These are stronger checks than plausible-looking generated text.

Tomorrow reconstruct BPTT and the prompt loop. Next week compare RNN/LSTM under an honest equal-budget protocol and plot actual input-gradient norms. The next lesson studies gates in detail,not a guaranteed cure for every temporal learning problem.

Sources and execution boundary

Primary sources: Pascanu et al.2013, PyTorch2.9 RNN shape contract, Attention Is All You Need, Mamba, Mamba2, Jamba1.5, Griffin, RWKV7.

Original optional StatQuest/Serrano video attribution and Karpathy blog reading recommendations are unverified navigation, not primary support. The named sources above support the reviewed mechanisms independently.

Executed:NumPy character training/sampling, preserved scalar BPTT,full NumPy every-coordinate gradients with independent torch parity,linear/nonnormal controls,actual seeded input-gradient measurement,bundled character training/sampling,full-prompt/state-continuity tests,clipping/TBPTT and affine-scan oracles. Unexecuted:Shakespeare training,LSTM quality comparison,SSM/GPU benchmarks,downloaded models and deployment. Previous: Word2Vec · Next: gates.