LSTM and GRU — Gates, State and Gradient Paths

Gates are controllable paths, not a guarantee

An ordinary RNN repeatedly applies a nonlinear state transformation. Its time derivative is a product of time-varying Jacobians, not just the recurrent matrix's eigenvalue raised to a power. LSTM introduced a separately carried cell state; the later forget-gate variant makes its retention learnable. Hochreiter and Schmidhuber's 1997 paper is not the same architecture as every modern four-projection implementation: the forget gate was added later. There is no evidence here for the old story that nobody believed the paper, nor for Google Translate deploying this design in 2014. Sutskever et al. describe a research translation system; GNMT's production-system paper is from 2016.

The objective is practical: derive and implement LSTM and GRU cells, compare their gradients under stated conditions, then choose an architecture using task accuracy, state memory and measured latency. Required background: the plain recurrent update, sigmoid/tanh derivatives, elementwise products, token cross-entropy and PyTorch modules/autograd. The CPU exercises below require only the already available PyTorch/NumPy runtime. They do not reproduce handwriting, speech, WMT or Shakespeare benchmarks.

LSTM equations and the two-dimensional calculation

For input width D and hidden width H, concatenate the current input and previous hidden state. Three sigmoid gates and one tanh candidate are computed by four learned projections. Calling the candidate a fourth sigmoid gate confuses the architecture.

The dataflow below separates the carried cell from its exposed hidden state. Every multiply is elementwise; all gates/candidate read the same concatenated input and previous hidden vector.

[x_t; h_previous] -> four affine maps -> sigmoid f, sigmoid i, tanh g, sigmoid o
c_previous ------> multiply by f ----> add <---- i multiplied by g
                                      |
                                     c_t ----> tanh ----> multiply by o ----> h_t
                                      |
                                      +---- carried cell input to next step

The forget gate is the fraction retained, not the fraction erased. The input gate controls writing; the output gate controls how much of the cell is exposed. The conveyor-belt analogy is useful if its leak valves remain visible: the belt is multiplied by f, not carried unchanged unconditionally. Hidden state is the view of the belt, not the belt itself.

For the original example, c previous is [0.8,−0.3], f=[0.95,0.02], i=[0.10,0.80] and g=[0.5,0.9]. Retention gives [0.76,−0.006], writing gives [0.05,0.72], so the new cell is [0.81,0.714]. With o=[0.6,0.7], hidden state is approximately [0.401754,0.429225]; compute it rather than rounding intermediate gates. Large cell magnitude can saturate the output tanh even while the direct cell path retains signal.

Peepholes expose cell values directly to gates. The common diagonal variant adds elementwise terms to input/forget gates from the old cell and to the output gate from the new cell. It is not simply a universal dense concatenation with the old cell for every gate. This adds derivative paths and parameters. PyTorch's ordinary LSTM has no peepholes; use a deliberately specified custom cell when precise timing tasks motivate that choice.

Implement both cells and prove the gate convention

Run the Python fences in order, or save them together as recurrent_lab.py. The custom LSTM uses f,i,g,o order. PyTorch uses i,f,g,o with separate input/hidden biases. The original lesson initialized the second block while claiming to initialize f: that was the input gate in its own ordering. Here the first block is initialized and a copied-weight comparison checks the permutation explicitly.

import math, copy
from pathlib import Path
import torch
from torch import nn
from torch.nn import functional as F
 
torch.manual_seed(33)
 
class MyLSTMCell(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super().__init__()
        self.H = hidden_dim
        self.linear = nn.Linear(input_dim + hidden_dim, 4 * hidden_dim)
        with torch.no_grad():
            self.linear.bias.zero_()
            self.linear.bias[:hidden_dim].fill_(1.0)  # f,i,g,o
    def forward(self, x, state):
        h0, c0 = state
        f, i, g, o = self.linear(torch.cat([x, h0], -1)).chunk(4, -1)
        c = f.sigmoid() * c0 + i.sigmoid() * g.tanh()
        h = o.sigmoid() * c.tanh()
        return h, (h, c)
 
def lstm_forward(cell, xs, state=None):
    B, T, _ = xs.shape
    if T == 0:
        raise ValueError('nonempty sequence required')
    if state is None:
        state = (xs.new_zeros(B, cell.H), xs.new_zeros(B, cell.H))
    outputs = []
    for t in range(T):
        h, state = cell(xs[:, t], state)
        outputs.append(h)
    return torch.stack(outputs, 1), state
 
class MyGRUCell(nn.Module):
    """Reset-before candidate; z denotes NEW-candidate fraction here."""
    def __init__(self, input_dim, hidden_dim):
        super().__init__()
        self.H = hidden_dim
        self.rz = nn.Linear(input_dim + hidden_dim, 2 * hidden_dim)
        self.candidate = nn.Linear(input_dim + hidden_dim, hidden_dim)
    def forward(self, x, h):
        r, z = self.rz(torch.cat([x, h], -1)).sigmoid().chunk(2, -1)
        candidate = self.candidate(torch.cat([x, r * h], -1)).tanh()
        return (1-z)*h + z*candidate
 
D, H = 3, 4
cell = MyLSTMCell(D, H).double()
ref = nn.LSTMCell(D, H).double()
order = torch.cat([torch.arange(H,2*H), torch.arange(H), torch.arange(2*H,4*H)])
with torch.no_grad():
    ref.weight_ih.copy_(cell.linear.weight[order, :D])
    ref.weight_hh.copy_(cell.linear.weight[order, D:])
    ref.bias_ih.copy_(cell.linear.bias[order]); ref.bias_hh.zero_()
x = torch.randn(2,D,dtype=torch.float64,requires_grad=True)
h = torch.randn(2,H,dtype=torch.float64,requires_grad=True)
c = torch.randn(2,H,dtype=torch.float64,requires_grad=True)
a, (_, ac) = cell(x,(h,c)); b, bc = ref(x,(h,c))
assert torch.allclose(a,b,atol=1e-12) and torch.allclose(ac,bc,atol=1e-12)
ga = torch.autograd.grad(a.sum()+ac.sum(), (x,h,c), retain_graph=True)
gb = torch.autograd.grad(b.sum()+bc.sum(), (x,h,c))
assert all(torch.allclose(u,v,atol=1e-12) for u,v in zip(ga,gb))
assert sum(p.numel() for p in cell.parameters()) == 4*H*(D+H+1)
assert sum(p.numel() for p in MyGRUCell(D,H).parameters()) == 3*H*(D+H+1)
assert sum(p.numel() for p in ref.parameters()) == 4*H*(D+H+2)
cnew = torch.tensor([.95,.02],dtype=torch.float64)*torch.tensor([.8,-.3]) + torch.tensor([.1,.8],dtype=torch.float64)*torch.tensor([.5,.9])
assert torch.allclose(cnew,torch.tensor([.81,.714],dtype=torch.float64),atol=1e-7)
print('cell forward/input-state gradients and counts: pass', cnew.tolist())

Bundling projections is algebraically equivalent to concatenating separate linear outputs; launch count and runtime depend on the backend. PyTorch need not use this exact concatenated input matmul. There is no guaranteed 2× speedup, and torch.sigmoid is not a numerically different cure for F.sigmoid. Prefer the supported direct API for clarity.

The derivative: direct path versus the whole recurrent system

Treat h and c as separate inputs to a non-peephole cell. Holding h fixed, the partial derivative of new c with respect to old c is diag(f). Through the full unrolled graph, h also depends on earlier c, and the total derivative has additional paths. In matrix notation those include diag(c previous) times the gate Jacobian, diag(g) times the input-gate Jacobian and diag(i) times the candidate Jacobian. Their sizes are not guaranteed to be negligible.

Along the direct path alone, k steps multiply the forget factors. At f=0.99 for 200 steps the multiplier is about 0.134, not 1; at f=0.999 it is about 0.819. At f=0.5 it is roughly 6.22e−61, representable in float64 but below float32's range. A few dozen halvings are small, not automatically floating-point zero. Sigmoid may also round to 1 numerically, despite its real-valued open interval.

Positive forget bias is an empirical initialization choice that increases initial retention, not a logically necessary bootstrapping requirement. At zero input and state, bias 1 yields about 0.731; bias 2 about 0.881. Neither guarantees hundred-step memory. The cell can still have exploding gradients through other paths. Clip the global gradient norm as a guard after correct loss normalization, not as a solution to attenuation.

def surviving_grad(f_value, steps=200):
    c0 = torch.zeros(1,dtype=torch.float64,requires_grad=True)
    c = c0
    for _ in range(steps):
        c = f_value*c
    c.sum().backward()
    return c0.grad.item()
 
for fv in (.5,.9,.99,.999,1.0):
    got = surviving_grad(fv)
    assert math.isclose(got,fv**200,rel_tol=1e-12,abs_tol=1e-70)
    print('frozen direct path',fv,got)
assert surviving_grad(.99) < .14  # rejects "near one means no decay"
 
def input_gradient_profile(kind, length=40):
    torch.manual_seed(7)
    rnn = {'rnn':nn.RNN,'lstm':nn.LSTM,'gru':nn.GRU}[kind](3,8,batch_first=True).double()
    x = torch.randn(1,length,3,dtype=torch.float64,requires_grad=True)
    y,_ = rnn(x)
    y[:,-1].square().sum().backward()
    return x.grad.norm(dim=-1).squeeze(0).tolist()
for kind in ('rnn','lstm','gru'):
    profile=input_gradient_profile(kind)
    assert all(math.isfinite(v) for v in profile)
    print('untrained profile first/last',kind,profile[0],profile[-1])

This measures a particular initialized network and loss, not the old invented nine-orders-of-magnitude advantage. For trained comparisons, keep data, token budget, optimizer and evaluation protocol fixed and repeat seeds. Equal hidden width is not equal parameter count. Save all per-time gradients before choosing a summary; a last-state loss probes a different problem from a loss at every step.

GRU conventions and architecture choice

Our GRU has reset r and update z, candidate tanh(W[x;r⊙h]+b), and h new=(1−z)⊙h+z⊙candidate. Its direct retained-state term is diag(1−z), plus indirect gate/candidate derivatives. PyTorch uses the opposite z interpretation: h new=(1−z)⊙candidate+z⊙h. It also applies reset after the hidden projection in its candidate. Reset-before and reset-after are generally unequal: a diagonal reset does not commute with a dense hidden matrix. Renaming z alone cannot make our class a weight-compatible nn.GRUCell.

One-bias custom counts are 4H(D+H+1) versus 3H(D+H+1). Built-in two-bias counts instead use D+H+2. A 25% matrix-count reduction at equal D/H does not prove faster wall-clock or equal quality. Chung et al.'s comparison includes polyphonic music and speech-signal modeling, not a universal language-model or translation ranking. Start with a simple recurrent baseline where streaming state is valuable, then compare validation performance and deployment cost.

State memory is O(H) per layer for LSTM/GRU; step compute includes O(DH+H²), embeddings and output head. Streaming state avoids storing every past key/value but compresses history. Dense attention offers direct content access to many earlier states, at greater context-dependent cost. Windowed attention can bound cache memory. Batching and fused recurrence can utilize hardware; claiming 99% idle cores is unjustified. No universal 10B-token threshold or 10× transformer-size penalty follows from these equations.

Bidirectional recurrence reads future context and is appropriate for an offline source encoder or sequence labeling, not unrestricted next-token generation. For one layer with equal input width it doubles recurrent parameters; higher stacked layers also receive twice-wide input, so whole-model cost is not simply 2×. A bidirectional classifier can concatenate final forward/backward states, use a linear head and evaluate on a held-out local labeled corpus; it is not valid to report SST-2 accuracy without that corpus and run.

Complete local character comparison

This is a small, complete RNN/LSTM/GRU experiment. By default it trains on a bundled artificial character pattern, not Shakespeare. compare_text(Path(...).read_text()) is the explicit local-only adapter for a corpus the reader already owns. Use a document-level held-out corpus for stronger evaluation; the simple contiguous split below is a teaching baseline. The vocabulary is fitted on training text only; held-out unseen characters are rejected explicitly instead of leaking validation vocabulary. The same recorded mapping is used for both splits and there are no automatic downloads.

class CharRecurrent(nn.Module):
    def __init__(self, vocab, kind, width=16):
        super().__init__()
        self.embed=nn.Embedding(vocab,12)
        self.rnn={'rnn':nn.RNN,'lstm':nn.LSTM,'gru':nn.GRU}[kind](12,width,batch_first=True)
        self.head=nn.Linear(width,vocab)
    def forward(self, ids, state=None):
        out,state=self.rnn(self.embed(ids),state)
        return self.head(out),state
 
@torch.no_grad()
def sample_recurrent(model, prompt, stoi, itos, count=40):
    if not prompt or any(c not in stoi for c in prompt):
        raise ValueError('nonempty in-vocabulary prompt required')
    if not isinstance(count,int) or isinstance(count,bool) or count<0:raise ValueError('nonnegative integer count required')
    previous_mode=model.training;model.eval()
    try:
        ids=torch.tensor([[stoi[c] for c in prompt]])
        logits,state=model(ids)  # consume the WHOLE prompt once
        result=prompt
        generator=torch.Generator().manual_seed(2)
        for _ in range(count):
            token=torch.multinomial(logits[:,-1].softmax(-1),1,generator=generator)
            result+=itos[token.item()]
            logits,state=model(token,state)
        return result
    finally:model.train(previous_mode)
 
def compare_text(text, steps=40, context=12, width=16, sample_count=40):
    if any(not isinstance(v,int) or isinstance(v,bool) or v<1 for v in (steps,context,width)):
        raise ValueError('steps, context and width must be positive integers')
    if len(text)<10*(context+1):raise ValueError('corpus too short')
    cut=int(.8*len(text));chars=sorted(set(text[:cut]));stoi={c:i for i,c in enumerate(chars)}
    if set(text[cut:])-set(chars):raise ValueError('held-out OOV characters')
    ids=torch.tensor([stoi[c] for c in text])
    train,val=ids[:cut],ids[cut:]
    reports={}
    for kind in ('rnn','lstm','gru'):
        torch.manual_seed(33);model=CharRecurrent(len(chars),kind,width=width)
        opt=torch.optim.Adam(model.parameters(),lr=.01)
        rng=torch.Generator().manual_seed(4);history=[]
        for update in range(steps):
            starts=torch.randint(len(train)-context,(8,),generator=rng)
            x=torch.stack([train[s:s+context] for s in starts]);y=torch.stack([train[s+1:s+context+1] for s in starts])
            model.train();opt.zero_grad(set_to_none=True)
            logits,_=model(x);loss=F.cross_entropy(logits.reshape(-1,len(chars)),y.reshape(-1))
            loss.backward();nn.utils.clip_grad_norm_(model.parameters(),1.,error_if_nonfinite=True);opt.step()
            history.append({'update':update+1,'training_batch_nll':float(loss.detach())})
        model.eval();nll_sum=0.;target_count=0
        with torch.no_grad():
            for start in range(0,len(val)-1,context):
                length=min(context,len(val)-1-start)
                vx=val[start:start+length][None];vy=val[start+1:start+length+1][None]
                nll_sum+=F.cross_entropy(model(vx)[0].reshape(-1,len(chars)),vy.reshape(-1),reduction='sum').item()
                target_count+=length
        vl=nll_sum/target_count
        model.zero_grad(set_to_none=True)
        probe=model.embed(val[:context][None]);probe.retain_grad()
        recurrent,_=model.rnn(probe);probe_logits=model.head(recurrent)
        F.cross_entropy(probe_logits[:,-1],val[context:context+1]).backward()
        profile=probe.grad.norm(dim=-1).squeeze(0).tolist()
        reports[kind]={'parameters':sum(p.numel() for p in model.parameters()),'validation_nll':vl,'validation_targets':target_count,'training_history':history,'trained_input_gradient_norms':profile,'sample':sample_recurrent(model,text[:4],stoi,chars,count=sample_count)}
    return reports
 
if __name__=='__main__':
    reports=compare_text('ab cab ba\n'*100)
    assert all(math.isfinite(r['validation_nll']) for r in reports.values())
    print('synthetic character contracts, not corpus benchmark',reports)

For the original 3000-step/width-128 practice, call compare_text(local_text, steps=3000, width=128, sample_count=500) and evaluate all fixed held-out windows, as the function does, retaining valid-token counts. Record NLL per token, repeated-seed variation, parameter counts, timing method and samples at the same temperature. Plot the returned training-batch history against consumed tokens, and run the same fixed whole-split validation loop at chosen checkpoints to add a validation curve; label the two sampling policies. No target losses of 2.0 or 1.5 are promised. Training, gradient profiles and sample fluency answer different questions.

The returned trained_input_gradient_norms probe the trained model, not a newly initialized replacement: retain the embedding outputs' gradient and differentiate the last held-out token loss. This isolates sensitivity to input representations while keeping the discrete token IDs fixed. No universal ratio between architectures follows. For a local chart without another dependency, the SVG exporter below plots each measured training history against consumed tokens. Set RECURRENT_FIGURE_PATH to export; otherwise it only constructs and parses the SVG in memory.

import os
import xml.etree.ElementTree as ET
 
def comparison_svg(reports,context=12):
    histories={k:r['training_history'] for k,r in reports.items()}
    maximum=max(p['training_batch_nll'] for h in histories.values() for p in h)
    last=max(p['update'] for h in histories.values() for p in h)
    lines=['<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 400">',
           '<rect width="720" height="400" fill="white"/>',
           '<text x="40" y="25" fill="black">Training batch NLL (not whole-split validation)</text>']
    for index,(kind,color) in enumerate([('rnn','#075985'),('lstm','#9f1239'),('gru','#166534')]):
        points=' '.join(f"{40+640*p['update']/last:.2f},{340-280*p['training_batch_nll']/max(maximum,1e-12):.2f}" for p in histories[kind])
        lines.append(f'<polyline points="{points}" fill="none" stroke="{color}"/>')
        lines.append(f'<text x="{40+index*130}" y="375" fill="{color}">{kind}</text>')
    lines.append(f'<text x="400" y="375" fill="black">0 to {last*8*context} consumed tokens</text>')
    return '\n'.join(lines+['</svg>'])
 
if __name__=='__main__':
    chart=comparison_svg(reports)
    assert len(ET.fromstring(chart).findall('{http://www.w3.org/2000/svg}polyline'))==3
    if os.environ.get('RECURRENT_FIGURE_PATH'):Path(os.environ['RECURRENT_FIGURE_PATH']).write_text(chart)

Bidirectional offline classification: packed final states

For classification of a complete sentence, both directions may see the whole input. Padding must not become extra recurrent timesteps: a zero embedding does not stop biases or hidden-state updates. Pack by true lengths and concatenate the final forward and reverse states. Taking only the last output position would not select the full reverse summary. Labels below mean a synthetic endpoint relation, not sentiment; a trusted local sentiment corpus needs its own train-only vocabulary, unknown-token policy and held-out labels.

from torch.nn.utils.rnn import pack_padded_sequence
 
class BiGRUClassifier(nn.Module):
    def __init__(self,vocab,width=8):
        super().__init__()
        self.embed=nn.Embedding(vocab,6,padding_idx=0)
        self.rnn=nn.GRU(6,width,batch_first=True,bidirectional=True)
        self.head=nn.Linear(2*width,2)
    def forward(self,ids,lengths):
        if lengths.ndim!=1 or len(lengths)!=len(ids) or (lengths<1).any() or (lengths>ids.shape[1]).any():
            raise ValueError('one valid positive length per example required')
        packed=pack_padded_sequence(self.embed(ids),lengths.cpu(),batch_first=True,enforce_sorted=False)
        _,hidden=self.rnn(packed)
        return self.head(torch.cat([hidden[-2],hidden[-1]],dim=-1))
 
def classification_fixture(seed,n):
    rng=torch.Generator().manual_seed(seed)
    lengths=torch.randint(3,7,(n,),generator=rng)
    ids=torch.zeros(n,6,dtype=torch.long)
    for i,length in enumerate(lengths):ids[i,:length]=torch.randint(1,5,(int(length),),generator=rng)
    labels=(ids[:,0]>ids[torch.arange(n),lengths-1]).long()
    return ids,lengths,labels
 
torch.manual_seed(133);classifier=BiGRUClassifier(5)
bx,bl,by=classification_fixture(1,32);tx,tl,ty=classification_fixture(2,16)
optimizer=torch.optim.Adam(classifier.parameters(),lr=.02)
for _ in range(40):
    optimizer.zero_grad(set_to_none=True)
    classification_loss=F.cross_entropy(classifier(bx,bl),by)
    classification_loss.backward();optimizer.step()
classifier.eval()
with torch.no_grad():
    predictions=classifier(tx,tl)
    # Appending/changing padded tokens cannot affect any valid packed sequence.
    padded=F.pad(tx,(0,3),value=4)
    torch.testing.assert_close(classifier(padded,tl),predictions)
    assert predictions.shape==(16,2) and torch.isfinite(predictions).all()
    print('synthetic classification only',{'nll':F.cross_entropy(predictions,ty).item(),
          'correct':int((predictions.argmax(-1)==ty).sum()),'total':len(ty)})

The packed one-layer bidirectional GRU is two directional GRUs with independent parameters. An independent check copies each direction into its own GRU, reverses each valid sequence (not its padded suffix), and verifies the concatenated states. This implements the original stretch mechanism without pretending a synthetic task measures SST-2 accuracy or bidirectionality's causal benefit.

Residual relatives, normalization and modern recurrence

Highway Networks use a transform gate and complementary carry gate across depth; the formula T⊙H(x)+(1−T)⊙x resembles a GRU interpolation. ResNet uses an ungated sum x+F(x), with Jacobian I+J_F, not a guarantee that the total derivative equals I. A transformer pre-norm block also adds branches; the original 2017 block used post-norm, not the pre-norm equations in the old recap. These are related structural ideas, not proof that every residual network is a direct historical descendant or that all multiplication destroys gradients. Orthogonal multiplication, for example, preserves Euclidean norms.

Layer normalization in recurrence is not categorically forbidden on the cell. Ba et al.'s LSTM formulation normalizes gate preactivations and the cell on the output path. Placement changes the dynamics; implement the selected equations and compare gradients instead of treating a fictional personal anecdote as evidence. The ordinary built-in nn.LSTM does not include LN. A zero forget bias is not a framework bug, and unsorted packed sequences do not imply a pure-Python fallback. Profile the actual device, dtype, shapes and packing configuration; no GPU/cuDNN experiment ran here.

xLSTM (2024) explores exponential gating, stabilized/normalized memory and scalar versus matrix memory variants. Scalar memory allows memory mixing; matrix memory increases storage capacity with covariance-style updates and supports a parallelizable variant. Neither is just the classic LSTM with one renamed gate. xLSTM-7B has its own later report; it must not be dated from the 2024 abstract. Mamba uses input-dependent selective state-space updates; Mamba-2 develops structured state-space duality; RWKV-7 uses generalized delta-rule state evolution. Griffin mixes gated recurrence and local attention, while Jamba combines Mamba, attention and MoE. Hybrid attention caches still grow where full attention is used. These are distinct mechanisms, not evidence of a universal recurrence victory or of private frontier architectures.

Practice and worked answers

1. Reconstruct the cell. Compute f,i,o with sigmoid and g with tanh from input/hidden projections, then c=f⊙c previous+i⊙g and h=o⊙tanh(c). The code's order is f,i,g,o; its first bias block is the forget block. Preserve both state tensors across steps.
2. Does the highway prevent decay? Only the direct partial is diag(f); its product is the retained-path multiplier. At 0.99 for 200 steps it is approximately 0.134. Indirect paths and output saturation remain. This corrects the original question's guarantee.
3. Work the example. The answer is [0.81,0.714]. The second dimension is mostly rewritten; the first mostly retained. Multiplying by the output gate after tanh gives a different hidden vector, not another copy of c.
4. Write GRU and count. Our reset-before GRU uses two gates, three projections and h new=(1−z)h+z candidate. Its one-bias count is 3H(D+H+1), versus 4H(D+H+1) for LSTM. Built-in reset/order and bias conventions must be reconciled separately.
5. Name the related skips. Highway has sigmoid gating, ResNet has ungated residual addition, and transformer blocks wrap attention/FFN with residuals. They are not the same equation, and addition provides a path rather than an unconditional training theorem.
6. Explain xLSTM. Exponential gating changes how stored information can be revised; matrix memory expands the memory representation, with stabilization/normalization and a parallelizable variant. Read its own equations before equating it with PyTorch LSTM or a particular 7B release.

The full local three-model comparison and bidirectional offline classification model above provide runnable starting points. Use source-length and parameter-matched alternatives, report rather than invent rankings, and revisit the equations tomorrow and after a week.

Sources, optional navigation and execution limits

Primary reading: 1997 LSTM, forget-gate and peephole discussion, Jozefowicz et al. 2015, Chung et al., PyTorch 2.9 LSTM, PyTorch GRU convention, LayerNorm, xLSTM. Exact passages and raw hashes accompany the review ledger; mathematical/CPU checks are separate evidence.

Optional original resources remain suggestions, not primary certification: StatQuest LSTM, Michael Phi, Olah. Previous: RNN · Next: seq2seq.

Executed scope is CPU cell/gradient arithmetic, bundled synthetic training, trained input-gradient profiles, loss-chart construction and packed bidirectional synthetic classification. Optional video title/channel/duration/segment attributions and the Olah recommendation were not independently verified. Local Shakespeare/SST-2/translation training, pretrained quality, GPU kernel speed and application release remain unexecuted; none is inferred from these contracts.