Attention Intuition — Sequence Bottlenecks and Soft Retrieval

Why this session exists

Attention lets a query retrieve a weighted mixture from retained representations. It relaxes the fixed-summary interface of a plain encoder-decoder and, when used without recurrent state transitions inside a layer, permits parallel processing of known query positions. Those are distinct benefits. Attention is not the whole Transformer, and recurrence remains useful in streaming systems. This lesson connects the retrieval metaphor to actual shapes, masks, alignment examples and falsifiable tests.

Prerequisites

S103 recurrence, S104 embeddings and S106 token IDs. Know weighted sums and softmax; S108 develops the backward pass.

(a) Intuition · 5 min

A translator can glance back at the source rather than memorize it and look away. The computational counterpart is a decoder query selecting from all encoder states. A fixed-width state can encode useful long inputs; there is no theorem that it cannot carry a 30- or 100-word sentence. The practical problem is lossy compression and difficult learning through long paths, not an absolute word-count capacity limit.

The three ideas that make attention work

Soft lookup: score each key, normalize over legal keys, blend values. Unlike a hash-table lookup, dense attention usually compares against every key and is not constant-time. Content routing: query/key representations influence weights; positional biases and masks can also influence them. Parallel query processing: once queries/keys/values are known, many comparisons and reductions can run together. Hardware does not literally execute all pairs simultaneously, and an RNN producing one new query per step remains recurrent even if it has attention.

A short history

Seq2seq (2014) uses a fixed-vector encoder-decoder. Bahdanau et al. (2014/2015) jointly learn alignment and translation with a soft search over source annotations. Later scoring variants include dot-product and learned multiplicative forms. The Transformer (2017) uses self-attention and encoder-decoder attention instead of recurrent/convolutional sequence transitions, plus feed-forward layers, residuals, normalization and positional signals. BERT and GPT-style models use different attention visibility and objectives; neither historical contribution means 'only attention' is a full model or every later architecture abandoned recurrence.

The seq2seq bottleneck

The decoder only accesses source details through the chosen summary. In an LSTM the transferred state may include multiple hidden/cell states, not literally only one h_T; the fixed-summary limitation still describes the interface. Earlier information is not simply thrown away at every step, but must survive the learned compression.

The fix: attention over all encoder states

For additive attention, a typical score is $e_{tj}=v_a^T\tanh(W_s s_{t-1}+W_h h_j+b)$, followed by $a_{tj}=\operatorname{softmax}_j(e_{tj})$ over valid source positions and $c_t=\sum_j a_{tj}h_j$. Decoder state/previous-token handling belongs to the surrounding recurrence. This removes exclusive reliance on one final summary but does not remove every information or optimization bottleneck. The retained source sequence costs memory, and a context vector is still a compressed read for each query.

The three roles: query, key, value

A query asks which records to use, a key supplies matching features, a value supplies the routed content. In projected self-attention, $Q=XW_Q$, $K=XW_K$, $V=XW_V$. In cross-attention, queries come from one representation sequence and keys/values from another. These roles are structural, not automatically human questions and answers. Hard argmax selection is not ordinarily differentiable with respect to its selection scores; soft attention provides a smooth route, although alternative estimators and routing mechanisms exist.

The three flavours: cross-attention, self-attention, causal self-attention

Self-attention does not inherently mean bidirectional: its mask can be causal, local or otherwise constrained. Cross-attention is defined by source roles, not merely unequal sequence lengths. Original CLIP is a dual-encoder contrastive model, not a canonical example of patch/text cross-attention; conditional denoisers and encoder-decoder models can use that operation.

Anatomy of one attention head

Project → score $QK^T$ → divide by $\sqrt{d_k}$ → apply visibility/padding masks before normalization → softmax across keys → multiply by $V$. The head output width is $d_v$, so it need not equal input width. For each valid row before dropout, the output is a convex combination of projected values. Output projection, residual addition and later nonlinearities are outside that convex-hull statement; attention dropout can also make sampled row sums differ from one.

Under independent, zero-mean, unit-variance query/key components, the dot-product variance is $d_k$ and scaling restores unit variance. LayerNorm does not itself guarantee independence after learned projections, and tied Q=K self-scores violate the assumption. Larger logits may saturate softmax; subtracting the row maximum prevents exponent overflow but does not undo statistical saturation. Long sequences increase the number of scores, not automatically each score's variance.

What an attention pattern actually looks like

Here is a constructed, not model-measured, English-to-French alignment for source the black cat and target le chat noir. Rows are target queries, columns are source positions:

Every row sums to one; adjective order can differ without a fixed positional matching rule. With source values (1,0), (0,1), (2,0), the chat context is (1.7,0.1). Real learned annotations contain contextual information, so even a sharp alignment is not a one-to-one word dictionary.

A random head on The cat sat on the mat can produce a nonuniform matrix but has learned no syntax. Different learned heads may emphasize nearby positions, entities or separators; specialization is empirical and may be redundant. An attention map alone is not a causal explanation: a high weight on a zero value contributes zero; residual and other heads can dominate. Test interventions and downstream effects, not just a colorful heatmap.

Order, path length and architecture choice

For unmasked position-free self-attention and permutation matrix $P$, $f(PX)=Pf(X)$: equivariance, not identical ordered outputs. Permuting K and V together leaves a fixed query's weighted result unchanged. Pooling can make a sequence representation invariant. A fixed causal mask breaks arbitrary permutation symmetry by encoding an ordering relation even without explicit position embeddings. Position signals such as additive embeddings or RoPE provide other order information; they do not guarantee extrapolation.

Dense attention has pairwise arithmetic $O(T^2d)$, plus projection work $O(Td^2)$. Naively stored weights use $O(T^2)$ per head; FlashAttention changes memory traffic/storage, not dense arithmetic to linear. Recurrence offers fixed-width incremental state but compressed history; temporal convolutions offer a bounded receptive field and parallel positions. Choose by deployment shape, observed dependency range and budget, not a mandate to use attention unless a stream is infinite.

(c) Hands-on · 25 min

The code prints a six-token random-initialization matrix and checks single-head attention against PyTorch 2.9 SDPA. Causal versus noncausal comparisons reuse the same parameters, unlike a comparison that changes both mask and model. Tests prove prefix isolation, padding behavior, permutation equivariance and the attribution counterexample. No language model is trained.

import math,time,statistics
import torch
from torch import nn
from torch.nn import functional as F
 
torch.set_num_threads(1);torch.manual_seed(17)
class SingleHeadAttention(nn.Module):
    def __init__(self,d_model):
        super().__init__()
        self.W_q=nn.Linear(d_model,d_model,bias=False)
        self.W_k=nn.Linear(d_model,d_model,bias=False)
        self.W_v=nn.Linear(d_model,d_model,bias=False)
    def forward(self,x,causal=False,valid=None):
        if x.ndim!=3 or x.shape[1]==0:raise ValueError('nonempty B,T,D required')
        q,k,v=self.W_q(x),self.W_k(x),self.W_v(x);B,T,D=q.shape
        allowed=torch.ones(B,T,T,dtype=torch.bool,device=x.device)
        if causal:allowed &= torch.ones(T,T,dtype=torch.bool,device=x.device).tril()
        if valid is not None:allowed &= valid[:,None,:]
        if not allowed.any(-1).all():raise ValueError('every query must have a valid key')
        scores=(q@k.transpose(-2,-1))/math.sqrt(D)
        a=scores.masked_fill(~allowed,-torch.inf).softmax(-1)
        return a@v,a
 
words=['The','cat','sat','on','the','mat'];vocab={w:i for i,w in enumerate(sorted(set(words)))}
embed=nn.Embedding(len(vocab),8).double();m=SingleHeadAttention(8).double()
x=embed(torch.tensor([[vocab[w] for w in words]]));out,a=m(x)
print('random-init routing only',a[0].detach().round(decimals=3))
assert out.shape==x.shape and a.shape==(1,6,6)
assert torch.allclose(a.sum(-1),torch.ones(1,6,dtype=torch.double))
q,k,v=m.W_q(x),m.W_k(x),m.W_v(x)
assert torch.allclose(out,F.scaled_dot_product_attention(q,k,v,dropout_p=0.),atol=1e-12)
causal,ac=m(x,causal=True);assert torch.count_nonzero(ac.triu(1))==0
assert ac[0,0,0]==1
changed=x.detach().clone();changed[:,3:]+=100
future,_=m(changed,causal=True)
assert torch.allclose(causal[:,:3],future[:,:3],atol=1e-12)
# Negative control: unmasked attention is affected by future inputs.
leaky,_=m(changed);assert not torch.allclose(out[:,:3],leaky[:,:3])
perm=torch.tensor([2,0,5,1,4,3]);permuted,_=m(x[:,perm])
assert torch.allclose(permuted,out[:,perm],atol=1e-12)
valid=torch.tensor([[True,True,True,True,False,False]])
padded,ap=m(x,valid=valid);assert torch.count_nonzero(ap[:,:,4:])==0
changed=x.detach().clone();changed[:,4:]+=100
assert torch.allclose(padded[:,:4],m(changed,valid=valid)[0][:,:4],atol=1e-12)
try:m(x,valid=torch.zeros(1,6,dtype=torch.bool));raise AssertionError('all-masked accepted')
except ValueError:pass
alignment=torch.tensor([[.8,.1,.1],[.1,.1,.8],[.1,.8,.1]],dtype=torch.double)
values=torch.tensor([[1.,0.],[0.,1.],[2.,0.]],dtype=torch.double)
assert torch.allclose((alignment@values)[1],torch.tensor([1.7,.1],dtype=torch.double))
# High attention mass is not the same as high value contribution.
weights=torch.tensor([[.99,.01]]);values=torch.tensor([[0.,0.],[10.,20.]])
assert torch.allclose(weights@values,torch.tensor([[.1,.2]]))
# Bounded local CPU forward timings, not a GPU or architecture-quality benchmark.
z=torch.randn(1,32,8,dtype=torch.double);rnn=nn.LSTM(8,8,batch_first=True).double().eval();m.eval()
with torch.no_grad():
    for label,fn in [('lstm',lambda:rnn(z)),('attention',lambda:m(z,causal=True))]:
        for _ in range(3):fn()
        times=[]
        for _ in range(5):
            t=time.perf_counter();fn();times.append(time.perf_counter()-t)
        print(label,'CPU fixture median seconds',statistics.median(times))
print('SDPA, causal/padding, equivariance and attribution checks passed')

Anatomy of the script

The last score axis indexes keys; row normalization implements one distribution per query. masked_fill(~allowed,-inf) uses an allowed-mask convention, whereas other APIs may use True to mean blocked. Every row must have a legal key in this reference implementation. Padding masks remove keys; padded query outputs still exist and must be ignored by the task loss/readout. The diagonal is legal for next-token prediction when input/target sequences are shifted correctly.

The CPU timing fixture includes returned weights, uses small shapes, eval and no-grad, warms up and reports a median. It is intentionally not evidence that one architecture trains faster or produces better text. A real hardware benchmark must match shapes, dtype, backward/forward work, kernels and measurement policy, synchronize accelerators where needed, and compare quality separately. Merely timing two unrelated randomly initialized modules does not 'prove parallelism'; the computation graph establishes available parallelism.

(d) Production reality · 15 min

Translation: teacher-force shifted decoder inputs exactly once, mask source and target padding, and define EOS/length limits. Attention supplies per-step source context but the decoder remains sequential at generation. Beam search changes search, not learned conditional probabilities; length penalties alter ranking. BLEU needs corpus, references and tokenization. GNMT documents a deep recurrent system with attention; it does not verify the earlier invented deployment-switch dates or latency percentages.

Long context: FlashAttention (2022) is exact dense softmax attention with IO-aware tiling and online normalization. It cannot have rescued GPT-3's 2020 training as the earlier story implied. Sliding windows change visibility, sparse methods change which pairs are computed, GQA shares K/V heads and reduces cache, and retrieval reduces supplied context while risking missed evidence. They solve different costs; none proves a named closed model uses a particular combination.

Interpretability: the 2022 induction-head study gives mechanistic/ablation evidence strongest in small attention-only models and more qualified evidence at larger scales. It is not a paper about particular Claude system-prompt heads preventing hallucination. A head map suggests a hypothesis; controlled ablation/patching and alternative explanations test it.

Where this shows up in the rest of the plan

S108 derives scores/gradients; S109 separates multiple routing distributions. Full blocks add FFNs, residuals and normalization. Positional signals modify what scores can represent. Conditional diffusion can query text, whereas original CLIP uses independent encoders aligned by a contrastive objective.

(e) Recall + stretch · 10 min

1. Bottleneck? A fixed source-summary interface forces compression; attention offers access to all retained source states without guaranteeing perfect recall.
2. Soft lookup? Score keys with a query, normalize over legal keys, sum weighted values; not a hash table's complexity.
3. Why scale? Independent component products have variance $d_k$; divide by its square root under that assumption.
4. RNN attention versus self-attention? Different sources/architectural recurrence. Self-attention alone is not a complete Transformer.
5. Parallelism? Known queries can be processed together within a layer; autoregressive outputs still depend on previous generated tokens.
6. Causal mask? Exclude future keys before softmax and shift targets correctly; check that changing the future cannot affect prefixes.
7. Three heads? Separate projections permit different patterns but may learn redundancy; random differences are not semantic specialization.
8. Complexity? Dense pairwise $O(T^2d)$; projections add $O(Td^2)$; materialization and IO depend on algorithm.
9. No positions? Unmasked attention is permutation-equivariant; fixed causal masks already impose an ordering constraint. It does not yield identical ordered outputs under arbitrary swaps.
10. 100K tokens? Budget exact-attention tiling, local/sparse visibility, K/V sharing and retrieval separately; quantify lost access and cache bytes rather than assert a vendor recipe or no-OOM guarantee.

Explain-out-loud test

Trace one row of the French alignment, explain why a 99% weight can contribute nothing, and distinguish training-position parallelism from token-generation sequentiality.

What comes next

Q/K/V math makes the derivatives and numerical contracts explicit. Learning-plan hub.

Primary references

• Bahdanau attention and Transformer: soft alignment and parallel sequence layers.
• FlashAttention: exact IO-aware attention, not linear pairwise compute.
• Induction heads: scoped mechanistic evidence.