Embeddings — word2vec, GloVe, Contrastive Learning

Why this session exists

An embedding is a learned vector representation, not a certificate of meaning. The loss, data and sampling policy determine which distinctions its geometry preserves. We will connect one-hot indexing to gradients, derive skip-gram negative sampling and GloVe, work cosine similarity by hand, and train/query a small CPU embedding model. Contrastive image–text objectives extend the representation idea without being identical to word2vec's binary loss.

Prerequisites

S100 PyTorch, S098 backpropagation and dot products/norms. S097 uses the same affine-map conventions; it is not a substitute title for a linear-algebra session.

(a) Intuition · 5 min

A restaurant map groups places according to useful attributes, but changing the task from cuisine to delivery time changes the useful map. Likewise a learned coordinate system can emphasize topic, substitutability, sentiment or relevance. Coordinates need not correspond to named human attributes. A rotation of both sides of a dot-product model can preserve scores while changing every axis; interpreting a single dimension needs evidence, not an assumption that all axes are either meaningful or noise.

Why dense vectors beat one-hot

Distinct one-hot vectors have cosine zero and equal Euclidean distance. That raw representation contains identity, not a built-in semantic ordering. But one-hot followed by a learned matrix is an embedding lookup: $e_i^TE=E_i$. Both use $Vd$ parameters; 50,000×300 is 15 million, not a 100× parameter saving over the same one-hot linear layer. Indexing saves materializing the one-hot activation and doing a sparse selection as a dense multiply.

Dense vectors let later layers share statistical strength and can support analogies, but neither neighborhood nor arithmetic is guaranteed. king - man + woman is a hypothesis to test with excluded input words and a stated corpus/model, not a universal semantic law. Sparse lexical features remain competitive for exact identifiers and some retrieval tasks.

The embedding revolution timeline

Distributional vectors and neural language-model embeddings predate 2013. word2vec (2013) developed efficient CBOW/skip-gram recipes, negative sampling and frequent-word subsampling. GloVe (2014) uses global co-occurrence counts with weighted regression. ELMo and BERT produce contextual representations rather than one vector per word type. SimCLR learns from augmented image pairs; CLIP learns paired image/text representations. These are different objectives and architectures, not proof that contrastive learning is the only modern recipe (for example, self-distillation need not use explicit negatives).

The skip-gram architecture

For a center $c$ and context $o$, full-softmax skip-gram uses

The two tables encode different roles; keeping the center table after training is a common export choice, not a mathematical requirement. CBOW instead aggregates context to predict the center. Full softmax is feasible for small vocabularies and supplies a normalized conditional distribution; negative sampling changes the objective rather than computing that distribution exactly.

Negative sampling — the trick that made it fast

With $K$ independent noise draws $n_k\sim q$, the per-positive loss is

Average over positive pairs after summing negatives. Averaging all $B K$ negative terms while averaging $B$ positive terms underweights the negative objective by $K$—the bug in the earlier lab. At all-zero scores, the correct loss is $(K+1)\log2$, not $2\log2$. Stable logsigmoid avoids explicitly logging near-zero probabilities. For positive score $s_+$ the derivative is $\sigma(s_+)-1$; for negative score $s_-$ it is $\sigma(s_-)$. Thus $\partial\ell/\partial v_c=(\sigma(s_+)-1)u_o+\sum_k\sigma(s_k)u_{n_k}$.

The original paper evaluates noise proportional to count raised to 3/4. It increases rare words' relative probability versus raw unigram, not their probability above all frequent words. This is empirical, not a universal optimum. Accidental positives and repeated negative IDs can occur under independent sampling; resampling them defines a different noise policy. Lookup gradient contributions accumulate for repeated IDs even though nn.Embedding stores a dense gradient tensor by default; sparse=True changes gradient storage and limits compatible optimizers. The lab uses the default dense gradients with Adam. Optimizer momentum/decay, tied output tables and full-softmax context gradients can update rows beyond the naive 'only visible words ever change' story.

Walking through a training step

Select a center and context within a declared window; lookup both roles; sample negatives; compute the binary loss; accumulate gradients for repeated rows; update. If radius is uniform from 1 through $W$, a context at distance $r\le W$ is included with probability $(W-r+1)/W$. Preserve original positions when filtering vocabulary: removing an unknown token must not silently make distant words adjacent. Respect sentence/document boundaries and split the corpus before vocabulary fitting. Frequent-word subsampling is a separate optional policy; it is not implemented by simply discarding every rare word.

word2vec vs GloVe vs BERT vs CLIP

GloVe minimizes a weighted regression over nonzero co-occurrences:

The paper's weighting saturates at one, with $f(x)=(x/x_{max})^\alpha$ below the cutoff. Biases and weighting matter; it is not unweighted factorization of every log(0) entry. Co-occurrence ratios motivate relation directions, but low rank and finite data make exact analogy equations unlikely.

BERT produces occurrence-dependent token states; its initial lookup is still static, and subwords complicate 'one vector per word'. A sentence retrieval model additionally needs a pooling and training objective appropriate to query/document relevance. Averaged static vectors discard word order but are not literally meaningless; test them as a baseline, especially for short lexical queries.

For normalized paired image/text embeddings, CLIP forms $S_{ij}=z_i^Tt_j/\tau$ and minimizes the mean of image-to-text and text-to-image cross entropy with diagonal matches. InfoNCE-style losses normalize over candidates; SGNS uses independent binary terms. SimCLR's two views need different augmentation assumptions, and false negatives can arise when different records describe the same thing. Contrastive quality depends on those choices. A CLIP dual encoder is not itself cross-attention between every image patch and text token.

Cosine similarity — the metric that runs the show

For $(3,4)$ and $(4,3)$, dot = 24 and both norms = 5, hence cosine = 0.96. For unit vectors $\|a-b\|^2=2-2a^Tb$: cosine and squared-Euclidean rankings agree. A zero vector has undefined cosine; an epsilon convention is a numerical fallback, not a geometric angle. Norm may encode frequency, confidence or another learned quantity; stripping it is a modeling choice. Retrieval indexes must use the same normalization/metric contract as the query encoder, not an assumed vendor default.

Geometry, ambiguity and retrieval choices

Words with similar contexts can include antonyms: good and bad are not logically equivalent even if close in a particular distributional model. A static bank row conflates senses; context can help but does not guarantee disambiguation. Rare tokens receive fewer observations, yet need not remain random; subwords share evidence but do not guarantee every piece is well trained. Separately trained coordinate systems cannot normally be mixed without alignment. Version model, tokenizer, pooling, normalization and index together.

Start retrieval with a pretrained baseline plus lexical retrieval and held-out relevance judgments. Fine-tune when domain pairs and measured errors justify it; audit click/popularity bias and false negatives. Train ID embeddings from interaction data when no linguistic pretrained representation exists, with a content-based cold-start path. New model versions usually require re-embedding or a validated alignment—not blindly treating a changed vector width as the only compatibility issue. Plotting t-SNE/UMAP neighborhoods is exploratory and distorts distances; evaluate retrieval metrics on relevant query slices.

(c) Hands-on · 25 min

This CPU fixture trains SGNS on inline sentences and answers nearest-neighbor/analogy queries. It checks the actual loss, repeated-ID gradients and short batches. Printed neighbors are toy outcomes; no royalty analogy, Wikipedia quality or training speed is promised.

import math,random,re
from collections import Counter
import torch
from torch import nn
from torch.nn import functional as F
 
torch.set_num_threads(1);torch.manual_seed(5)
rng=random.Random(5)
class SkipGram(nn.Module):
    def __init__(self,vocab,dim):
        super().__init__();self.center=nn.Embedding(vocab,dim);self.context=nn.Embedding(vocab,dim)
        nn.init.uniform_(self.center.weight,-.5/dim,.5/dim);nn.init.zeros_(self.context.weight)
    def forward(self,c,p,n):
        v=self.center(c);u=self.context(p);neg=self.context(n)
        pos=(v*u).sum(-1);scores=torch.einsum('bkd,bd->bk',neg,v)
        return (-F.logsigmoid(pos)-F.logsigmoid(-scores).sum(-1)).mean()
 
def pairs(sentences,stoi,window=2):
    result=[]
    for words in sentences:
        for i,w in enumerate(words):
            if w not in stoi:continue
            radius=rng.randint(1,window)
            for j in range(max(0,i-radius),min(len(words),i+radius+1)):
                if j!=i and words[j] in stoi:result.append((stoi[w],stoi[words[j]]))
    if not result:raise ValueError('no usable context pairs')
    return result
 
def query(emb,stoi,v,exclude=(),top=5):
    if v.norm()<1e-12:return []
    sims=emb@F.normalize(v,dim=0);allowed=[i for i in range(len(emb)) if i not in set(exclude)]
    if top<=0:return []
    ranked=sorted(allowed,key=lambda i:float(sims[i]),reverse=True)[:top]
    words={i:w for w,i in stoi.items()};return [words[i] for i in ranked]
 
def analogy(emb,stoi,a,b,c,top=5):
    if any(w not in stoi for w in (a,b,c)):return []
    ia,ib,ic=[stoi[w] for w in (a,b,c)]
    return query(emb,stoi,emb[ia]-emb[ib]+emb[ic],(ia,ib,ic),top)
 
sentences=[s.split() for s in ['cat eats fish','dog eats meat','cat likes milk',
    'dog likes walks','queen rules kingdom','king rules kingdom','woman reads book','man reads book']]*20
counts=Counter(w for s in sentences for w in s);vocab=sorted(counts);stoi={w:i for i,w in enumerate(vocab)}
weights=torch.tensor([counts[w]**.75 for w in vocab]);weights/=weights.sum()
ps=pairs(sentences,stoi);m=SkipGram(len(vocab),12);opt=torch.optim.Adam(m.parameters(),lr=.03)
K=4
for epoch in range(5):
    rng.shuffle(ps);total=nseen=0
    for start in range(0,len(ps),31):
        batch=ps[start:start+31];c,p=map(torch.tensor,zip(*batch))
        neg=torch.multinomial(weights,len(batch)*K,replacement=True).reshape(len(batch),K)
        opt.zero_grad(set_to_none=True);loss=m(c,p,neg);loss.backward();opt.step()
        total+=loss.item()*len(batch);nseen+=len(batch)
    print(epoch,'toy SGNS objective',total/nseen)
emb=F.normalize(m.center.weight.detach(),dim=-1)
print('cat neighbors',query(emb,stoi,emb[stoi['cat']],(stoi['cat'],)))
print('toy analogy',analogy(emb,stoi,'king','man','woman'))
assert analogy(emb,stoi,'unknown','man','woman')==[]
assert len(query(emb,stoi,emb[0],range(len(vocab)-1),100))==1
z=SkipGram(3,2)
with torch.no_grad():z.center.weight.zero_();z.context.weight.zero_()
c=torch.tensor([0,0]);p=torch.tensor([1,1]);n=torch.tensor([[2,2,2,2],[2,2,2,2]])
assert abs(z(c,p,n).item()-(K+1)*math.log(2))<1e-6
wrong=2*math.log(2);assert abs(z(c,p,n).item()-wrong)>1
# Repeated IDs accumulate: two identical positive samples have the same mean gradient as one.
z=SkipGram(3,2).double()
with torch.no_grad():z.context.weight.normal_()
a=z(c,p,n);a.backward();grad=z.center.weight.grad.clone();z.zero_grad()
z(c[:1],p[:1],n[:1]).backward();assert torch.allclose(grad,z.center.weight.grad)
assert torch.allclose(F.cosine_similarity(torch.tensor([[3.,4.]]),torch.tensor([[4.,3.]])),torch.tensor([.96]))
# Symmetric in-batch contrastive loss has uniform-logit baseline log(B).
image=F.normalize(torch.randn(4,6),dim=-1);text=F.normalize(torch.randn(4,6),dim=-1)
s=image@text.T/.2;labels=torch.arange(4)
contrast=(F.cross_entropy(s,labels)+F.cross_entropy(s.T,labels))/2
assert torch.isfinite(contrast)
assert abs(F.cross_entropy(torch.zeros(4,4),labels).item()-math.log(4))<1e-6
print('SGNS reduction, repeated-ID, query and contrastive oracles passed')

Anatomy of the script

The dynamic window is drawn per center, not per pair. Full and short minibatches contribute, avoiding division by zero when a corpus is smaller than a fixed batch. Positive and negative tables are separate; einsum makes the (B,K,D) contraction explicit. Export normalizes once, excludes the query/input IDs, and bounds top by available candidates. Analogy arguments may repeat; missing words or zero resultant vectors have explicit behavior. The fixture's regex-free token lists are deliberately simple; a real corpus needs documented Unicode/case and sentence-boundary rules.

For a real retrieval experiment, fit vocabulary and sampling counts on training documents only, freeze a model version, then evaluate held-out labeled relevance with recall@k/MRR/nDCG and lexical baselines. Training SGNS loss is not a probability perplexity or an assertion of semantic quality. Saving an embedding requires both table and ID mapping plus normalization/objective/version metadata; this snippet deliberately creates no persistent checkpoint.

(d) Production reality · 15 min

Listing/session recommendations: treating items as tokens can learn co-view neighborhoods. Geographic availability and marketplace choice sets should constrain candidate generation and may shape negative sampling. Compare same-market and global noise under an honest temporal/user split; the earlier Airbnb failure narrative and +21% conversion claim are not established here.

Music cold start: an ID table cannot give an unseen item a learned interaction history. Audio/text encoders can supply content features; combine or align them with interaction representations under the task loss. Merely concatenating arbitrary spaces changes dimensionality and metric weighting; validate modality scaling and missing-modality policy. No specific Spotify 4096-dimensional architecture is asserted.

CLIP: the primary paper describes symmetric in-batch cross entropy and compares predictive/contrastive objectives under its experiments. That supports a representation-learning design, not a universal claim that contrastive training is 12× better than generation, or that DINO uses the same negative-sampling loss. Audit pair quality, duplicate images/captions, harmful bias and domain-shift performance separately from image–text alignment.

Where this shows up in the rest of the plan

Tokenization defines the row IDs; attention transforms token states. Retrieval and vector databases store representations under a versioned metric. Recommenders may use dot products, nonlinear rankers or other methods—not every system is just lookup plus dot product. CLIP pairs modalities through a shared comparison space, which is distinct from token-level cross-attention.

(e) Recall + stretch · 10 min

1. One-hot useless? It encodes identity; a learned following matrix is exactly a lookup. Raw one-hot lacks graded geometry.
2. Skip-gram? Predict contexts from a center; CBOW reverses those roles.
3. Negative sampling? $K+1$ binary terms replace vocabulary normalization with a different objective; no universal speed factor.
4. Cosine of (3,4), (4,3)? 24/25 = 0.96; magnitude-normalized directions differ.
5. Static versus contextual? One table row per token type versus occurrence-dependent states; downstream pooling still matters.
6. GloVe? Weighted log-count regression with two tables and biases over observed counts, not naive log-zero factorization.
7. Contrastive learning? Define positive views and candidate negatives, score them with temperature, normalize and optimize. False negatives and augmentation invariance are assumptions.
8. Normalize? To use angular similarity; do not discard informative norms without evaluation.
9. Ten-million-sentence search? Benchmark sentence-retrieval encoders, sparse retrieval and averaging under actual queries, negation and identifiers; choose quality/storage/latency together.
10. Why analogies? Some learned relation offsets can align under distributional objectives. Geometry is not guaranteed, and selected demonstrations must exclude inputs and report failures.

Explain-out-loud test

Derive the SGNS score gradients, show why averaging negatives changes the objective, and explain a high-cosine counterexample to factual equivalence.

What comes next

Transfer learning adapts reusable representations while managing state and evaluation boundaries. Learning-plan hub.

Primary references

• word2vec negative-sampling paper: objective, sampling and subsampling.
• GloVe: weighted co-occurrence objective.
• CLIP: paired encoders and symmetric contrastive loss.
• PyTorch 2.9 Embedding source: lookup, gradients and padding options.