Word2Vec: complete skip-gram training, geometry and honest evaluation

An embedding is a learned table indexed by discrete IDs. Word2Vec makes that table useful by training a word to predict its context,or context to predict a word. The striking part is not that every analogy works: it is that a predictive objective can create useful regularities in a compact representation.

Distributional learning without the mythology

Mikolov et al.'s2013 papers develop efficient CBOW/skip-gram models and then negative sampling,subsampling and phrase representations. They are not evidence for the original lunch-napkin/Jeff-Dean dialogue,citation-count story or claim that learned embeddings were invented at that lunch. Neural language-model embeddings predate Word2Vec. Nor is “swimmed” the past tense that a correct morphology exercise should expect.

The distributional hypothesis says contextual patterns can be informative about meaning. In “the furry ___ purred,”cat and kitten may fit related contexts. But antonyms also share contexts,and one spelling such as bank has multiple senses. A static row compresses those uses;it is not a dictionary or a factual-entailment engine.

A springs picture is only an analogy. SGNS uses two embedding tables and scores input–output dot products. It does not directly minimize Euclidean distances between every pair of input rows. If V→VA and U→UA⁻ᵀ for invertible A,the dot-product scores stay unchanged,while input-space cosine geometry can change. The objective alone therefore does not force a unique semantic coordinate system or orthogonal gender/geography axes.

You will build a complete CPU SGNS experiment on a bundled miniature corpus,derive its stable loss and gradients,query nearest neighbors and analogies with explicit exclusions,evaluate vocabulary coverage,and compare static/contextual representations. Required knowledge:array shapes,embedding lookup,logistic and softmax loss,autograd and optimizer updates. This starts the language/sequence branch after image patches;word order beyond a local window needs additional modeling,not necessarily recurrence specifically.

Lookup, one-hot multiplication and repeated IDs

Let E have shape(V,D). A batch of token IDs selects its rows. A one-hot row eᵢ computes eᵢE=Eᵢ;PyTorch Linear instead stores its weight as(D,V),so transpose E when comparing APIs. The lookup avoids constructing the V-dimensional one-hot input.

Repeated IDs sum gradient contributions. Default nn.Embedding gradients are dense tensors with zero entries in unused rows;sparse=True changes gradient representation and optimizer compatibility. A zero current gradient does not guarantee an unused row never changes under weight decay,previous optimizer momentum,weight tying or max-norm behavior.

import re, math, copy
from collections import Counter
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
 
torch.manual_seed(31)
E=nn.Embedding(5,3).double();ids=torch.tensor([2,0,2,4])
looked=E(ids)
onehot=F.one_hot(ids,5).double()
torch.testing.assert_close(looked,onehot@E.weight)
looked.sum().backward()
expected=torch.tensor([[1,1,1],[0,0,0],[2,2,2],[0,0,0],[1,1,1]],dtype=torch.float64)
assert torch.equal(E.weight.grad,expected)
print('lookup rows',looked.detach().tolist(),'row gradient',E.weight.grad.tolist())

The larger original exampleV10000,D128 is the same primitive and has1,280,000 parameters;embedding IDs42,7,999,3141 produces(4,128). Tokenizer/vocabulary order is part of the model:save it alongside the table,not in a separate reconstruction that may reorder rows.

Skip-gram, full softmax and CBOW

For “the quick brown fox jumps over the lazy dog” and radius2,the center fox has contexts quick,brown,jumps,over. Skip-gram predicts those neighbors,not only the next word. With center vector v_c and output vector u_o,

The old expression called a “loss” without the minus sign was a log-likelihood to maximize. Full softmax is a normalized conditional model. Its V dot products each cost O(D). With10¹⁰ pairs,V10⁵,D128,there are10¹⁵ dot products,roughly2.56×10¹⁷ multiply/add FLOPs,not10¹⁵ FLOPs. At an ideal10GFLOP/s this alone is25.6M seconds,roughly296days;real batching/hardware can change throughput greatly. Do not turn a dot-product count into a measured2013 wall time.

Hierarchical softmax factors a vocabulary probability along tree paths (often a Huffman tree),with path-dependent O(log V)-type work. Negative sampling instead changes the training objective to binary discrimination. It is not a free unbiased full-softmax replacement.

CBOW reverses the direction:average/sum context embeddings,then predict the center. With mean pooling,each context input receives1/m of the gradient through that mean. A basic full-softmax CBOW computation is:

context_ids=torch.tensor([[1,2,3],[0,2,4]])
center_ids=torch.tensor([0,1])
in_table=nn.Embedding(5,4);out_table=nn.Embedding(5,4)
context=in_table(context_ids).mean(dim=1)
logits=context@out_table.weight.T
cbow_loss=F.cross_entropy(logits,center_ids)
cbow_loss.backward()
assert logits.shape==(2,5) and torch.isfinite(in_table.weight.grad).all()

CBOW makes one center prediction from its context;skip-gram emits multiple context pairs. Published relative strengths on frequent/rare words are empirical and depend on budget/window/sampling,not a law that everyone now uses skip-gram or it always wins on small corpora.

Negative sampling: the exact objective and derivative

For one observed pair and K independently sampled noise words,

Let s₊=vᵀu₊ and s_k=vᵀu_k. Then

Repeated negatives and a negative equal to the positive aggregate into the same row. A sampled “negative” is a noise draw,not proof the pair never occurs in language. This implementation allows collisions under its declared objective;implementations that reject the positive are using a different sampling policy.

Use logsigmoid or softplus on logits;log(sigmoid(-1000)) can underflow. The work of evaluating sampled scores is(K+1)D versusVD for full softmax,so100000/6≈16666.7 is a score-count ratio,not a measured speedup. Sampling,embedding-memory traffic,optimizer state and matrix batching all matter.

class SkipGramNS(nn.Module):
    def __init__(self,vocab_size,embed_dim=32):
        super().__init__()
        self.in_embed=nn.Embedding(vocab_size,embed_dim)
        self.out_embed=nn.Embedding(vocab_size,embed_dim)
        nn.init.uniform_(self.in_embed.weight,-.5/embed_dim,.5/embed_dim)
        nn.init.zeros_(self.out_embed.weight)
    def forward(self,centers,contexts,negatives):
        v=self.in_embed(centers);u=self.out_embed(contexts);un=self.out_embed(negatives)
        positive=(v*u).sum(-1)
        negative=torch.bmm(un,v.unsqueeze(-1)).squeeze(-1)
        return (-F.logsigmoid(positive)-F.logsigmoid(-negative).sum(-1)).mean()
 
def train_step(model,opt,centers,contexts,negative_weights,k=5,generator=None):
    if k<1 or len(centers)==0: raise ValueError('positive batch and negative count required')
    if (negative_weights<0).any() or not torch.isfinite(negative_weights).all() or negative_weights.sum()<=0:
        raise ValueError('invalid noise weights')
    neg=torch.multinomial(negative_weights,k*len(centers),replacement=True,
                         generator=generator).reshape(len(centers),k).to(centers.device)
    loss=model(centers,contexts,neg)
    opt.zero_grad(set_to_none=True);loss.backward();opt.step()
    return loss.item()
 
# At output-zero initialization all logits are0: each pair costs(K+1)*log2.
m=SkipGramNS(5,4).double()
c=torch.tensor([0,0]);o=torch.tensor([1,2]);negative=torch.tensor([[2,2,1],[3,4,4]])
initial=m(c,o,negative);assert abs(initial.item()-4*math.log(2))<1e-12
initial.backward()
assert torch.count_nonzero(m.in_embed.weight.grad)==0
assert torch.count_nonzero(m.out_embed.weight.grad)>0
# Both positive and negative terms update output rows; the input has NO first-step signal.
z=torch.tensor([-1000.,1000.],dtype=torch.float64)
assert torch.isfinite(F.logsigmoid(z)).all()
assert not torch.isfinite(torch.log(torch.sigmoid(z))).all()

This corrects the original explanation of output-zero initialization:the input gradient is initially zero because every u is zero. Positive and negative output gradients both exist;the positive term does not uniquely “organize the input before outputs move.” Initializing both tables to zero would leave all dot-product gradients zero.

Noise distribution, subsampling and dynamic windows

For counts f(w),noise weights use $q(w)\propto f(w)^{.75}$. Exponent1 preserves relative frequencies;0 gives uniform mass over positive-count vocabulary items;.75 flattens the distribution. The original paper reports an empirical choice,not a theorem about the universally optimal exponent or GloVe using the same negative sampler.

Positive-stream subsampling is a different operation. For relative frequency p(w),the simplified paper keep probability is min(1,sqrt(t/p(w))). The original released C code uses a closely related expression involving $(\sqrt{p/t}+1)t/p$ with an effective cap1. State which version is implemented. For p=.05,t=1e−5,the simplified keep probability is about.01414. On a tiny corpus that can discard nearly everything;disable subsampling there instead of misinterpreting an empty dataset as success.

A uniformly sampled window radius from1…W includes a neighbor at distance d with probability(W−d+1)/W. This changes expected pair frequency,not exactly “five times the gradient” after nonlinear training. Keep sentence boundaries explicit. Dropping words before window generation also changes effective distances.

def tokenize(text):
    return re.findall(r"[a-z']+",text.lower())
 
def build_vocab(sentences,min_count=1):
    counts=Counter(w for sentence in sentences for w in tokenize(sentence))
    itos=sorted(w for w,n in counts.items() if n>=min_count)
    if not itos: raise ValueError('empty vocabulary')
    stoi={w:i for i,w in enumerate(itos)}
    freqs=np.array([counts[w] for w in itos],dtype=np.float64)
    return itos,stoi,freqs
 
def build_negative_sampler(freqs,alpha=.75):
    freqs=np.asarray(freqs,dtype=np.float64)
    if freqs.ndim!=1 or freqs.size==0 or not np.isfinite(freqs).all() or np.any(freqs<=0):
        raise ValueError('nonempty finite positive count vector required')
    if not np.isfinite(alpha) or alpha<0: raise ValueError('finite nonnegative exponent required')
    # Normalize in log space: finite counts can overflow when raised to alpha.
    logs=np.log(freqs);scaled=alpha*(logs-logs.max())
    weights=np.exp(scaled);return weights/weights.sum()
 
def subsample(ids,freqs,rng,t=1e-5):
    if t<=0: raise ValueError('positive subsampling threshold required')
    p=freqs/freqs.sum();keep=np.minimum(1,np.sqrt(t/p))
    return [i for i in ids if rng.random()<keep[i]]
 
def make_skipgram_pairs(ids,window=2,rng=None):
    if window<1: raise ValueError('window must be positive')
    rng=np.random.default_rng() if rng is None else rng
    pairs=[]
    for i,center in enumerate(ids):
        radius=int(rng.integers(1,window+1))
        for j in range(max(0,i-radius),min(len(ids),i+radius+1)):
            if i!=j: pairs.append((center,ids[j]))
    return pairs
 
f=np.array([10000.,100.,1.])
q1=build_negative_sampler(f,1);q075=build_negative_sampler(f,.75);q0=build_negative_sampler(f,0)
assert q1[0]>q075[0]>q0[0]
assert abs(q075.sum()-1)<1e-12
assert all((5-d+1)/5>0 for d in range(1,6))
assert abs(math.sqrt(1e-5/.05)-.01414213562373095)<1e-15

The sampler rejects empty/nonfinite count vectors and nonfinite exponents; log-space normalization avoids overflow from directly raising large counts to a power. Very small probabilities can still underflow to zero in float64; this is a numerical approximation, not a different intended noise distribution.

The regex is a declared toy tokenizer:numbers and non-English letters are discarded. It would turn a literal <NUM> marker into the word num;that may not match downstream numeric tokens. For a real corpus,version normalization,case policy,Unicode handling,minimum count,OOV strategy and sentence boundaries. Phrase detection such as New_York needs a preprocessing decision before lookup;it is not automatically learned by a plain word tokenizer.

Complete bundled CPU training experiment

This miniature corpus is authored in the program,not downloaded text8. It contains repeated controlled contexts and is suitable for checking loss updates and query mechanics. It cannot establish general English semantics,Google-News analogy accuracy or a pretrained model's quality.

sentences=[
    'the king and queen wear a crown',
    'the prince and princess wear a crown',
    'the king rules the kingdom with the queen',
    'the prince visits the kingdom with the princess',
    'the cat and kitten drink water',
    'the dog and puppy drink water',
    'the cat and dog are a pet',
    'the kitten and puppy are a pet',
    'paris is a city in france',
    'rome is a city in italy',
    'london is a city in england',
    'berlin is a city in germany',
]
itos,stoi,freqs=build_vocab(sentences)
rng=np.random.default_rng(31)
pairs=[]
for sentence in sentences*8:
    ids=[stoi[w] for w in tokenize(sentence)]
    pairs.extend(make_skipgram_pairs(ids,window=2,rng=rng))
pairs=torch.tensor(pairs,dtype=torch.long)
weights=torch.tensor(build_negative_sampler(freqs),dtype=torch.float32)
torch.manual_seed(31);model=SkipGramNS(len(itos),24)
opt=torch.optim.Adam(model.parameters(),lr=.02)
gen=torch.Generator().manual_seed(31)
# A fixed diagnostic objective permits a before/after comparison despite sampled training losses.
diag=pairs[:256]
diag_neg=torch.multinomial(weights,5*len(diag),replacement=True,generator=gen).reshape(len(diag),5)
with torch.no_grad(): before=model(diag[:,0],diag[:,1],diag_neg).item()
losses=[]
for step in range(240):
    ix=torch.randint(len(pairs),(128,),generator=gen);batch=pairs[ix]
    losses.append(train_step(model,opt,batch[:,0],batch[:,1],weights,5,gen))
with torch.no_grad(): after=model(diag[:,0],diag[:,1],diag_neg).item()
assert np.isfinite(losses).all() and after<before
embeddings=model.in_embed.weight.detach().double()
print('bundled SGNS fixed diagnostic loss',before,after,'vocabulary',len(itos),'pairs',len(pairs))

For an already-local text8 or other trusted corpus,stream tokens/sentence windows and pairs instead of materializing every pair for100MB in Python lists. Keep a bounded shuffle buffer,batch negative draws and record tokenizer/counts/window/noise policy. The reference uses Adam and dense gradients for clarity,not an exact reproduction of original linearly decayed SGD. Alias sampling can preprocess a categorical distribution in O(V) and draw in O(1);whether it beats a particular torch.multinomial implementation requires profiling that version/device. No3k→40k-pairs/s story or25-minute GPU result is claimed.

Cosine queries, exclusions and a worked analogy

Normalize rows for cosine,check zero norms and explicitly mask the query ID;“take topk+1 then drop the first” fails if equal rows/ties reorder the self match. Similarity is not truth,logical entailment or a guaranteed stable frequency-independent semantic score.

def cosine_similarity(a,b):
    if a.norm()==0 or b.norm()==0: raise ValueError('zero vector has no cosine direction')
    return F.cosine_similarity(a.reshape(1,-1),b.reshape(1,-1),dim=1).item()
 
def rank_query(table,query,exclude=(),k=5):
    if query.norm()==0: raise ValueError('zero query')
    norms=table.norm(dim=1)
    scores=(table@query)/(norms.clamp_min(1e-12)*query.norm())
    scores[norms==0]=-torch.inf
    for i in exclude: scores[i]=-torch.inf
    valid=torch.isfinite(scores).sum().item(); k=min(k,valid)
    if k<1: return []
    order=torch.argsort(scores,descending=True,stable=True)[:k]
    return [(int(i),float(scores[i])) for i in order]
 
def most_similar(word,k=5):
    if word not in stoi: raise KeyError('out-of-vocabulary: '+word)
    idx=stoi[word]
    return [(itos[i],s) for i,s in rank_query(embeddings,embeddings[idx],(idx,),k)]
 
def analogy(a,b,c,k=1,exclude_inputs=True):
    ids=[stoi[w] for w in (a,b,c)]
    query=embeddings[ids[1]]-embeddings[ids[0]]+embeddings[ids[2]]
    return [(itos[i],s) for i,s in rank_query(embeddings,query,ids if exclude_inputs else (),k)]
 
for w in ('king','cat','paris'): print(w,most_similar(w,3))
# This constructed geometry tests arithmetic, NOT learned semantic discovery.
toy=torch.tensor([[1.,0.],[1.,1.],[2.,0.],[2.,1.]],dtype=torch.float64)
# IDs: man0,woman1,king2,queen3. king-man+woman=queen exactly.
query=toy[2]-toy[0]+toy[1]
assert torch.equal(query,toy[3]) and rank_query(toy,query,(0,1,2),1)[0][0]==3
# Duplicate rows must still exclude the actual query ID, even when ties precede it.
duplicates=torch.tensor([[1.,0.],[1.,0.],[0.,1.]])
assert rank_query(duplicates,duplicates[1],(1,),1)[0][0]==0
print('learned king-man+woman query omitted: man/woman are absent from bundled vocabulary')

Approximate consistent offsets can emerge when relations have regular contextual signatures. That is not a general proof of abstract gender concepts,independent orthogonal axes or reliable analogies for rare relations. Linzen's analogy analysis warns that nearest-neighbor structure and candidate exclusions can explain part of apparent offset success. Evaluate the exclusions transparently and include a nearest-to-b baseline;do not guarantee that dropping exclusions always returns b.

For the original practice set—man:king::woman:queen;athens:greece::baghdad:iraq;walking:walked::swimming:swam;good:better::bad:worse;usa:dollar::japan:yen—first report which words exist. A missing word is an OOV case,not a wrong prediction secretly removed from the denominator. The bundled corpus deliberately cannot answer these;the query algorithm remains completely runnable on any trusted local matching table.

Evaluation and visualization without geometric overclaims

Intrinsic tests ask about similarity rankings or analogies. Extrinsic tests measure an actual downstream task such as retrieval,NER or classification. The relation is not guaranteed:static-word similarity may be irrelevant to polysemy,rare jargon or sentence retrieval. Report covered and total examples,and a protocol-specific score;the classic Google analogy set's19,544 questions and14 groups describe that release,not an assurance of72% accuracy for this corpus.

WordSim-353 and SimLex-999 illustrate why the rating question matters: the SimLex paper explicitly separates genuine similarity from association/relatedness. A word can be strongly associated with a profession or field without being similar to it. Use the benchmark's actual instructions and release, rather than calling every human rating “semantic similarity.”

For similarity ratings,use Spearman rank correlation with tie handling. For an analogy benchmark,report full-denominator accuracy plus coverage and conditional accuracy when OOVs are excluded. For retrieval,split by query/document provenance and report recall/precision or ranking metrics with exact candidate sets. The captured MTEB paper revision describes58 datasets across8 task families and112 languages;current MTEB versions are different objects,so “mid70s versus low50s” is not a version-free comparison.

def evaluate_analogies(questions):
    correct=covered=0;rows=[]
    for a,b,c,answer in questions:
        if any(w not in stoi for w in (a,b,c,answer)):
            rows.append({'question':(a,b,c,answer),'status':'oov'});continue
        covered+=1;prediction=analogy(a,b,c,1)
        hit=bool(prediction) and prediction[0][0]==answer
        correct+=hit;rows.append({'question':(a,b,c,answer),'prediction':prediction,'correct':hit})
    total=len(questions)
    return {'total':total,'covered':covered,'correct':correct,
            'accuracy_all':correct/total if total else None,
            'accuracy_covered':correct/covered if covered else None,'rows':rows}
 
report=evaluate_analogies([('man','king','woman','queen'),('paris','france','rome','italy')])
assert report['total']==2 and report['covered']==1
# PCA preserves a linear projection; its2D distances still discard residual dimensions.
centered=embeddings-embeddings.mean(0)
_,_,vt=torch.linalg.svd(centered,full_matrices=False)
xy=centered@vt[:2].T
assert xy.shape==(len(itos),2) and torch.isfinite(xy).all()
print('analogy coverage',report,'PCA first five',xy[:5].tolist())

A t-SNE or UMAP plot is exploratory:neighborhoods,cluster gaps,density and parallel arrows may be distorted. UMAP does not guarantee globally truthful distances or speed superiority. The original promise of four clean clusters and parallel gender/capital arrows after a nonlinear2D projection was unjustified. Measure offsets in the original normalized space and test sensitivity to projection settings/seeds.

For an environment where the optional libraries already exist,the complete visualization function below returns coordinates without downloading anything. The audit executes its t-SNE branch on twelve local vectors and checks finite2D output. UMAP is not installed and its branch is not invoked; neither projection is claimed equivalent to PCA.

def optional_projection(vectors,method='tsne',seed=31):
    array=np.asarray(vectors,dtype=np.float64)
    if len(array)<4: raise ValueError('need at least four rows')
    if method=='tsne':
        from sklearn.manifold import TSNE
        return TSNE(n_components=2,perplexity=min(5,len(array)-1),init='random',
                    learning_rate='auto',random_state=seed).fit_transform(array)
    if method=='umap':
        from umap import UMAP
        return UMAP(n_neighbors=min(5,len(array)-1),min_dist=.3,random_state=seed).fit_transform(array)
    raise ValueError('unknown projection')
 
tsne_xy=optional_projection(embeddings[:12].numpy(),method='tsne')
assert tsne_xy.shape==(12,2) and np.isfinite(tsne_xy).all()
print('local t-SNE coordinates',tsne_xy[:3].tolist())

To inspect a labeled plot without installing a plotting package, the following standard-library SVG writer plots either set of coordinates. It does not draw semantic-relation arrows or infer clusters. Set EMBEDDING_FIGURE_PREFIX to a chosen local path prefix to export both plots; no network or application server is involved.

import os
from pathlib import Path
from xml.sax.saxutils import escape
import xml.etree.ElementTree as ET
 
def embedding_svg(coordinates,labels,title):
    coordinates=np.asarray(coordinates,dtype=float)
    if coordinates.shape!=(len(labels),2) or not np.isfinite(coordinates).all():
        raise ValueError('finite two-dimensional coordinates required')
    lo=coordinates.min(0);span=np.maximum(coordinates.max(0)-lo,1e-9)
    scaled=(coordinates-lo)/span
    lines=['<svg xmlns="http://www.w3.org/2000/svg" width="960" height="640" viewBox="0 0 960 640">',
           '<rect width="960" height="640" fill="#ffffff"/>',
           '<text x="36" y="34" font-size="20" fill="#111827">'+escape(title)+'</text>',
           '<text x="36" y="59" font-size="13" fill="#374151">Toy corpus projection: distances and apparent clusters are not semantic proof.</text>']
    for (x,y),label in zip(scaled,labels):
        px=45+810*x;py=585-485*y
        lines.append(f'<circle cx="{px:.2f}" cy="{py:.2f}" r="3" fill="#0369a1"/>')
        lines.append(f'<text x="{px+5:.2f}" y="{py-5:.2f}" font-size="12" fill="#111827">{escape(label)}</text>')
    return '\n'.join(lines+['</svg>'])
 
pca_svg=embedding_svg(xy.numpy(),itos,'PCA of bundled SGNS input vectors')
tsne_svg=embedding_svg(tsne_xy,itos[:12],'t-SNE of twelve bundled SGNS vectors')
assert len(ET.fromstring(tsne_svg).findall('{http://www.w3.org/2000/svg}circle'))==12
prefix=os.environ.get('EMBEDDING_FIGURE_PREFIX')
if prefix:
    Path(prefix+'-pca.svg').write_text(pca_svg)
    Path(prefix+'-tsne.svg').write_text(tsne_svg)

Descendants: different representations and objectives

• GloVe: weighted least-squares fitting of log co-occurrence counts with biases. It is not SGNS and does not require the.75 negative sampler. The840B-token Common-Crawl vector release is a release identity,not the corpus used by this notebook.
• fastText: represents words using character n-gram contributions (including boundary conventions and word representation),allowing vectors for unseen surface forms with known subword pieces. Character n-grams are not linguistically guaranteed morphemes like “friend” or “ness.”
• ELMo: context-dependent representations from a deep bidirectional language model. BERT and autoregressive transformers start from token lookups but produce context-dependent hidden states;sharing the primitive does not make them Word2Vec objectives.
• SBERT: sentence-pair training produces useful sentence embeddings. E5,BGE,GTE,NV-Embed,Nomic are families/checkpoints with task instructions,pooling,dimension and normalization contracts. Voyage,Cohere,OpenAI embedding APIs have their own model/version limits. Do not compare unspecified dimensions or leaderboard scores across all these families.
• Matryoshka representation learning: explicitly trains useful nested prefixes. Arbitrarily truncating an ordinary embedding is not guaranteed to preserve retrieval quality. Prefix normalization and supported sizes belong to the checkpoint's contract.

Retrieval may use approximate vector indexes,hybrid lexical/dense candidates and a contextual reranker;not every task becomes nearest-neighbor lookup. Static embeddings remain useful for constrained/domain-specific systems and are not forbidden in2025. Their advantages are inexpensive lookup and predictable storage;their costs are polysemy,OOV handling and corpus dependence. Contextual encoders cost a forward pass but can run on CPU when workload/model permit;they do not inherently require a paid API or accelerator.

For an assumedV128000,D16384 table,the arithmetic is2,097,152,000 parameters. That is a hypothetical shape calculation,not a verified Llama4 architecture claim. Actual model card/tokenizer/config must identify V,D,tying and special tokens. Modern LLM pretraining usually uses normalized next-token likelihood because it needs generative probabilities over the vocabulary;SGNS's binary noise discrimination is not the same objective,even if it evaluates fewer scores. GPU parallelism helps full-softmax computation but is not the only reason for choosing it.

Linear features, social bias and diagnostic discipline

Park,Choe and Veitch's Linear Representation Hypothesis work formalizes notions of linear representations and geometry under assumptions;it does not prove every high-level concept in every model must be one clean axis. Toy Models of Superposition studies how features can share dimensions. Scaling Monosemanticity trains sparse autoencoders to extract interpretable features from a particular model. These connect to representation geometry but are not simply Word2Vec analogies rediscovered unchanged.

Bolukbasi et al. demonstrate problematic gender associations in pretrained word embeddings. Treat demographic analogies as measurements of a corpus/model's associations,not statements about people's capabilities or natural roles. Removing one direction is not proof that all downstream bias disappears. Use subgroup-aware,task-relevant evaluation and inspect training provenance.

Debugging checklist:validate tokenizer/vocabulary contract;check pair boundaries/counts;ensure subsampling leaves enough pairs;compare stable loss to a hand oracle;inspect first-step gradients;measure norm distributions and nonfinite values;profile sampling before swapping algorithms;store versioned IDs alongside both tables. Frequent-word neighbors are a hypothesis about corpus/sampling,not proof of a literal geometric centroid. LayerNorm in a later transformer does not make all learning rates safe.

Worked answers and retained practice

1. Skip-gram objective: predict observed context words from a center;full softmax maximizes normalized conditional log-likelihood,while SGNS minimizes positive/noise binary losses.
2. SGNS formula:−logσ(v·u₊)−Σlogσ(−v·uₖ);the center gradient and both output gradients are derived above. Sum repeated-ID contributions.
3. Empirical controls: window W,negative count K,noise exponent.75,subsampling threshold t,and initialization policy. Those are more than the original “three constants”;each changes the task/updates.
4. **Cost:**V/(K+1)=100000/6≈16666.7 sampled-score ratio,not runtime speedup. Include D FLOPs per dot and sampling/optimization overhead.
5. **Why analogies sometimes work:**consistent relation offsets can fit contextual statistics. Exclusion rules,frequency,polysemy and benchmark choice affect success;no universal semantic law follows.
6. Would SGNS replace LLM CE? Not without changing the statistical objective;SGNS does not directly give normalized next-token probabilities. Parallel matrix kernels make full-softmax cost manageable in many regimes,but objective fidelity matters too.
7. **Output-zero-init question:**initial input gradient is zero;both positive and negative output rows receive gradients. The old positive-dominance explanation was backwards.
8. **Text8 stretch:**if a trusted local file is already available,run a bounded streamed version,compare D32 versusD128 at equal token budget and report OOV/covered/all accuracy for the five named analogies. Do not download a corpus or declare a promised3/5 result.

Tomorrow derive the binary gradients and explain two tables without notes. On day7 compare query exclusions and downstream metrics. The next lesson carries learned character vectors through a recurrent state and explicitly tests temporal credit assignment.

Sources and execution boundary

Primary sources: first Word2Vec paper, negative-sampling paper, original C implementation, PyTorch2.9 Embedding, Linzen's analogy analysis, GloVe, fastText, ELMo, SBERT, MTEB, Matryoshka, LRH, bias study, UMAP.

Original optional Karpathy/AssemblyAI video titles, durations and recommended segments were not independently verified here. They are not evidence for the mechanisms above; primary-source passages and executable checks carry that role.

Executed:exact bundled CPU training,lookup accumulation,stable-loss/initial-gradient controls,query/exclusion tests,coverage accounting,PCA,a twelve-vector t-SNE projection and labeled SVG export. Optional UMAP,large-corpus analogies,downloaded embeddings and paid services are not executed. Query outputs in the raw test report are measured toy results,not invented semantic benchmarks. Previous: ViT · Next: RNN.