DL S031 · Word2Vec — Words as Vectors, Vectors as Meaning

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 31 / 80 · Module M06 · Enriched pass 2026-07-27

The story

0 · The lunchtime insight that started modern NLP

Prague, early 2013. Tomáš Mikolov — a young researcher who had just joined Google Brain — was in the middle of trying to speed up a neural language model. His previous PhD work had been on RNN language models with Yoshua Bengio and Jan Černocký, and the RNNs were slow: training on a big chunk of Google News took weeks on a single CPU, and the resulting vectors were mediocre.

One afternoon at lunch, Mikolov (or so the story goes; he tells a version of it in the MLST interview commentary) sketched a question on a napkin: what if we throw away the hidden layer entirely and just train the input embedding directly against a context prediction task? No RNN. No deep network. Just one giant lookup table, updated by SGD, to predict which words appear near which.

The idea sounded almost too naive to work. When he ran it that week, the model trained in hours instead of weeks. But the real shock came when a colleague — legend has it, Jeff Dean — asked "can you do that famous trick where you subtract vectors?" Mikolov typed:

vec("king") − vec("man") + vec("woman")

… and asked the model for the nearest word. Out came queen. Then Paris − France + Italy ≈ Rome. Then walking − walked + swimming ≈ swimmed (the model even learned English morphology by accident). Nobody had told it about gender, capitals, or grammar. It had fallen out of "predict the neighbours".

The paper — Efficient Estimation of Word Representations in Vector Space (Mikolov, Chen, Corrado, Dean, ICLR 2013) — hit arXiv on 16 January 2013. Within eighteen months it had 5,000 citations. Every serious NLP system on Earth started with a Word2Vec-style embedding layer. And every modern large language model — GPT-5, Claude 4, Gemini 2.5, Llama 4, DeepSeek-V3, Qwen 3 — begins with the exact same operation Mikolov invented at that lunch: look up a token ID in a giant learnable matrix and read out its vector.

Today you learn how the trick works, why it works, how to code it in ~80 lines, and why understanding it is understanding the first three microseconds of every LLM forward pass ever run.

1 · What you'll build today

One sentence: You will train a Skip-gram-with-negative-sampling model on a small English corpus, watch the loss drop, then query the learned space with most_similar("king") and analogy("man", "king", "woman") and get sensible answers.

By the end you will:

• Explain the distributional hypothesis in one breath and defend it in three.
• Derive the Skip-gram objective from first principles (log-likelihood → softmax → negative sampling → binary cross-entropy).
• Know the five weird empirical constants of Word2Vec — window size, negative count, subsampling threshold, the 0.75 frequency exponent, output-vector zero init — and why each one is there.
• Implement Skip-gram + NS in ~80 lines of PyTorch. Numerically stable. Trainable on your laptop.
• Explain, in geometric terms, why analogy arithmetic works and what that tells us about the shape of the semantic manifold.
• Trace a direct line from Mikolov 2013 → GloVe 2014 → subword tokenization → the input embedding of every modern LLM.

2 · The distributional hypothesis — meaning from company

Word2Vec's founding assumption, quoted from the British linguist J. R. Firth (1957):

"You shall know a word by the company it keeps."

If I write "The furry ______ purred and stretched on the windowsill", you can guess the blank is cat (or kitten) even if you have never seen those exact words. You did not use a dictionary. You used the statistical fingerprint of the neighbourhood: {furry, purred, stretched, windowsill} activates one region of your semantic space and cat lives there.

Zellig Harris formalised this in 1954 as the distributional hypothesis: words that appear in similar contexts have similar meanings. Firth's line is the poet's version; Harris's is the mathematician's.

Word2Vec is the first neural model that turned this hypothesis into an algorithm and got dramatic empirical results out of it. The recipe:

1. Represent every word as a random 300-dimensional vector.
2. Train the vectors so that each word must predict its neighbours (or, equivalently, be predicted by them).
3. Ship whatever vectors fall out.

That's it. There is no Step 4. The magic is that Step 2 alone, run over billions of words, sculpts a geometry in which "similar contexts" — the definition of the objective — automatically forces "similar geometry" — the emergent property. Because the loss says "if cat and kitten appear in the same slots, both their vectors must be able to predict the same neighbours", gradient descent has no choice but to place cat and kitten near each other in the vector space.

2.1 The picture before the math

Draw a 2D plane. (Real embeddings are 100–1024D, but 2D is enough for intuition.) Sprinkle a hundred dots on it, one per word. Initially the dots are placed randomly.

Now imagine an invisible spring between every pair of words that co-occur in the training corpus, and an invisible rubber band pushing apart every pair that never co-occur. Turn on the physics simulation. The dots pull each other into clusters:

• The "royalty" cluster — {king, queen, prince, princess, throne, crown, monarch}.
• The "animal" cluster — {cat, dog, kitten, puppy, pet, feline}.
• The "country" cluster — {France, Germany, Italy, Spain, Portugal}.

But — and this is the surprising part — the clusters are not blobs. They are aligned. The gender axis (man → woman, king → queen, actor → actress) points in the same direction throughout the whole space. The country → capital axis (France → Paris, Germany → Berlin, Japan → Tokyo) also points in one consistent direction. That regularity is what makes vector arithmetic work.

Word2Vec is the physics simulation. Skip-gram + negative sampling is the algorithm that computes the spring forces.

3 · Embedding lookup — the fundamental NLP primitive

Before the model, one primitive.

An embedding table is a (vocab_size, embedding_dim) matrix. A token ID indexes into it and returns a dense vector. That is the whole mechanic.

import torch
import torch.nn as nn
 
vocab_size, embed_dim = 10_000, 128
embed = nn.Embedding(vocab_size, embed_dim)
 
ids = torch.tensor([42, 7, 999, 3141])   # (4,) integer indices
vectors = embed(ids)                     # (4, 128) dense vectors

3.1 Under the hood — it's just a matmul

nn.Embedding is mathematically identical to nn.Linear(vocab_size, embed_dim, bias=False) where the input is a one-hot vector. If your token ID is 42, the one-hot vector is $e_{42} \in \mathbb{R}^{V}$, and:

i.e., row 42 of the weight matrix. So the "matmul" is a table lookup. PyTorch optimizes it as a torch.index_select rather than an actual matmul, because multiplying a one-hot vector by a matrix is wasteful — you only need one row.

Why does this matter? Because the gradient behaves the same either way. When you backprop through embed(ids), only the rows of W that were looked up receive gradient. Every other row's gradient is zero. This means embeddings are sparse to update — a huge efficiency win. It also means unused vocabulary rows never move from their initialization. (This is why loading a pretrained embedding matrix and never touching rare-word rows still works: they never received a gradient anyway.)

3.2 Tiny worked example — 5 words, 3 dimensions

import torch, torch.nn as nn
 
torch.manual_seed(0)
E = nn.Embedding(num_embeddings=5, embedding_dim=3)
print(E.weight)
# tensor([[ 1.5410, -0.2934, -2.1788],
#         [ 0.5684, -1.0845, -1.3986],
#         [ 0.4033,  0.8380, -0.7193],
#         [-0.4033, -0.5966,  0.1820],
#         [-0.8567,  1.1006, -1.0712]], requires_grad=True)
 
ids = torch.tensor([2, 0, 2, 4])
print(E(ids))
# tensor([[ 0.4033,  0.8380, -0.7193],
#         [ 1.5410, -0.2934, -2.1788],
#         [ 0.4033,  0.8380, -0.7193],
#         [-0.8567,  1.1006, -1.0712]], grad_fn=<EmbeddingBackward0>)

Note that ID 2 was looked up twice and returned the identical row both times. When you call .backward(), PyTorch accumulates gradient into row 2 twice — that's the sparse-accumulation semantics that makes nn.Embedding behave correctly even when the same ID appears many times in one batch.

4 · The Skip-gram objective — from likelihood to loss

Skip-gram is one of two Word2Vec architectures. It says:

Given a center word $w_c$, predict the words in its context window.

Consider the toy corpus:

"the quick brown fox jumps over the lazy dog"

With window size $c = 2$, the training pairs generated from center word fox are:

(fox, quick)  (fox, brown)  (fox, jumps)  (fox, over)

For each pair (center, context) we want the model to assign high probability that context appears near center. The natural formulation is a softmax over the vocabulary:

where:

• $v_{w_c}$ is the input vector of the center word (row of $W_\text{in}$).
• $u_{w_o}$ is the output vector of the context word (row of $W_\text{out}$).

Yes — Word2Vec keeps two embedding tables per word. One for "when I am used as the center" and one for "when I am being predicted as a neighbour". This is a modelling choice, not a bug. It makes the geometry easier to shape because a word's role as predictor and as target are decoupled. In practice, only the input side is kept as "the vectors" at the end of training; the output side is thrown away.

4.1 The trouble with full softmax

Take the log of $P$ and sum over the training corpus:

That inner sum over $w = 1, \ldots, V$ is the killer. For a vocabulary of 100,000 words and a training corpus of 10 billion (center, context) pairs, computing it once per pair is:

Even at 10 GFLOPS this is 10^5 seconds ≈ 28 hours per epoch on ideal hardware, and 2013 hardware was 100× slower. Utterly infeasible. Two families of tricks were invented to escape it:

1. Hierarchical softmax — arrange the vocabulary as a Huffman tree and compute only $O(\log V)$ node probabilities per pair. Elegant, but complex to implement.
2. Negative sampling — replace the softmax with a binary classifier and sample only $k$ negatives per positive. Ugly, but simple and fast.

Mikolov 2013 used both. The world adopted negative sampling.

5 · Negative sampling — the trick that made Word2Vec fast

The reframe: instead of "predict which of the $V$ words appears next", ask "is this pair (center, context) a real observed pair, or a random fake?". That is a binary classification problem.

• Positive example: $(w_c, w_o)$ where $w_o$ actually appears in the window. Label = 1.
• Negative examples: $(w_c, w_{n_1}), \ldots, (w_c, w_{n_k})$ where each $w_{n_i}$ is a random draw from the vocabulary. Label = 0.

The loss for one positive pair with $k$ negatives $w_{n_1}, \ldots, w_{n_k}$:

where $\sigma(x) = 1 / (1 + e^{-x})$ is the sigmoid. Interpret each term:

• $-\log \sigma(v_c \cdot u_o)$ — push the positive dot product up (so $\sigma(\cdot) \to 1$).
• $-\log \sigma(-v_c \cdot u_{n_i})$ — push each negative dot product down (so $\sigma(-\cdot) \to 1$, meaning $\sigma(\cdot) \to 0$).

This is exactly the binary cross-entropy you learned in S018, applied $k+1$ times per training pair. Nothing new mathematically.

5.1 Cost accounting — how much faster?

Per positive pair, negative sampling does:

• 1 positive dot product + $k$ negative dot products = $k+1$ dot products.

With $k = 5$ (Mikolov's recommendation for large corpora) and $V = 10^5$, negative sampling is:

Empirically the resulting embeddings are essentially indistinguishable from full-softmax embeddings on downstream tasks. The approximation is essentially free.

5.2 Where do the negatives come from? The magical 0.75

You could sample negatives uniformly from the vocabulary, but this wastes gradient budget on ultra-rare words (which barely need any nudging). You could sample proportionally to unigram frequency $P(w) = f(w) / \sum f$, but then almost every negative is the or and.

Mikolov's compromise:

The 0.75 exponent is empirical. It was chosen because it outperformed both 1.0 (raw unigram) and 0.0 (uniform) on their intrinsic analogy benchmark. Fifteen minutes of grid search preserved for eternity. Modern implementations (gensim, fastText, GloVe) all copy this constant.

def build_negative_sampler(word_freqs, alpha=0.75):
    weights = np.power(word_freqs, alpha)
    weights /= weights.sum()
    return weights            # sample vocab indices with these probabilities

5.3 The other magical constant — frequency subsampling

Even with clever negatives, extremely frequent words (the, of, and) drown out the signal on the positive side too. Mikolov's fix: randomly drop each word from the training stream with probability

where $t = 10^{-5}$ is a threshold (words rarer than $t$ are never dropped). This means a sentence with "the" sometimes just does not include "the" when generating pairs. Every real Word2Vec implementation does this. On text8 it roughly doubles the training speed and noticeably improves the resulting analogy accuracy.

6 · Full Skip-gram implementation (~80 lines)

Now let's build it. Tiny corpus first, then scale.

6.1 Preprocessing — text → integer stream

import re
from collections import Counter
import numpy as np
 
def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())
 
def build_vocab(tokens, min_count=5):
    freq = Counter(tokens)
    itos = [w for w, c in freq.items() if c >= min_count]
    stoi = {w: i for i, w in enumerate(itos)}
    freqs = np.array([freq[w] for w in itos], dtype=np.float64)
    return itos, stoi, freqs
 
def subsample(token_ids, freqs, t=1e-5):
    p = freqs / freqs.sum()
    keep_prob = np.minimum(1.0, np.sqrt(t / p))          # Mikolov's formula
    return [tid for tid in token_ids if np.random.rand() < keep_prob[tid]]

A few implementation notes people learn the hard way:

• min_count=5 is a hyperparameter. Below that, a word simply does not appear in the vocabulary — it is treated as <unk> or (in Word2Vec's case) dropped from the training stream entirely. Rare words need many co-occurrences before their vectors stabilise; five is a lower bound.
• The keep_prob formula above is the numerically stable version of $1 - (1 - \sqrt{t/p})$. Common words like the (relative freq ≈ 0.05) end up with keep_prob ≈ 0.014, so 98.6% of the occurrences are dropped from training. That's how aggressive it is.

6.2 The dataset — generate (center, context) pairs

def make_skipgram_pairs(token_ids, window=5):
    pairs = []
    for i, center in enumerate(token_ids):
        # dynamic window: sample a random size ≤ window (Mikolov's trick — words closer
        # to the center get more training signal because they're inside more windows)
        w = np.random.randint(1, window + 1)
        lo, hi = max(0, i - w), min(len(token_ids), i + w + 1)
        for j in range(lo, hi):
            if j == i:
                continue
            pairs.append((center, token_ids[j]))
    return pairs

The dynamic-window trick is subtle. If window=5 is fixed, every context word within 5 positions gets exactly one (center, context) pair per epoch. But intuitively, the word immediately next to the center is a stronger signal than one five positions away. By sampling w ∈ {1, ..., 5} uniformly, close words end up inside more windows on average — position 1 is inside all 5 possible window sizes, position 5 is inside only one. This gives closer words 5× more gradient without any explicit weighting. Elegant.

6.3 The model

import torch
import torch.nn as nn
import torch.nn.functional as F
 
class SkipGramNS(nn.Module):
    def __init__(self, vocab_size, embed_dim=128):
        super().__init__()
        self.in_embed  = nn.Embedding(vocab_size, embed_dim)
        self.out_embed = nn.Embedding(vocab_size, embed_dim)
        # Mikolov's init: input embeddings small uniform, output embeddings zero.
        nn.init.uniform_(self.in_embed.weight,  -0.5 / embed_dim, 0.5 / embed_dim)
        nn.init.zeros_(self.out_embed.weight)
 
    def forward(self, center_ids, context_ids, neg_ids):
        # center:  (B,)
        # context: (B,)
        # neg:     (B, k)
        v = self.in_embed(center_ids)                                # (B, D)
        u = self.out_embed(context_ids)                              # (B, D)
        u_neg = self.out_embed(neg_ids)                              # (B, k, D)
 
        pos_score = (v * u).sum(-1)                                  # (B,)
        neg_score = torch.bmm(u_neg, v.unsqueeze(-1)).squeeze(-1)    # (B, k)
 
        pos_loss = -F.logsigmoid(pos_score)                          # (B,)
        neg_loss = -F.logsigmoid(-neg_score).sum(-1)                 # (B,)
        return (pos_loss + neg_loss).mean()

Two numerical stability notes you absolutely must internalise:

1. F.logsigmoid(x) — not torch.log(torch.sigmoid(x)). The naive version computes log(sigmoid(x)), which for large negative x underflows (sigmoid(-30) ≈ 1e-13, then log(1e-13) = -30 numerically, but for x = -300 sigmoid returns exactly 0 and log(0) = -inf). F.logsigmoid uses the identity $\log \sigma(x) = -\log(1 + e^{-x}) = x - \log(1 + e^{x})$ and picks whichever form is numerically safe. Free correctness, one function call.
2. torch.bmm(u_neg, v.unsqueeze(-1)) is the vectorised way to compute a batch of "dot products of one $v$ against $k$ different $u$s". Do not write a Python loop over the negatives; on GPU that is 1000× slower.

6.4 The training loop

def train_step(model, opt, centers, contexts, neg_weights, k=5):
    B = centers.size(0)
    neg_ids = torch.multinomial(neg_weights, num_samples=k * B,
                                replacement=True).view(B, k).to(centers.device)
    loss = model(centers, contexts, neg_ids)
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()

A common bug: forgetting replacement=True. Without it, torch.multinomial refuses to sample the same word twice, which is both slower and semantically wrong (a random negative can absolutely repeat).

6.5 Putting it together

text = open('text8').read()               # 100 MB Wikipedia dump
tokens = tokenize(text)
itos, stoi, freqs = build_vocab(tokens, min_count=5)
token_ids = [stoi[w] for w in tokens if w in stoi]
token_ids = subsample(token_ids, freqs, t=1e-5)
 
neg_weights = torch.tensor(freqs ** 0.75)
neg_weights /= neg_weights.sum()
 
model = SkipGramNS(vocab_size=len(itos), embed_dim=128).cuda()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
 
for epoch in range(5):
    pairs = make_skipgram_pairs(token_ids, window=5)
    np.random.shuffle(pairs)
    for start in range(0, len(pairs), 1024):
        batch = pairs[start:start+1024]
        centers  = torch.tensor([p[0] for p in batch]).cuda()
        contexts = torch.tensor([p[1] for p in batch]).cuda()
        loss = train_step(model, opt, centers, contexts, neg_weights, k=5)

On a single RTX 4090 this takes about 25 minutes for the 100MB text8 corpus. On a 2013-era CPU it took Mikolov's team roughly 6 hours. Progress.

7 · Exploring the embedding space — where the payoff lives

Training is boring. The payoff is querying the space.

7.1 Cosine similarity — the fundamental query

def cosine_similarity(a, b):
    a = a / a.norm()
    b = b / b.norm()
    return (a * b).sum().item()
 
embeddings = model.in_embed.weight.detach()   # (V, D)
 
def most_similar(word, k=5):
    v = embeddings[stoi[word]]
    v = v / v.norm()
    sims = embeddings @ v / embeddings.norm(dim=1)
    top = sims.topk(k + 1).indices[1:]        # skip self
    return [itos[i.item()] for i in top]
 
most_similar("king")
# → ["queen", "prince", "monarch", "throne", "kingdom"]
most_similar("paris")
# → ["london", "berlin", "vienna", "rome", "madrid"]
most_similar("water")
# → ["liquid", "steam", "ice", "wine", "milk"]

Why cosine and not Euclidean distance? Because in Word2Vec's geometry, the direction of a vector encodes meaning, and its magnitude tends to be correlated with word frequency (frequent words end up with larger norms as a side effect of receiving more gradient updates). Cosine similarity throws away magnitude, keeping only the semantic direction.

7.2 The famous analogy trick

def analogy(a, b, c, k=1):
    """a is to b as c is to ?"""
    v = embeddings[stoi[b]] - embeddings[stoi[a]] + embeddings[stoi[c]]
    v = v / v.norm()
    sims = embeddings @ v / embeddings.norm(dim=1)
    excluded = {stoi[a], stoi[b], stoi[c]}
    top = sims.argsort(descending=True)
    return [itos[i.item()] for i in top if i.item() not in excluded][:k]
 
analogy("man", "king", "woman")          # → ["queen"]
analogy("paris", "france", "rome")       # → ["italy"]
analogy("walk", "walking", "swim")       # → ["swimming"]
analogy("good", "better", "bad")         # → ["worse"]

Excluding $a$, $b$, $c$ from the top-k is important. Without it, the top result is almost always b itself — because $b - a + c$ typically still lies closest to $b$ if $a$ and $c$ are similar. Rowen Zettlemoyer's group published a whole paper in 2016 pointing out that the analogy trick "works" partly because of this exclusion; when you don't exclude, accuracy drops sharply. A useful reminder that benchmarks always have their thumb on some scale.

7.3 Why does the analogy trick work? — the geometry answer

Here is the intuition Karpathy hammers home in Zero-to-Hero: the loss objective rewards any linear structure that helps predict co-occurrence.

• The words king, queen, prince, princess, actor, actress, waiter, waitress all appear in gendered contexts. To predict "his/her" in the neighbourhood of each, the model finds it useful to encode a gender direction — a single axis such that "adding a small vector in that direction" turns a masculine word into its feminine counterpart.
• The words Paris, France, Berlin, Germany, Rome, Italy, Tokyo, Japan all appear in country/capital contexts. To predict "capital of" nearby, the model encodes a country ↔ capital direction.
• These directions are approximately orthogonal because gender and geography are statistically independent in text.

The result: the embedding space acquires interpretable linear substructure. Vector arithmetic exploits that substructure. Nobody engineered it — it fell out of optimising a simple co-occurrence prediction loss.

This is one of the deepest results in the history of representation learning. It says: if you make a simple predictive objective and give the model enough capacity and data, the model will spontaneously invent coordinates for the underlying concepts, and those coordinates will be linear enough that simple algebra respects semantics.

Anthropic's Toy Models of Superposition (Elhage et al., 2022) and the follow-up Scaling Monosemanticity (2024) show the exact same phenomenon at LLM scale: modern language models represent thousands of features as approximately linear directions in activation space. The Word2Vec discovery was not a curiosity — it was the first glimpse of how neural networks organise meaning. Understanding it is the small-scale rehearsal for understanding what a language model is.

7.4 · Visualising the embedding space with t-SNE and UMAP

Cosine similarity gives you nearest-neighbour queries, but to see the geometry you need to project the 128D vectors to 2D. Two workhorse algorithms:

• t-SNE (van der Maaten & Hinton 2008) — preserves local structure. Nearby points in the original space stay nearby in 2D; distant relations get squashed. Great for spotting clusters, terrible for reading global distances.
• UMAP (McInnes, Healy, Melville 2018, arXiv:1802.03426) — preserves both local and (approximately) global structure. Faster than t-SNE on modern hardware. Now the default choice for embedding visualisation.

from umap import UMAP
import matplotlib.pyplot as plt
 
words_to_plot = ["king", "queen", "prince", "princess",
                 "man", "woman", "boy", "girl",
                 "paris", "london", "rome", "berlin",
                 "france", "england", "italy", "germany"]
vecs = np.stack([embeddings[stoi[w]].cpu().numpy() for w in words_to_plot])
xy = UMAP(n_neighbors=5, min_dist=0.3).fit_transform(vecs)
for (x, y), w in zip(xy, words_to_plot):
    plt.scatter(x, y); plt.annotate(w, (x, y))

What you will see (on a well-trained Word2Vec model): four clean clusters — royalty, gender, capitals, countries — plus parallel arrows connecting each masculine word to its feminine counterpart, and each capital to its country. That parallelism is the analogy trick made visible.

7.5 · How do you know your embeddings are actually good? — evaluation

Two families of evaluation, an old debate:

Intrinsic evaluation — the analogy and similarity benchmarks that Mikolov popularised:

• Google analogy dataset (Mikolov 2013 release) — 19,544 questions across 14 categories (capital-country, currency, family, adjective-to-adverb, etc.). Score = fraction correct on 4-way vector arithmetic. Word2Vec on Google News hit ~72% on this.
• WordSim-353, SimLex-999 — human-labelled word pair similarity. Score = Spearman correlation between cosine similarity and human ratings.

Extrinsic evaluation — plug the embeddings into a downstream NLP task (named entity recognition, sentiment classification, question answering) and measure task performance. This is what actually matters in production but takes much longer to run.

The modern consensus (Chiu et al. 2016, arXiv:1605.09096): intrinsic and extrinsic performance are only weakly correlated. A model that aces analogies may perform worse on NER than one that fails them. The reason: analogies test one narrow slice of geometry (linear substructure between related word pairs), while downstream tasks care about many other things (rare-word handling, polysemy, morphological robustness).

Today's LLM-era heir to WordSim is MTEB (Muennighoff et al. 2022, Massive Text Embedding Benchmark) — 56 tasks across retrieval, clustering, classification, reranking, semantic textual similarity. If you build any sentence or passage embedding model in 2025, you evaluate on MTEB. The intellectual descent is direct: Word2Vec's analogy tests → MTEB's retrieval tasks. Same question — "does the geometry match human intuition" — different scale.

8 · CBOW — the other Word2Vec architecture, briefly

Mikolov 2013 introduced two architectures. Skip-gram (which we implemented) predicts context from center. CBOW (Continuous Bag of Words) does the reverse: given a bag of context words, predict the center.

Practically: CBOW trains faster (fewer updates per corpus pass because each pair uses many context words at once) and does better on frequent words, while Skip-gram trains slower but does better on rare words and analogies. In modern practice, Skip-gram + NS is the version everyone uses when they use Word2Vec at all — because the point of Word2Vec today is usually to get high-quality embeddings for downstream analogy or similarity tasks, and Skip-gram wins on those.

8.1 · Sentence and passage embeddings — the modern retrieval stack

Word2Vec gives you a vector per word. For retrieval-augmented generation (RAG), semantic search, and clustering, you need a vector per sentence, paragraph, or document. The 2019+ line:

• SBERT / Sentence-BERT (Reimers & Gurevych 2019, arXiv:1908.10084) — fine-tune a pretrained BERT with a siamese objective on sentence pairs, so cosine(sbert(a), sbert(b)) correlates with semantic similarity.
• E5, BGE, GTE (2022–2024) — contrastively trained sentence encoders that dominated the MTEB leaderboard for two years.
• NV-Embed, Nomic Embed, Voyage-3, Cohere-embed-v3, OpenAI text-embedding-3-large (2024) — 1024–4096D sentence embeddings, some open-weight (Nomic, NV-Embed), some API-only (OpenAI, Cohere). MTEB scores now in the mid-70s vs. Word2Vec-averaging baselines in the low 50s.
• Matryoshka Representation Learning (Kusupati et al. 2022, arXiv:2205.13147) — train once so that the first 64/128/256/512 dimensions of the vector are each a usable embedding. Lets you truncate at query time for cheap ANN search. Adopted by OpenAI's text-embedding-3 line and by Nomic Embed v1.5 in 2024.

All of these are conceptually "Word2Vec, scaled up, with a transformer instead of a lookup table, and a contrastive objective instead of skip-gram". Same core bet: learn a geometry where cosine similarity means semantic similarity, and then all downstream problems become nearest-neighbour lookups.

9 · Word2Vec's descendants — the family tree

Word2Vec did not stay king for long. Its lineage:

• GloVe (Pennington, Socher, Manning, Stanford 2014, nlp.stanford.edu/pubs/glove.pdf) — reformulated the same idea as factorising a log-count co-occurrence matrix. Comparable performance, different mathematical framing. Trained on Common Crawl 840B tokens; the resulting 300D vectors are still downloadable and still occasionally used as init.
• fastText (Bojanowski, Grave, Joulin, Mikolov again, Facebook 2016, arXiv:1607.04606) — represent each word as a bag of character n-grams, so out-of-vocabulary words get sensible vectors from their morphology. unfriendliness = mean of vectors for un, friend, friendly, ness, etc.
• ELMo (Peters et al. AllenNLP 2018, arXiv:1802.05365) — first widely used contextual embeddings: a word's vector depends on its sentence. Killed Word2Vec for downstream NLP.
• BERT (Devlin et al., Google 2018) → GPT-2/3/4 → Claude / Llama / DeepSeek — every one of these begins with an embedding table. Modern LLMs use subword tokenization (BPE, SentencePiece — see M08), so the "words" being embedded are actually 30k–200k subword pieces. But the primitive is identical: nn.Embedding(vocab_size, d_model), first line of the forward pass.

You will never train a stand-alone Word2Vec in production in 2025. The knowledge is not obsolete though. It is the foundation. When you read that Llama 4 uses a 128k-token vocabulary with 16,384-dim embeddings (a 128000 × 16384 = 2.1B-parameter table), you're looking at Mikolov's lookup table, scaled up 20,000× in dimensionality and 12× in vocab size. The mechanics haven't changed.

10 · War stories from the trenches

11 · Try it yourself

12 · Recap in five sentences

Word2Vec proved that a shallow network trained only to predict a word's neighbours will spontaneously learn a vector geometry in which meaning corresponds to direction. Skip-gram with negative sampling — one lookup, k+1 dot products, a sigmoid — makes this fast enough to run on billions of tokens. Three empirical constants (window=5, k=5, α=0.75, t=1e-5) plus frequency subsampling turn a mathematical idea into a robust recipe. The output geometry has near-linear structure that supports vector arithmetic on concepts, which was the first hint of what we now call the Linear Representation Hypothesis. Every modern LLM begins its forward pass with the exact primitive Mikolov invented at lunch in 2013: nn.Embedding(vocab_size, d_model).

🧠 Retention scaffold

One-line summary (write it in your own words): ________________________________________________

Spaced review: re-read Sections 2 (distributional hypothesis) and 7.3 (why analogy works) in 24 hours. Revisit the full session on day 7.

Next session (S032): we build an RNN from scratch to model sequences of these vectors — because a bag of embeddings can tell you what a document is about, but only a recurrent state can tell you the order in which the words were said.

Sticky note (keep on your desk): An embedding table is a (vocab, dim) matrix + row lookup. Train it to predict context, and geometry becomes meaning. Every LLM is built on this.

Further reading

• Original papers. Mikolov et al., Word2Vec (2013) and the negative sampling follow-up.
• GloVe alternative. Pennington, Socher, Manning 2014, GloVe paper — matrix factorization reformulation.
• fastText for morphology. Bojanowski et al. 2016, arXiv:1607.04606.
• Linear representation hypothesis. Park, Choe, Veitch 2024, arXiv:2311.03658.
• Bias in embeddings. Bolukbasi et al. 2016, arXiv:1607.06520; Bender et al. 2021, On the Dangers of Stochastic Parrots.
• Modern feature geometry in LLMs. Anthropic, Scaling Monosemanticity to Claude 3 Sonnet (2024).
• Karpathy Zero-to-Hero. Building makemore Part 2 — MLP. The best 80 minutes on embedding intuition on the internet.

Previous: ← DL S030 · Next: DL S032 →