S104 · Embeddings — word2vec, GloVe, Contrastive Learning

Module M12: Deep Learning · Session 104 of 130 · Track: ML 📐

What you'll be able to do after this session

• Explain Embeddings to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S103 — RNNs & LSTMs — Sequences & the Vanishing Gradient

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Word Embedding and Word2Vec, Clearly Explained — StatQuest — Josh Starmer's plain-English word2vec walkthrough.
• 🎥 Deep dive (30–60 min) · Stanford CS224n — Word Vectors and Word Senses — Chris Manning derives word2vec from scratch.
• 🎥 Hands-on demo (10–20 min) · Train word2vec with Gensim in 10 minutes — Copy-paste-runnable demo on a real corpus.
• 📖 Canonical article · The Illustrated Word2vec — Jay Alammar — The clearest visual explanation of skip-gram + negative sampling online.
• 📖 Book chapter / tutorial · Efficient Estimation of Word Representations (Mikolov 2013) — The original word2vec paper — short and readable.
• 📖 Engineering blog · Google Research — Transformer, a novel neural network architecture — Frames why embeddings seed every modern NLP model.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Turning things into vectors.

Neural networks eat numbers. But most of the world's interesting data — words, users, products, songs, molecules — is discrete symbols. An embedding is the trick that turns any discrete thing into a dense vector of real numbers in some learned space, such that similar things end up close together. "king" and "queen" become nearby vectors. Two customers who buy similar products become nearby vectors. Two songs listeners often play together become nearby vectors. Everything downstream — classification, search, recommendation, clustering — becomes a matter of measuring distances or dot products in that space.

The magic isn't the vectors themselves; it's that we learn them jointly with a task. Word2vec (Mikolov 2013) discovered that if you train a shallow net to predict a word from its context (or vice versa) on a giant corpus, the side effect is that the hidden layer's weights become vectors with striking algebraic structure: king − man + woman ≈ queen. Nobody told the model about analogies; they emerged from the co-occurrence statistics of language.

Two decades later, every foundation model — CLIP for images, sentence-transformers for search, protein-embedding models — is at heart the same idea: define a contrastive or predictive task, train a large net, and the intermediate representation is the product.

Gotcha to carry forever: Cosine similarity ≠ Euclidean distance. Embeddings are usually L2-normalised and compared with cosine similarity (or equivalently dot product on normalised vectors). Comparing raw Euclidean distances between un-normalised embeddings is one of the most common vector-search bugs on the planet.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

How word2vec skip-gram actually works. You have a corpus. For every centre word, you look at a window of neighbours (say ±2) and train a network to predict those neighbours from the centre word:

After training on billions of tokens, the embedding matrix E (vocab_size × 300) is the model — you throw the output layer away. Rows are word vectors.

Worked example — the analogy trick. Take pretrained GloVe vectors. Compute v("king") − v("man") + v("woman"). Look up the nearest word by cosine similarity. Result (with high probability): queen. This works because the "gender axis" is encoded as a roughly constant direction in the embedding space — it lives in vector-arithmetic land.

Worked example — contrastive learning. For images, CLIP takes a batch of N (image, caption) pairs, computes N image embeddings and N text embeddings, and trains so that (a) the diagonal of the N × N similarity matrix is high (matched pairs) and (b) the off-diagonal is low (mismatched pairs). Same idea, different modality.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

# embeddings_demo.py — train word2vec on a tiny corpus, do analogies.
# pip install gensim numpy
from gensim.models import Word2Vec
import numpy as np
 
sentences = [
    "the king ruled the kingdom".split(),
    "the queen ruled the kingdom".split(),
    "the man walked to the market".split(),
    "the woman walked to the market".split(),
    "the boy played in the park".split(),
    "the girl played in the park".split(),
    "the king and the queen ruled together".split(),
    "the man and the woman walked together".split(),
] * 500                                       # repeat for enough signal
 
model = Word2Vec(sentences, vector_size=50, window=3, min_count=1,
                 sg=1, negative=5, epochs=100, seed=42)
 
def cos(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
print("king vs queen :", cos(model.wv["king"],  model.wv["queen"]))
print("king vs market:", cos(model.wv["king"],  model.wv["market"]))
 
try:
    analogy = model.wv.most_similar(positive=["king", "woman"],
                                    negative=["man"], topn=3)
    print("king - man + woman ~", analogy)
except KeyError as e:
    print("word missing:", e)
 
print("nearest to 'walked':", model.wv.most_similar("walked", topn=3))
print("vocab size:", len(model.wv.key_to_index))

Checklist while it runs:

1. Confirm model.wv["king"].shape == (50,).
2. cos(king, queen) should be markedly higher than cos(king, market) after training.
3. On a toy corpus the analogy may be shaky; on a real 1B-token corpus it's rock solid.
4. Try sg=0 (CBOW mode) and see how similarities change.
5. Bump vector_size=200 and note runtime.

Try this modification: Project vectors with sklearn.decomposition.PCA(n_components=2) and matplotlib.scatter the words. Related words should cluster visibly.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Embeddings are the invisible backbone of modern search, recommendation, and RAG. Almost every "AI-powered" feature you use today is, under the hood, "encode a query into a vector and find the nearest N vectors in a giant index".

Vector search at scale. Storing 100M 768-dim float32 vectors is 300GB, and brute-force cosine similarity across 100M vectors per query is impossibly slow. Production systems use approximate nearest-neighbour (ANN) libraries — FAISS (Meta), ScaNN (Google), HNSWlib, or hosted services like Pinecone, Weaviate, and pgvector — which trade a tiny bit of recall (99% vs 100%) for 100–1000× speedups via graph or product-quantisation indexes.

Bugs that cost real money:

• Not L2-normalising vectors before feeding to a cosine index. Two vectors of very different magnitude will look "similar" for the wrong reason.
• Mixing embedding models across write and read paths. If you re-encode your document corpus with a new model but forget to update the query encoder, your search silently rots.
• Freshness lag. Product catalogues change hourly, but re-embedding takes hours. Incremental indexing (only re-embed changed items) is a real engineering problem.
• Distribution drift. Uber, Airbnb, and Netflix all publish on this: an embedding trained on last quarter's data slowly stops matching this quarter's user behaviour. You need continuous retraining or fine-tuning.

Modern picture. For text, text-embedding-3-large (OpenAI) and bge-large-en (BAAI) have essentially replaced word2vec — they produce sentence-level embeddings that dominate benchmarks. For images, CLIP and its descendants. For users/products, in-house two-tower models trained on click logs. But the mental model — "learn a space where similar things are close" — is exactly the same as Mikolov's 2013 paper. Google's blog on the Transformer notes explicitly that the first thing the Transformer does with a token is look it up in an embedding table — the entire LLM is built on that foundation.

(e) Quiz + exercise · 10 min

1. In one sentence, what is an embedding and what property must it satisfy?
2. Why is cosine similarity usually preferred over Euclidean distance for comparing word embeddings?
3. What does the skip-gram objective in word2vec actually predict?
4. Why does king − man + woman ≈ queen work? What structural property of the space is this exploiting?
5. What is an approximate nearest-neighbour (ANN) index, and why do you need one at 100M-vector scale?

Stretch (connects to S103 — RNNs & LSTMs): In a sequence model, the input embedding table is often the largest single tensor. Why, and what techniques (e.g. weight tying) do modern LLMs use to shrink it?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Embeddings? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S105): Transfer Learning & Fine-Tuning Classical DL
• Previous (S103): RNNs & LSTMs — Sequences & the Vanishing Gradient
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.