DL S074 · Multimodal — CLIP → LLaVA
Build a CLIP-style dual-encoder from scratch, then bolt a vision encoder onto an LLM to make it see. Part of the 'Deep Learning & LLMs From Scratch' 80-session self-study series.
🎯 Build a CLIP-style dual-encoder from scratch, then understand exactly how LLaVA gives an LLM eyes with a single linear projection.
Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 74 / 80 · Module M12 · ~2 hours
The story we're starting with
Here's a puzzle. You have a language model. It reads text tokens and predicts text tokens. That's the whole trick — for 73 sessions we've been treating it as a program that pushes integers through a big pile of matrix multiplies. Now someone hands you a JPEG and says: "make the model describe this photo."
Naively you'd think: "okay, new architecture, new training run, throw away everything." That's what people did in 2019. Then in 2021 OpenAI dropped CLIP, and in 2023 the LLaVA folks noticed something so cheap it borders on insulting: if you already have a vision model that produces a vector for an image, and you already have a language model that eats vectors (that's what token embeddings are — vectors), the only thing between them is a linear layer. One nn.Linear. That's the bridge. That's how GPT-4V and Gemini and every open multimodal model actually works under the hood.
The reason it works comes down to CLIP. CLIP taught us that if you train an image encoder and a text encoder to produce vectors that live in the same space — such that "a photo of a cat" and a photo of a cat point in the same direction — you get an image encoder whose outputs already speak a language the language model can understand. LLaVA then just says: freeze both, learn a projection, done. Today we're going to build the CLIP training objective from scratch (it's 20 lines), then walk through the LLaVA architecture end-to-end so that when you fine-tune your capstone model on image-text pairs next week, nothing is a mystery.
- Derive the CLIP contrastive loss from first principles and implement it in ~15 lines of PyTorch.
- Explain why CLIP uses a symmetric loss (image-to-text + text-to-image) and what breaks if you drop one direction.
- Draw the LLaVA architecture — vision encoder → projector → LLM — and say what each part is trained/frozen at each stage.
- Compute the number of visual tokens LLaVA injects for a 336×336 image with a ViT-L/14 backbone.
- Name at least two failure modes of naive multimodal fine-tuning and how the LLaVA-1.5 paper fixed them.
Prerequisites
- S036–S040 · Transformers. You need to be fluent with attention and how token embeddings enter a transformer.
- S030 · Vision transformers. We treat images as sequences of patch tokens. This session assumes you've seen a ViT once.
- S062 · Fine-tuning basics. The last stage of LLaVA is instruction fine-tuning; you should know what that means.
- S015 · Cross-entropy and softmax. The CLIP loss is two cross-entropies bolted together.
1 · The CLIP idea in one paragraph
Take 400 million image-caption pairs scraped off the internet. For each pair, feed the image through an image encoder (a ViT) and the caption through a text encoder (a transformer). You now have two vectors per pair. Train both encoders such that matching image-caption pairs produce vectors with high cosine similarity, and non-matching pairs produce vectors with low similarity. That's it. That's CLIP.
The genius is in how you do the "non-matching" part. You don't need to construct negative examples by hand. Every training batch of size N already contains N×N pairs: N of them are the real matches (the diagonal), and N²−N are automatic negatives (everything off-diagonal). Free supervision at O(N²) density for O(N) data cost.
1.1 · The math, slowly
Let I be the batch of image embeddings, shape (N, d). Let T be the batch of text embeddings, shape (N, d). Both L2-normalized so each row is a unit vector.
Compute the similarity matrix:
S = I @ T.T # shape (N, N), S[i,j] = cos(image_i, text_j)Scale by a learned temperature τ (typically τ ≈ 0.07 at init, exponentiated so it stays positive):
logits = S / τ # (N, N)Now interpret each row as a classification over N candidate captions:
loss_i2t = cross_entropy(logits, labels=[0, 1, 2, ..., N-1])And each column as a classification over N candidate images:
loss_t2i = cross_entropy(logits.T, labels=[0, 1, 2, ..., N-1])Final loss:
loss = (loss_i2t + loss_t2i) / 2That's the whole thing. Ten lines.
1.2 · Numeric worked example
Take N=3, d=2. Say after the encoders and L2 normalization:
I = [[ 1, 0], [0, 1], [ 0.7, 0.7]]
T = [[ 0.9, 0.1], [0.1, 0.9], [0.6, 0.8]]Compute S = I @ T.T:
S = [[0.90, 0.10, 0.60],
[0.10, 0.90, 0.80],
[0.70, 0.70, 0.98]]The diagonal is where the "correct" pairs live: 0.90, 0.90, 0.98. Good — they're the largest values in their rows and columns. If τ = 0.1, logits = S / 0.1 = [[9,1,6],[1,9,8],[7,7,9.8]]. Softmax rows, take log of the diagonal, negate — that's your per-row loss. You'll get a small number because the diagonal dominates. Now imagine the encoders had produced garbage embeddings so the diagonal was 0.1 and off-diagonal was 0.9. The loss would be huge. Training pushes the diagonal up.
1.3 · Implementing it
Here's the pure PyTorch version. First the encoders (stubbed — real ones are a ViT and a text transformer we've built in earlier sessions):
import torch, torch.nn as nn, torch.nn.functional as F
class CLIP(nn.Module):
def __init__(self, image_encoder, text_encoder, d_embed=512):
super().__init__()
self.image_enc = image_encoder # returns (B, d_img)
self.text_enc = text_encoder # returns (B, d_txt)
self.image_proj = nn.Linear(image_encoder.out_dim, d_embed, bias=False)
self.text_proj = nn.Linear(text_encoder.out_dim, d_embed, bias=False)
# log-temperature so it stays positive after exp
self.log_tau = nn.Parameter(torch.tensor(2.6592)) # 1/0.07 ≈ e^2.66The two projection heads exist because the encoders can have different native dimensionalities (e.g. 768 for ViT-B, 512 for a GPT-2-sized text encoder). We map both into a shared d_embed.
Now the forward + loss:
def forward(self, images, tokens):
img = self.image_enc(images) # (B, d_img)
txt = self.text_enc(tokens) # (B, d_txt)
img = F.normalize(self.image_proj(img), dim=-1) # (B, d)
txt = F.normalize(self.text_proj(txt), dim=-1) # (B, d)
tau = self.log_tau.exp().clamp(max=100.0) # (safety cap)
logits = tau * img @ txt.t() # (B, B)
targets = torch.arange(len(img), device=img.device)
loss_i2t = F.cross_entropy(logits, targets)
loss_t2i = F.cross_entropy(logits.t(), targets)
return (loss_i2t + loss_t2i) / 2, logitsNotice log_tau is learned. The network can decide how "sharp" it wants the similarity distribution to be. Early in training it'll want a soft (small τ inverse) distribution; late in training it sharpens.
Build an 8×8 similarity matrix by generating 8 random unit vectors and computing pairwise cosines. Softmax the rows at τ = 1.0, 0.1, 0.01. At τ = 1.0 the distribution is nearly uniform; at τ = 0.01 it collapses onto whichever off-diagonal entry happens to be largest. That's the peril the paper's learned-τ clamp (max ~4.6, i.e. τ ≥ 0.01) was designed to prevent — without the clamp, the model can drive itself into overconfidence and stop learning.
First time I implemented this I skipped the F.normalize calls. Training loss plummeted in 100 steps — I thought I was a genius. Then eval accuracy was 12%. What happened: without normalization the model just scaled up its embeddings until the diagonal was arbitrarily large and won cross-entropy trivially. Zero-shot performance is garbage because there's no shared 'unit sphere' the encoders agreed on. The L2 normalization + temperature is the whole reason CLIP works as a retrieval model.
2 · Why the symmetric loss?
You might ask: if we're already training on the (N, N) similarity matrix, why do we need both directions? The intuition: the two cross-entropies enforce different things.
cross_entropy(logits, targets)says: for each image, its true caption should score higher than every other caption in the batch. This teaches the image encoder to produce vectors that discriminate against distractor captions.cross_entropy(logits.T, targets)says: for each caption, its true image should score higher than every other image in the batch. This teaches the text encoder to discriminate against distractor images.
Drop one direction and one encoder gets lazy. In practice: drop t2i and the text encoder collapses to a low-rank subspace because nothing forces it to distinguish images. The retrieval numbers on Flickr30k drop by 8–10 points.
3 · From CLIP to LLaVA — bolting eyes onto an LLM
Now the trick. LLaVA takes:
- A frozen CLIP ViT-L/14 (the image encoder half of a pretrained CLIP model).
- A frozen LLM (originally Vicuna-13B; you can use your own capstone model).
- A learnable projection
W ∈ ℝ^{d_vision × d_llm}.
At inference, given an image and a text prompt:
For a 336×336 image with ViT-L/14 (patch size 14), you get (336/14)² = 576 visual "tokens". Those 576 vectors get shoved into the LLM's context window as if they were text embeddings for the first 576 positions, followed by the actual text prompt embeddings.
3.1 · The projection layer, in code
class LlavaProjector(nn.Module):
def __init__(self, d_vision=1024, d_llm=4096):
super().__init__()
# LLaVA-1.0 used a single Linear.
# LLaVA-1.5 upgraded to a 2-layer MLP with GELU — big quality jump.
self.proj = nn.Sequential(
nn.Linear(d_vision, d_llm),
nn.GELU(),
nn.Linear(d_llm, d_llm),
)
def forward(self, v): # v: (B, n_patches, d_vision)
return self.proj(v) # (B, n_patches, d_llm)That's the entire "multimodal" component. A GELU-sandwich. Two nn.Linears and one activation. That's what makes GPT-4V-like behavior possible on a laptop.
3.2 · The training recipe
LLaVA training is two stages:
Stage 1: feature alignment. Freeze both encoders. Train only the projector on ~600k image-caption pairs (LAION + CC3M subset). Objective: standard LLM next-token loss on the caption, conditioned on the projected visual tokens. This teaches W to translate "vision space" into a language the LLM already speaks.
Stage 2: visual instruction tuning. Unfreeze the LLM (or LoRA it), keep the ViT frozen. Train on 158k GPT-4-generated instruction-following samples ("Describe this image", "What's unusual about this photo?", "Convert this chart to a table"). The projector continues to train too.
Total compute: 8 A100s, ~1 day per stage. That's it. You can literally build a functional GPT-4V competitor on rented GPUs for under $500.
4 · Diagram — the full LLaVA pipeline
Note the two "frozen" boxes. That's why LLaVA is trainable on modest hardware.
5 · Common pitfalls
A classic LLaVA-1.0 failure: ask 'how many people are in this image?' and it says '3' when there are 7. Root cause: 576 tokens is not many, and average-pooling any of them destroys spatial info. LLaVA-1.5 fixed this partially by using an unpooled grid; further fixes (LLaVA-NeXT) use a dynamic tiling scheme to give the LLM higher-resolution regions.
If Stage 2 instruction data is skewed toward long descriptive answers, the model learns to always produce a paragraph, even when the image is just a red square. Mix in refusal samples ('I cannot determine that from the image') and short-answer samples. LLaVA-1.5's data mix is public — copy it.
576 tokens per image is a lot when you also want a 4k context window. If you batch 8 images and 4k text, that's 4608 visual tokens per sample. KV cache blows up. Fix: use a Q-former (BLIP-2 style) that compresses vision tokens down to 32 or 64 learned queries. Trade-off: some spatial resolution loss.
6 · The 2024–2025 landscape — where CLIP/LLaVA went next
The recipe you just built is 2023. Since then the field moved on four fronts. You should know the names because papers you read this year will drop them without explanation.
6.1 · SigLIP and SigLIP-2 (Google, 2023 → 2025)
CLIP's softmax over the whole batch is expensive: the (N,N) similarity matrix means every GPU has to see every text embedding. Zhai et al. (2023, Sigmoid Loss for Language Image Pre-Training, arXiv:2303.15343) replaced softmax with a sigmoid per-pair loss. Each (image, text) cell becomes an independent binary classification: "do these go together, yes or no?" No cross-batch normalization, so batches shard trivially across devices. SigLIP hit the same or better zero-shot ImageNet accuracy with 4× smaller batches.
SigLIP-2 (Tschannen et al., Feb 2025, arXiv:2502.14786) added a captioning decoder loss on top of the sigmoid contrastive loss, self-distillation from a stronger teacher, and better multilingual coverage. It's now the default vision tower inside most 2025 open-source VLMs (PaliGemma-2, several LLaVA-OV variants).
6.2 · BLIP-2 and the Q-Former (Salesforce, 2023)
Remember the "576 tokens is a lot" war story? Li et al. (2023, arXiv:2301.12597) introduced the Q-Former — a small transformer with 32 learned query vectors that cross-attend to the ViT patch tokens and compress them down to 32 output vectors. You lose some spatial fidelity but you get an 18× shorter visual prefix. Every downstream VLM that has to fit many images into a context window (video understanding especially) uses some variant of this.
6.3 · LLaVA-NeXT → LLaVA-OneVision (2024)
LLaVA-NeXT (Jan 2024) fixed high-resolution by AnyRes tiling: split a big image into 336×336 sub-tiles, encode each, concatenate. A 672×672 image becomes 4 tiles × 576 tokens + a low-res global view. Counting, OCR, and chart-QA jumped 20–30 points.
LLaVA-OneVision (Li et al., Aug 2024, arXiv:2408.03326) unified single-image, multi-image, and video into one model with the same architecture, using different token-budget schedules per modality. It's the reference open-source baseline for 2025 VLM research — start there before writing your own.
6.4 · Molmo and PixMo (Ai2, Sep 2024)
Deitke et al. (arXiv:2409.17146) showed you don't need billions of GPT-4-generated instruction samples. Their PixMo dataset (~1M highly curated, human-annotated image-caption + pointing samples) trained a family of Molmo models where Molmo-72B beat GPT-4V and Gemini 1.5 Pro on 11 academic benchmarks. Key trick: audio-transcribed dense captions (annotators spoke long descriptions instead of typing short ones) — much richer supervision per sample. If you're building your own VLM dataset in 2025, read PixMo before you write a labeling doc.
6.5 · Qwen2.5-VL and native dynamic resolution (Alibaba, Jan 2025)
Qwen2.5-VL (Bai et al., arXiv:2502.13923) drops the tile-and-stitch hack. It uses M-RoPE (multimodal rotary position encoding) with separate temporal, height, and width axes, and a ViT that accepts arbitrary aspect ratios natively. It also encodes absolute time for video (not just frame index), so "what happened around 3:47?" works. Qwen2.5-VL-72B is currently (mid-2025) the strongest open-weights VLM on OCR and document understanding.
6.6 · Further reading (URLs)
- CLIP: https://arxiv.org/abs/2103.00020 · Radford et al. 2021
- LLaVA-1.5: https://arxiv.org/abs/2310.03744 · Liu et al. 2023
- SigLIP: https://arxiv.org/abs/2303.15343 · Zhai et al. 2023
- SigLIP-2: https://arxiv.org/abs/2502.14786 · Tschannen et al. 2025
- BLIP-2 (Q-Former): https://arxiv.org/abs/2301.12597 · Li et al. 2023
- LLaVA-OneVision: https://arxiv.org/abs/2408.03326 · Li et al. 2024
- Molmo / PixMo: https://arxiv.org/abs/2409.17146 · Deitke et al. 2024 · https://molmo.allenai.org/
- Qwen2.5-VL: https://arxiv.org/abs/2502.13923 · Bai et al. 2025
- HF VLM leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
7 · Try it yourself
The cheapest path from "read this session" to "trained a VLM once":
pip install transformers accelerate datasets- Load a SigLIP-2 vision tower and a TinyLlama-1.1B LLM.
- Wire up the 2-layer MLP projector (60 lines).
- Take 10k image-caption pairs from LAION-COCO (free, on HuggingFace).
- Train the projector for 1 epoch on a single T4 (Colab free tier will do). About 90 minutes.
- Prompt it with 5 personal photos. Does it caption them coherently?
Expected result: broad-strokes captions correct ("a dog on grass"), fine details wrong (breed, count, colors). That's exactly what LLaVA-1.0 saw in 2023 — you've just reproduced the state-of-the-art from two years ago in an afternoon. Now you understand what LLaVA-NeXT + PixMo needed to fix.
8 · Recap
CLIP taught two encoders to agree on a shared vector space using a symmetric batch-contrastive loss — a free O(N²) supervision signal from O(N) data. LLaVA then noticed that if the vision encoder already speaks a language, all you need is one small MLP to translate it into the LLM's embedding space. Everything since — SigLIP-2, Q-Former, AnyRes, native-resolution ViTs, PixMo — is the same architecture with better data, better tokens-per-image budgets, or better position encodings. The next frontier (video, tools-in-VLM, action tokens) is the same trick again: whatever you can encode into a vector that lives near the LLM's embedding space, the LLM can reason about.
"CLIP learns to describe images. Its embedding of a photo captures what's in the photo, so nearest-neighbour search in that space finds visually similar images and the text tower is just a convenient way to query it."
CLIP learns only what the contrastive objective rewards: making a matched image-text pair score higher than the other pairs in the same batch. It captures whatever features discriminate one caption from the other captions it was contrasted against, and nothing more. Attributes that never distinguish captions — counting, spatial relations, negation, fine-grained attribute binding — are simply not learned, because getting them wrong never cost anything during training.
The myth is seductive because CLIP's zero-shot behaviour is genuinely astonishing on the things it does know, and success on a wide range of categories reads as general understanding. It is also reinforced by the interface: you hand it arbitrary text and get a sensible score, which feels like comprehension rather than matching. The precise reason it is wrong is that the contrastive loss is a ranking objective over a batch, not a reconstruction or prediction objective. A feature is learned exactly to the extent that it separates the correct caption from the negatives you happened to sample. If "three dogs" and "two dogs" almost never appear in the same batch, the count carries no gradient — and the model that never had to learn it will confidently produce a high similarity for both.
Probe for a property the objective never rewarded:
import torch, clip
model, prep = clip.load('ViT-B/32')
img = prep(Image.open('two_cats.jpg')).unsqueeze(0)
texts = ['two cats', 'three cats', 'a cat',
'a photo with no cats', 'a cat on the left of a dog',
'a dog on the left of a cat']
t = clip.tokenize(texts)
with torch.no_grad():
logits, _ = model(img, t)
for s, x in zip(logits.softmax(-1)[0].tolist(), texts):
print(round(s, 3), x)Counting, negation, and left-right order are where the scores stop tracking the truth. That is not a bug — it is the objective, faithfully learned.
Why does the contrastive loss need to be symmetric — image-to-text and text-to-image — when both directions score the same similarity matrix? It looks like doing the same thing twice.
- 1The similarity matrix holds every image-text pair in the batch; the correct pairs are on the diagonal and everything off-diagonal is a negative.forced by · random pairings across a batch are almost certainly mismatched, which gives O(N²) supervision from O(N) labelled examples
- 2A one-directional loss applies softmax across one axis only — say, for each image, over all captions. That asks: given this image, which caption is right?forced by · softmax normalises along a single axis, so gradients only push apart the entries in that row
- 3That objective is satisfiable by a degenerate solution: a caption embedding that is close to every image is never penalised, because it is never the thing being ranked.forced by · a text vector's own row is never normalised over, so nothing constrains how many images it can be near
- 4Adding the transposed loss asks the mirror question — given this caption, which image is right — which penalises exactly that hub behaviour.forced by · a caption near many images now loses in every one of those columns, so the degenerate solution becomes costly
- 5The two directions together force the map to be approximately bijective on the batch: each embedding must be near its partner and far from all others, in both roles.forced by · a matching, rather than a one-sided ranking, is what makes the space usable for retrieval in either direction
Therefore symmetry is not a redundancy but the constraint that rules out hub embeddings and makes the shared space genuinely shared, rather than a text index into images or the reverse.
And note what this predicts. First: batch size should matter enormously and in a specific way, because the number of negatives per positive is the batch size — so the useful signal scales with it, and small-batch contrastive training should underperform badly rather than merely converge slower. Second: since supervision comes only from in-batch negatives, the composition of the batch should matter as much as its size — a batch of near-duplicate captions provides almost no gradient, which is why hard-negative construction is a lever. Third: the temperature parameter should be load-bearing rather than cosmetic, since it sets how sharply the softmax distinguishes the diagonal from near-misses; too high and everything is equally close, too low and only the hardest negative contributes. All three are things you can verify by ablation on a small run.
Picture a single vector space as a room with two doors into it. Images enter through one encoder, text through the other. Training does one thing only: it drags each matched pair together and shoves every unmatched in-batch pair apart, in both directions at once. After enough of that, position in the room means something both doors agree on.
Everything multimodal since is the same move. LLaVA notices that if a vision encoder already produces points in a room the language model understands, one small projection is enough to place image tokens directly into the LLM's own embedding sequence — the LLM then treats them as words it happens not to have a spelling for.
- Alignment is learned by contrast, not by description. The model knows exactly the distinctions its negatives forced it to make.
- Supervision scales with in-batch negatives, so batch size and batch composition are architecture-level decisions, not training details.
- Symmetry prevents hub embeddings. A one-sided loss admits a vector that is close to everything.
- Once something is a vector near the LLM's embedding space, the LLM can reason about it — which is why the same recipe extends to audio, video, and actions.
- The projection layer is small because the hard work is already done by the encoders. Freeze what is trained, train what translates.
Fire this the moment you see: a retrieval system that fails on counts, negation, or spatial relations · zero-shot classification where the label wording changes the answer more than the image does · a contrastive model trained with a small batch and disappointing results · a VLM that describes an image fluently but gets attribute binding wrong · someone treating an image embedding as a complete description rather than as a discriminative summary · a plan to add a new modality (start by asking what it is contrasted against).
You are connecting a vision encoder to an LLM. How much of each side do you train?
Work outward from the projection. Train it alone first, because it is cheap and it tells you where the ceiling is; if the result is bad in a way that looks like the model cannot see something, the vision features are the bottleneck, and if it looks like the model sees but will not follow instructions, the LLM is. Those two failures have opposite fixes and are easy to distinguish once you look for the distinction.
The evaluation trap is worth stating plainly: multimodal fine-tuning degrades text-only ability, and a benchmark suite consisting entirely of image tasks will report success while the model gets worse at half of what you ship. Keep a text-only regression set from before you started and run it every time. The other durable lesson is about tokens per image — visual tokens consume context and compute exactly like text tokens do, so resolution is not a free quality knob but a direct trade against how much room is left for the conversation.
🧠 Retention scaffold
One-line summary (write it in your own words): _______________________________
Spaced review: re-read §1 (CLIP loss) + §3 (LLaVA projector) in 24 hours. Revisit the full session on day 7 alongside the SigLIP-2 paper.
Next session (S075): we teach the same LLM to act — call a Python interpreter, hit a search API, plan multi-step tasks. Same trick as LLaVA in spirit: tools just produce tokens.
Sticky note (keep on your desk): A multimodal model is a text LLM with one extra nn.Linear glued to whatever encoder you can find. Everything else is data quality and token budget.
Previous: ← DL S073 · Next: DL S075 → Agents & Tool Use