DL S074 · Multimodal — CLIP → LLaVA

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 74 / 80 · Module M12 · ~2 hours

The story we're starting with

Here's a puzzle. You have a language model. It reads text tokens and predicts text tokens. That's the whole trick — for 73 sessions we've been treating it as a program that pushes integers through a big pile of matrix multiplies. Now someone hands you a JPEG and says: "make the model describe this photo."

Naively you'd think: "okay, new architecture, new training run, throw away everything." That's what people did in 2019. Then in 2021 OpenAI dropped CLIP, and in 2023 the LLaVA folks noticed something so cheap it borders on insulting: if you already have a vision model that produces a vector for an image, and you already have a language model that eats vectors (that's what token embeddings are — vectors), the only thing between them is a linear layer. One nn.Linear. That's the bridge. That's how GPT-4V and Gemini and every open multimodal model actually works under the hood.

The reason it works comes down to CLIP. CLIP taught us that if you train an image encoder and a text encoder to produce vectors that live in the same space — such that "a photo of a cat" and a photo of a cat point in the same direction — you get an image encoder whose outputs already speak a language the language model can understand. LLaVA then just says: freeze both, learn a projection, done. Today we're going to build the CLIP training objective from scratch (it's 20 lines), then walk through the LLaVA architecture end-to-end so that when you fine-tune your capstone model on image-text pairs next week, nothing is a mystery.

Prerequisites

• S036–S040 · Transformers. You need to be fluent with attention and how token embeddings enter a transformer.
• S030 · Vision transformers. We treat images as sequences of patch tokens. This session assumes you've seen a ViT once.
• S062 · Fine-tuning basics. The last stage of LLaVA is instruction fine-tuning; you should know what that means.
• S015 · Cross-entropy and softmax. The CLIP loss is two cross-entropies bolted together.

1 · The CLIP idea in one paragraph

Take 400 million image-caption pairs scraped off the internet. For each pair, feed the image through an image encoder (a ViT) and the caption through a text encoder (a transformer). You now have two vectors per pair. Train both encoders such that matching image-caption pairs produce vectors with high cosine similarity, and non-matching pairs produce vectors with low similarity. That's it. That's CLIP.

The genius is in how you do the "non-matching" part. You don't need to construct negative examples by hand. Every training batch of size N already contains N×N pairs: N of them are the real matches (the diagonal), and N²−N are automatic negatives (everything off-diagonal). Free supervision at O(N²) density for O(N) data cost.

1.1 · The math, slowly

Let I be the batch of image embeddings, shape (N, d). Let T be the batch of text embeddings, shape (N, d). Both L2-normalized so each row is a unit vector.

Compute the similarity matrix:

S = I @ T.T          # shape (N, N), S[i,j] = cos(image_i, text_j)

Scale by a learned temperature τ (typically τ ≈ 0.07 at init, exponentiated so it stays positive):

logits = S / τ       # (N, N)

Now interpret each row as a classification over N candidate captions:

loss_i2t = cross_entropy(logits, labels=[0, 1, 2, ..., N-1])

And each column as a classification over N candidate images:

loss_t2i = cross_entropy(logits.T, labels=[0, 1, 2, ..., N-1])

Final loss:

loss = (loss_i2t + loss_t2i) / 2

That's the whole thing. Ten lines.

1.2 · Numeric worked example

Take N=3, d=2. Say after the encoders and L2 normalization:

I = [[ 1, 0], [0, 1], [ 0.7, 0.7]]
T = [[ 0.9, 0.1], [0.1, 0.9], [0.6, 0.8]]

Compute S = I @ T.T:

S = [[0.90, 0.10, 0.60],
     [0.10, 0.90, 0.80],
     [0.70, 0.70, 0.98]]

The diagonal is where the "correct" pairs live: 0.90, 0.90, 0.98. Good — they're the largest values in their rows and columns. If τ = 0.1, logits = S / 0.1 = [[9,1,6],[1,9,8],[7,7,9.8]]. Softmax rows, take log of the diagonal, negate — that's your per-row loss. You'll get a small number because the diagonal dominates. Now imagine the encoders had produced garbage embeddings so the diagonal was 0.1 and off-diagonal was 0.9. The loss would be huge. Training pushes the diagonal up.

1.3 · Implementing it

Here's the pure PyTorch version. First the encoders (stubbed — real ones are a ViT and a text transformer we've built in earlier sessions):

import torch, torch.nn as nn, torch.nn.functional as F
 
class CLIP(nn.Module):
    def __init__(self, image_encoder, text_encoder, d_embed=512):
        super().__init__()
        self.image_enc = image_encoder    # returns (B, d_img)
        self.text_enc  = text_encoder     # returns (B, d_txt)
        self.image_proj = nn.Linear(image_encoder.out_dim, d_embed, bias=False)
        self.text_proj  = nn.Linear(text_encoder.out_dim,  d_embed, bias=False)
        # log-temperature so it stays positive after exp
        self.log_tau = nn.Parameter(torch.tensor(2.6592))  # 1/0.07 ≈ e^2.66

The two projection heads exist because the encoders can have different native dimensionalities (e.g. 768 for ViT-B, 512 for a GPT-2-sized text encoder). We map both into a shared d_embed.

Now the forward + loss:

    def forward(self, images, tokens):
        img = self.image_enc(images)                   # (B, d_img)
        txt = self.text_enc(tokens)                    # (B, d_txt)
        img = F.normalize(self.image_proj(img), dim=-1)  # (B, d)
        txt = F.normalize(self.text_proj(txt), dim=-1)   # (B, d)
 
        tau = self.log_tau.exp().clamp(max=100.0)      # (safety cap)
        logits = tau * img @ txt.t()                   # (B, B)
 
        targets = torch.arange(len(img), device=img.device)
        loss_i2t = F.cross_entropy(logits,     targets)
        loss_t2i = F.cross_entropy(logits.t(), targets)
        return (loss_i2t + loss_t2i) / 2, logits

Notice log_tau is learned. The network can decide how "sharp" it wants the similarity distribution to be. Early in training it'll want a soft (small τ inverse) distribution; late in training it sharpens.

2 · Why the symmetric loss?

You might ask: if we're already training on the (N, N) similarity matrix, why do we need both directions? The intuition: the two cross-entropies enforce different things.

• cross_entropy(logits, targets) says: for each image, its true caption should score higher than every other caption in the batch. This teaches the image encoder to produce vectors that discriminate against distractor captions.
• cross_entropy(logits.T, targets) says: for each caption, its true image should score higher than every other image in the batch. This teaches the text encoder to discriminate against distractor images.

Drop one direction and one encoder gets lazy. In practice: drop t2i and the text encoder collapses to a low-rank subspace because nothing forces it to distinguish images. The retrieval numbers on Flickr30k drop by 8–10 points.

3 · From CLIP to LLaVA — bolting eyes onto an LLM

Now the trick. LLaVA takes:

1. A frozen CLIP ViT-L/14 (the image encoder half of a pretrained CLIP model).
2. A frozen LLM (originally Vicuna-13B; you can use your own capstone model).
3. A learnable projection W ∈ ℝ^{d_vision × d_llm}.

At inference, given an image and a text prompt:

For a 336×336 image with ViT-L/14 (patch size 14), you get (336/14)² = 576 visual "tokens". Those 576 vectors get shoved into the LLM's context window as if they were text embeddings for the first 576 positions, followed by the actual text prompt embeddings.

3.1 · The projection layer, in code

class LlavaProjector(nn.Module):
    def __init__(self, d_vision=1024, d_llm=4096):
        super().__init__()
        # LLaVA-1.0 used a single Linear.
        # LLaVA-1.5 upgraded to a 2-layer MLP with GELU — big quality jump.
        self.proj = nn.Sequential(
            nn.Linear(d_vision, d_llm),
            nn.GELU(),
            nn.Linear(d_llm, d_llm),
        )
 
    def forward(self, v):        # v: (B, n_patches, d_vision)
        return self.proj(v)      # (B, n_patches, d_llm)

That's the entire "multimodal" component. A GELU-sandwich. Two nn.Linears and one activation. That's what makes GPT-4V-like behavior possible on a laptop.

3.2 · The training recipe

LLaVA training is two stages:

Stage 1: feature alignment. Freeze both encoders. Train only the projector on ~600k image-caption pairs (LAION + CC3M subset). Objective: standard LLM next-token loss on the caption, conditioned on the projected visual tokens. This teaches W to translate "vision space" into a language the LLM already speaks.

Stage 2: visual instruction tuning. Unfreeze the LLM (or LoRA it), keep the ViT frozen. Train on 158k GPT-4-generated instruction-following samples ("Describe this image", "What's unusual about this photo?", "Convert this chart to a table"). The projector continues to train too.

Total compute: 8 A100s, ~1 day per stage. That's it. You can literally build a functional GPT-4V competitor on rented GPUs for under $500.

4 · Diagram — the full LLaVA pipeline

Note the two "frozen" boxes. That's why LLaVA is trainable on modest hardware.

5 · Common pitfalls

6 · The 2024–2025 landscape — where CLIP/LLaVA went next

The recipe you just built is 2023. Since then the field moved on four fronts. You should know the names because papers you read this year will drop them without explanation.

6.1 · SigLIP and SigLIP-2 (Google, 2023 → 2025)

CLIP's softmax over the whole batch is expensive: the (N,N) similarity matrix means every GPU has to see every text embedding. Zhai et al. (2023, Sigmoid Loss for Language Image Pre-Training, arXiv:2303.15343) replaced softmax with a sigmoid per-pair loss. Each (image, text) cell becomes an independent binary classification: "do these go together, yes or no?" No cross-batch normalization, so batches shard trivially across devices. SigLIP hit the same or better zero-shot ImageNet accuracy with 4× smaller batches.

SigLIP-2 (Tschannen et al., Feb 2025, arXiv:2502.14786) added a captioning decoder loss on top of the sigmoid contrastive loss, self-distillation from a stronger teacher, and better multilingual coverage. It's now the default vision tower inside most 2025 open-source VLMs (PaliGemma-2, several LLaVA-OV variants).

6.2 · BLIP-2 and the Q-Former (Salesforce, 2023)

Remember the "576 tokens is a lot" war story? Li et al. (2023, arXiv:2301.12597) introduced the Q-Former — a small transformer with 32 learned query vectors that cross-attend to the ViT patch tokens and compress them down to 32 output vectors. You lose some spatial fidelity but you get an 18× shorter visual prefix. Every downstream VLM that has to fit many images into a context window (video understanding especially) uses some variant of this.

6.3 · LLaVA-NeXT → LLaVA-OneVision (2024)

LLaVA-NeXT (Jan 2024) fixed high-resolution by AnyRes tiling: split a big image into 336×336 sub-tiles, encode each, concatenate. A 672×672 image becomes 4 tiles × 576 tokens + a low-res global view. Counting, OCR, and chart-QA jumped 20–30 points.

LLaVA-OneVision (Li et al., Aug 2024, arXiv:2408.03326) unified single-image, multi-image, and video into one model with the same architecture, using different token-budget schedules per modality. It's the reference open-source baseline for 2025 VLM research — start there before writing your own.

6.4 · Molmo and PixMo (Ai2, Sep 2024)

Deitke et al. (arXiv:2409.17146) showed you don't need billions of GPT-4-generated instruction samples. Their PixMo dataset (~1M highly curated, human-annotated image-caption + pointing samples) trained a family of Molmo models where Molmo-72B beat GPT-4V and Gemini 1.5 Pro on 11 academic benchmarks. Key trick: audio-transcribed dense captions (annotators spoke long descriptions instead of typing short ones) — much richer supervision per sample. If you're building your own VLM dataset in 2025, read PixMo before you write a labeling doc.

6.5 · Qwen2.5-VL and native dynamic resolution (Alibaba, Jan 2025)

Qwen2.5-VL (Bai et al., arXiv:2502.13923) drops the tile-and-stitch hack. It uses M-RoPE (multimodal rotary position encoding) with separate temporal, height, and width axes, and a ViT that accepts arbitrary aspect ratios natively. It also encodes absolute time for video (not just frame index), so "what happened around 3:47?" works. Qwen2.5-VL-72B is currently (mid-2025) the strongest open-weights VLM on OCR and document understanding.

6.6 · Further reading (URLs)

• CLIP: https://arxiv.org/abs/2103.00020 · Radford et al. 2021
• LLaVA-1.5: https://arxiv.org/abs/2310.03744 · Liu et al. 2023
• SigLIP: https://arxiv.org/abs/2303.15343 · Zhai et al. 2023
• SigLIP-2: https://arxiv.org/abs/2502.14786 · Tschannen et al. 2025
• BLIP-2 (Q-Former): https://arxiv.org/abs/2301.12597 · Li et al. 2023
• LLaVA-OneVision: https://arxiv.org/abs/2408.03326 · Li et al. 2024
• Molmo / PixMo: https://arxiv.org/abs/2409.17146 · Deitke et al. 2024 · https://molmo.allenai.org/
• Qwen2.5-VL: https://arxiv.org/abs/2502.13923 · Bai et al. 2025
• HF VLM leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard

7 · Try it yourself

The cheapest path from "read this session" to "trained a VLM once":

1. pip install transformers accelerate datasets
2. Load a SigLIP-2 vision tower and a TinyLlama-1.1B LLM.
3. Wire up the 2-layer MLP projector (60 lines).
4. Take 10k image-caption pairs from LAION-COCO (free, on HuggingFace).
5. Train the projector for 1 epoch on a single T4 (Colab free tier will do). About 90 minutes.
6. Prompt it with 5 personal photos. Does it caption them coherently?

Expected result: broad-strokes captions correct ("a dog on grass"), fine details wrong (breed, count, colors). That's exactly what LLaVA-1.0 saw in 2023 — you've just reproduced the state-of-the-art from two years ago in an afternoon. Now you understand what LLaVA-NeXT + PixMo needed to fix.

8 · Recap

CLIP taught two encoders to agree on a shared vector space using a symmetric batch-contrastive loss — a free O(N²) supervision signal from O(N) data. LLaVA then noticed that if the vision encoder already speaks a language, all you need is one small MLP to translate it into the LLM's embedding space. Everything since — SigLIP-2, Q-Former, AnyRes, native-resolution ViTs, PixMo — is the same architecture with better data, better tokens-per-image budgets, or better position encodings. The next frontier (video, tools-in-VLM, action tokens) is the same trick again: whatever you can encode into a vector that lives near the LLM's embedding space, the LLM can reason about.

🧠 Retention scaffold

One-line summary (write it in your own words): _______________________________

Spaced review: re-read §1 (CLIP loss) + §3 (LLaVA projector) in 24 hours. Revisit the full session on day 7 alongside the SigLIP-2 paper.

Next session (S075): we teach the same LLM to act — call a Python interpreter, hit a search API, plan multi-step tasks. Same trick as LLaVA in spirit: tools just produce tokens.

Sticky note (keep on your desk): A multimodal model is a text LLM with one extra nn.Linear glued to whatever encoder you can find. Everything else is data quality and token budget.

Previous: ← DL S073 · Next: DL S075 → Agents & Tool Use