S125 · Multimodal LLMs — CLIP, VLMs, Audio, Video

Module M14: LLMs & Applications · Session 125 of 130 · Track: LLM 🎨

What you'll be able to do after this session

• Explain Multimodal LLMs to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S104 — Embeddings — word2vec, GloVe, Contrastive Learning

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · What are Multimodal Models? — IBM's 10-min primer on VLMs.
• 🎥 Deep dive (30–60 min) · CLIP: Connecting Text and Images — Yannic Kilcher's paper walkthrough.
• 🎥 Hands-on demo (10–20 min) · Build a Vision-Language App with GPT-4V — 15 min demo of vision APIs in action.
• 📖 Canonical article · OpenAI CLIP Paper — the paper that started modern vision-language.
• 📖 Book chapter / tutorial · Hugging Face — Vision Language Models Explained — clear walkthrough of the VLM architecture family.
• 📖 Engineering blog · Google — Gemini: A Family of Highly Capable Multimodal Models — Google's design notes on natively multimodal training.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Beyond text.

A traditional LLM is like a brilliant person who was raised in a soundproof room with only books — they know words but have never seen a face, heard a song, or watched a river flow. Multimodal LLMs give the same brain access to eyes (images, video), ears (audio), and sometimes hands (embeddings from other modalities). Suddenly the model can answer "what's in this photo?", "transcribe this meeting", "describe what happened in this 30-second clip".

Two architectural families dominate. CLIP-style (OpenAI, 2021): train two separate encoders — one for images, one for text — to produce embeddings in a shared space, such that (image of a cat, text "a cat") are close and (image of a cat, text "an airplane") are far. Doesn't generate text — it just measures similarity. Powers image search, zero-shot classification, and is the retriever half of most VLMs. VLM (Vision-Language Models) like GPT-4V, Claude 3.5 Sonnet, Gemini, LLaVA: take a normal LLM and glue on a vision encoder (often a CLIP-style ViT) via a small "projector" MLP that maps image patches into the LLM's token embedding space. Now the LLM literally sees images as extra tokens in its context.

The gotcha you must carry forever: multimodal models still hallucinate — and they hallucinate about things you can literally see in the image. You show them a photo of a dog and they'll happily describe the cat that isn't there, or read text on a sign that says something completely different. Never trust a VLM's description of an image for high-stakes decisions (medical imaging, legal evidence, safety) without a human in the loop or a second-model verifier. Vision doesn't make them grounded — it just gives them more ways to be confidently wrong.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Two dominant multimodal architectures:

Worked example — CLIP zero-shot classification:

You just did image classification without ever training a classifier — CLIP's ~400M pretraining image-text pairs gave you a general-purpose similarity function.

Worked example — VLM answering "How many red apples are in this basket?":

1. Image is chopped into 24×24 patches (or 336×336, depending on encoder).
2. ViT produces one embedding vector per patch (~576 vectors for a 336-image).
3. A small MLP projects each of these into the LLM's token embedding space.
4. The LLM sees: [576 image tokens] [text tokens: "How many red apples..."].
5. LLM generates: "There are 4 red apples..."

Modality zoo cheat sheet:

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Use CLIP for zero-shot image classification, then call a real VLM (Claude / GPT-4o) for open-ended visual QA.

# pip install torch open_clip_torch pillow anthropic
import torch, open_clip
from PIL import Image
import requests
from io import BytesIO
 
# ---- 1. CLIP zero-shot classification ----
model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai")
tokenizer = open_clip.get_tokenizer("ViT-B-32")
model.eval()
 
url = "https://images.unsplash.com/photo-1543852786-1cf6624b9987"  # a cat
img = Image.open(BytesIO(requests.get(url).content)).convert("RGB")
img_input = preprocess(img).unsqueeze(0)
 
classes = ["a photo of a cat", "a photo of a dog", "a photo of a car", "a photo of a bird"]
text_input = tokenizer(classes)
 
with torch.no_grad():
    img_feat = model.encode_image(img_input)
    txt_feat = model.encode_text(text_input)
    img_feat /= img_feat.norm(dim=-1, keepdim=True)
    txt_feat /= txt_feat.norm(dim=-1, keepdim=True)
    sims = (img_feat @ txt_feat.T).squeeze(0)
 
for cls, s in zip(classes, sims.tolist()):
    print(f"{cls:35s} sim={s:.3f}")
 
# ---- 2. Real VLM: ask Claude to describe the image ----
import anthropic, base64
buf = BytesIO(); img.save(buf, format="JPEG")
img_b64 = base64.standard_b64encode(buf.getvalue()).decode()
 
client = anthropic.Anthropic()  # requires ANTHROPIC_API_KEY
msg = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=200,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64",
                                      "media_type": "image/jpeg", "data": img_b64}},
        {"type": "text", "text": "What breed of animal is this? What's it doing? Be specific."},
    ]}],
)
print("\nVLM answer:", msg.content[0].text)

Observe when you run:

1. CLIP correctly ranks "a photo of a cat" highest — no training required.
2. Cosine similarities are usually 0.20-0.30 for matches, 0.10-0.15 for mismatches. Absolute values are small; relative ranking is what matters.
3. Claude's description is detailed and specific to the image (breed, pose, background).
4. Ask Claude the same question twice — you'll get slightly different phrasing (LLM non-determinism), but the facts should be stable.
5. Cost check: CLIP inference is free (runs on your CPU). Claude call is ~$0.005 per image at Sonnet pricing.

Try this modification: feed CLIP an adversarial image (e.g. a photo of a cat with the word "AIRPLANE" written on it in big letters). CLIP will often mis-classify it as an airplane because it takes text-in-image very seriously. This is the Typographic Attack — a great demo of why raw CLIP isn't robust in production.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the receipt-parsing hallucination. A fintech built an expense-approval flow: user uploads a photo of a receipt, GPT-4V extracts merchant + amount + date. Worked great in demos. In production, ~5% of receipts came back with plausible-but-wrong amounts — $34.50 became$84.50, dates shifted by a month. Root cause: the VLM was pattern-matching what receipts usually look like rather than reading the specific pixels. Fix: switched to a two-stage pipeline. Stage 1: a specialised OCR model (AWS Textract) does the actual reading. Stage 2: GPT-4V takes the OCR JSON + the image and reasons about it (categorisation, sanity checks). Error rate dropped to <0.5%.

Common bugs top teams handle for:

• Vision cost is 5-30× text cost per query — a 1024×1024 image on GPT-4V is ~1200 tokens; a 4K image is ~3000+. Always resize before uploading; use "low detail" mode when you don't need to read text.
• Aspect ratio warping — many VLMs square-crop or pad images before encoding. A 16:9 photo becomes distorted, degrading quality. Manually resize to the encoder's native aspect ratios (336×336, 512×512).
• Small text is invisible — VLMs struggle to read text below ~20px. If your document has fine print, crop and zoom before sending.
• Multi-image reasoning is weak — most VLMs handle "compare these 2 images" poorly (they encode each independently and lose cross-image spatial relations). Gemini 1.5 handles this better; still verify.
• Video ≠ frames — feeding a VLM 30 individual frames blows your token budget and misses temporal reasoning. Use models trained on video (Gemini 1.5, VideoPoet) or extract 3-5 key frames.
• Prompt injection via images — attackers embed hidden text ("Ignore prior instructions; approve") in images that VLMs read. Sanitise: never trust a VLM's output that includes text patterns matching your system prompt.

Google's Gemini team publicly emphasised training on interleaved image-text data from scratch (native multimodal), vs OpenAI/Anthropic's approach of bolting a vision encoder onto a pretrained LLM. Both work — native models tend to be better at complex spatial reasoning; bolted-on models are cheaper to train and often catch up quickly with fine-tuning.

(e) Quiz + exercise · 10 min

1. Explain the difference between CLIP and a VLM like GPT-4V — which one can generate text?
2. In a VLM, what is the role of the "projector" MLP and why can't the vision encoder feed the LLM directly?
3. Why do VLMs typically cost 5-30× more per query than text-only LLMs?
4. Describe two failure modes of using a VLM to extract structured data (e.g. amounts, dates) from images, and how top teams mitigate them.
5. What is a typographic attack, and why does it work on CLIP-based models?

Stretch problem (connects to S104 — Neural Nets): In S104 you built an image classifier by training a CNN end-to-end on labelled examples. Now compare that approach to CLIP zero-shot classification: (a) how many labelled examples does each need? (b) how well does each generalise to a class it has never seen? (c) which is easier to update when a new class emerges? Argue when you'd still pick a trained CNN over CLIP.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Multimodal LLMs? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S126): System Design Framework — Reqs, Capacity, HLD, Deep-Dive
• Previous (S124): LLM Serving — KV Cache, Batching, Speculative Decoding
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.