Multimodal LLMs — CLIP, VLMs, Video, Audio
A deep-dive on Multimodal LLMs — part of a 36-topic evergreen learning series.
Why this session matters
Part of a 36-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.
Session 35 · LLM track 🎨
2026 is the multimodal year. This session covers CLIP (contrastive image-text), VLMs (LLaVA, GPT-4V, Gemini), video understanding (frame sampling, temporal reasoning), audio (Whisper, speech LLMs), and the emerging any-to-any models. Includes practical patterns for RAG-over-images, video summarization, and multi-modal agents.
Pre-read (30 min before session)
Watch 1–2 of these before the deep-dive:
- CLIP Explained — Yannic Kilcher
- Multimodal LLMs — AI Coffee Break
- Video Understanding with LLMs — DeepLearningAI
Then skim the CLIP paper.
Deep-dive (90 min)
1. CLIP + contrastive multimodal (25 min)
CLIP: image encoder + text encoder, contrastive loss on 400M image-text pairs. Zero-shot classification via cosine similarity to text embeddings of class names. SigLIP, ALIGN, EVA-CLIP as evolutions. Use cases: image search, deduplication, moderation.
2. VLMs (25 min)
Architecture: vision encoder → projection → LLM (LLaVA style). Or unified transformers (Gemini). Training: pretrain (image-caption) → instruction tune (visual QA). Popular open VLMs: LLaVA-1.6, Qwen-VL, InternVL, Idefics3. Closed: GPT-4V/o, Claude Sonnet, Gemini. Evaluation: MMBench, MMMU, DocVQA.
3. Video & audio (20 min)
Video: frame sampling strategies (uniform, keyframe, adaptive). Temporal reasoning (Video-LLaVA, VideoChat). Long-form video (needle-in-video). Audio: Whisper for ASR. AudioLM / speech LLMs (Moshi, GPT-4o voice). TTS: Bark, StyleTTS, ElevenLabs.
4. Production patterns (20 min)
Multimodal RAG (index images + captions + OCR text). Document AI (layout + text + tables). Any-to-any agents (image in, code out, or voice in, action out). Cost/latency: vision tokens are expensive; batch requests, use smaller VLMs for triage.
Reading list
- CLIP paper (Radford et al 2021)
- Flamingo, LLaVA, Kosmos papers for VLM evolution
- Lilian Weng's blog: Multimodal AI
Research links
Hands-on drill
Use LLaVA (via ollama pull llava or HuggingFace) to describe 5 images. Then chain: image → LLaVA description → GPT-4 to answer questions about the image. Compare quality vs GPT-4V direct.
Post-session checklist
- Can you explain how CLIP was trained in 60 seconds?
- Can you explain the LLaVA architecture (vision encoder → projection → LLM) in 60 seconds?
- Can you explain why video is expensive and how frame sampling helps in 60 seconds?
- Did you complete the hands-on drill above?
- Did you write 3 flashcards for tomorrow's recall?
- What's the one thing you'd want to revisit in the next revision session?
What's next
Session 36 continues the series. See the hub page for the full sequence and revision pattern.