Multimodal LLMs — CLIP, VLMs, Audio, Video

Why this session exists

Adding images or audio supplies observations, not guaranteed grounding. A scholar given eyes may answer new questions while still guessing incorrectly. Not every production system needs every modality. You will distinguish contrastive matching from generation, derive CLIP's objective, trace a visual projector, estimate resolution/token tradeoffs, compare OCR and VLM pipelines, and defend against image-borne instructions.

Embeddings (104), CNNs (102), Transformers (112) and retrieval (117) are prerequisites. Image search is one application of CLIP-like encoders, not evidence that CLIP powers every image RAG or VLM.

CLIP: shared geometry, not a text generator

CLIP's 2021 paper trains separate image and text encoders on 400 million internet image–text pairs. “Zero-shot” means no labeled examples for the downstream classifier adaptation, not no training supervision. Encode an image and candidate descriptions such as “a photo of a cat,” normalize each nonzero vector, compute similarities and select the highest-scoring label. The label set can change at inference, but describing an unseen class does not guarantee recognition of a concept absent from pretraining.

For normalized image vectors I and text vectors T in an N-pair batch, logits are Z=exp(t)*I*T^T, where t is a learned log scale. With one matched pair per index, the symmetric loss is

A row predicts the matching caption among N texts; a column predicts the matching image among N images. This aligns representations without requiring paired vectors to be numerically identical. Shortcut cues can still minimize loss; contrastive learning does not prove that all discriminative features are semantically correct. Duplicate images or multiple valid captions violate a single-diagonal-positive assumption and need a multipositive/masking treatment.

Cosine is meaningful for any nonzero vectors with division by their norms. Normalization merely makes dot product equal cosine; the original “cosine only meaningful for unit vectors” claim was wrong. CLIP's learned scale changes softmax confidence but not argmax for a positive scale. Softmax over supplied labels is not a calibrated probability that the object exists, and an unknown image is still assigned one of those labels unless you implement/validate rejection. The original demo omitted the learned scale, so its percentages were not the model's standard logits-to-softmax output.

VLM bridges, patch arithmetic and architecture limits

A LLaVA-style model converts vision features into representations usable by a language model. If vision width is dv and language width dl, a linear projector is a dl-by-dv map; an MLP can add nonlinear capacity. Matching dimensions is necessary for that interface but not sufficient for semantic alignment—training supplies it. Other VLMs use resamplers, cross-attention, pooling or joint encoders. Do not assert that every proprietary model is a frozen LLM plus a projector MLP. The initial LLaVA paper connects a vision encoder and LLM with visual instruction tuning; later variants change the bridge.

For a square 336-pixel image with 14-pixel patches, (336/14)^2=576 patches, a 24-by-24 grid. If patch width were 24 pixels, the grid would be 14-by-14=196. The original conflated those meanings. Class tokens, feature selection, pooling, tiling and resampling can change what reaches the decoder. Projected visual vectors are not text vocabulary IDs, and modality/position information can distinguish them; the Transformer does not magically forget all modality structure.

Higher resolution may preserve small text but increases encoder work and often visual-token/KV cost. Detail smaller than one patch is not automatically absent: the patch projection can encode within-patch information, while resizing/compression and later pooling can destroy detail. Cropping can help or remove needed context. Determine the effective resolution floor experimentally with random digits of decreasing pixel height, fixed crops, controlled resampling and exact transcription accuracy, not a universal 20px rule.

Historically, Show and Tell (2014 preprint) studied image captioning, ViT (2020 preprint) patch Transformers, CLIP (2021) image/text alignment, Flamingo (2022) visual conditioning with cross-attention, and LLaVA (2023) instruction tuning. They are illustrative milestones, not proofs that one was the first possible architecture or that every later VLM copied it. “Native multimodal” describes training/model design, not necessarily training every component from scratch or guaranteed superiority over adapters; undisclosed GPT-4V internals are not inferred here.

Audio, video, documents and cost accounting

Audio systems may transcribe (Whisper), learn speech representations (HuBERT), translate speech (SeamlessM4T) or generate speech in multimodal systems. A transcript loses tone, overlap, timing and other acoustic evidence; retain timestamps and speaker uncertainty for questions that depend on them. Video involves temporal order and often audio. Sampled frames can miss a short event; independent encoding does not prevent a later model from comparing ordered frames, but timestamps and a suitable temporal model matter. Thirty frames of a minute is not automatically good or bad: sample for the task and test missed-event recall. A transcription-plus-keyframe approach is a cost/detail tradeoff, not a universal 10× saving.

Document models such as LayoutLM-style systems use layout information; OCR-free generative document models and specialist OCR solve different pieces. Video generation systems, action representations and document extraction are not interchangeable just because they process nontext data.

Provider billing is model/version specific and may count patches, tiles or other units. The historical GPT-4o-style high-detail scheme uses resize rules plus a base and per-512-tile cost, not raw pixel area alone; it must not be generalized to every GPT/Claude/Gemini model. For a declared illustrative base85 + 170-per-tile schedule with four tiles, cost is 765 units. For a purely illustrative area/750 schedule, 1024² pixels would give about 1398 units before resizing; this does not validate the original Claude-specific 1200 estimate. These are conditional arithmetic examples, not current billing quotes or API guarantees. The OpenAI image guide snapshot documents model-specific tile/base values and resize steps; inspect the exact model's current documentation and observed usage before budgeting. Resizing can save cost and destroy OCR/counting quality; measure both.

Hands-on: contrastive objective and bridge on CPU

This executable program uses synthetic vectors and patches. It tests CLIP-style normalization, learned-scale arithmetic, symmetric cross entropy, wrong-pair/zero-vector controls and projector shapes. It is not pretrained CLIP classification, image recognition, VLM training or a vision API result.

import math
import torch
import torch.nn.functional as F
 
torch.manual_seed(125)
torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)
 
def unit(x):
    norms = x.norm(dim=-1, keepdim=True)
    if (norms == 0).any():
        raise ValueError("zero embedding")
    return x/norms
 
def clip_loss(images, texts, scale):
    z = scale*unit(images)@unit(texts).T
    labels = torch.arange(len(images))
    return (F.cross_entropy(z, labels)+F.cross_entropy(z.T, labels))/2
 
images = torch.eye(3, requires_grad=True)
texts = torch.eye(3)
scale = 4.
loss = clip_loss(images, texts, scale)
# Independent closed form: diagonal scale, two zero-logit negatives.
expected = math.log(math.exp(scale)+2)-scale
assert math.isclose(loss.item(), expected, rel_tol=1e-12)
loss.backward(); assert torch.isfinite(images.grad).all()
wrong = clip_loss(images.detach(), texts[[1,2,0]], scale)
assert wrong > loss
z = unit(torch.tensor([[2.,0.,0.]]))@unit(texts).T
assert z.argmax(-1).item() == (10*z).argmax(-1).item()
assert not torch.allclose(z.softmax(-1), (10*z).softmax(-1))
try:
    unit(torch.zeros(1,3))
except ValueError:
    pass
else:
    raise AssertionError("zero vectors rejected")
assert (336//14)**2 == 576 and (336//24)**2 == 196
patches = torch.randn(2,576,8)
projector = torch.nn.Linear(8,12)
visual = projector(patches)
text_tokens = torch.randn(2,5,12)
combined = torch.cat([visual,text_tokens],dim=1)
assert combined.shape == (2,581,12)
# Two identical pairs are both valid: diagonal-only CE has irreducible log(2).
duplicates = torch.tensor([[1.,0.],[1.,0.]])
assert math.isclose(clip_loss(duplicates,duplicates,4).item(),math.log(2))
assert 85+170*4 == 765
assert math.isclose(1024*1024/750,1398.1013333333333)
print("contrastive/bridge/patch/cost contracts pass; no vision model evaluated")

Restored CLIP and VLM application recipe

The original OpenCLIP/Anthropic code was not unsafe dynamic execution merely because it called model.eval(). Its unexecuted model-loading/API boundary must remain explicit. A complete authorized real-image experiment proceeds as follows:

1. Use a permitted local image and local pretrained checkpoint with recorded hashes. Pin OpenCLIP, tokenizer and preprocessing configuration. Apply the matching resize/crop/normalization and model.eval() with inference/no-grad mode; mismatched preprocessing invalidates comparison.
2. Encode the image and supplied class prompts; normalize nonzero features; apply model.logit_scale.exp() as documented by the matching architecture before any softmax. Include the attacked label (for example airplane) in the candidate set if testing whether it wins. Record raw similarities, labels and preprocessing; no universal 0.2–0.3 matching range exists.
3. For a separately authorized VLM call, resize/crop according to the task, encode actual image bytes to base64 with the correct MIME type, and send image and text blocks. The Anthropic Python 0.69.0 image-source schema supports a base64 source with a specified image media type. Use an available model ID from the current account, not an assumed still-live Claude 3.5 alias.
4. Parse all relevant returned content blocks, handle refusals/truncation and log input/output usage without exposing image or credential data. Do not assume content[0] is always text. Ask for observations and uncertainty rather than unsupported breed identification from an ambiguous photo.
5. Compare original, resized, cropped and text-overlaid images with a ground-truth rubric. Repeat tests to estimate variability. Do not claim CLIP picks cat or the VLM remains factual until it actually does on those bytes.

No image/weights were downloaded or external model called in this audit. The procedure retains both zero-shot classification and open-ended visual QA as meaningful exercises, not retired topics.

For the original red-apples question, trace pixels through patch features and the bridge into decoder attention, then require a count tied to visible instances. An answer of four is only an illustrative answer until a labeled image establishes four. Test overlap/occlusion, red-versus-green distractors, multiple-image identity, rotations and crops separately. Keep the image order and spatial references explicit; compare full image and crops because a crop can improve text resolution while removing an object needed for counting.

OCR, hallucination and untrusted image text

For stable high-volume forms, OCR plus rules can be inexpensive and auditable with coordinates, but layout variation and OCR errors need handling. VLM end-to-end handles flexible language/layout tasks but can invent plausible missing fields or misread digits. OCR plus a text LLM preserves textual provenance but can lose spatial structure unless coordinates/layout are retained. Neither OCR nor a VLM universally wins character accuracy; compare on your actual document distribution.

For receipts, require amount/date/currency schema checks, source coordinates/crops, arithmetic reconciliation (subtotal+tax-discounts=total under the document's rounding rules) and human review of ambiguity/disagreement. Agreement between two models is not proof, and disagreement is not the only possible confidence signal. Never silently fill an unreadable amount from what receipts “usually” contain. The original fintech error-rate improvements were unsupported; this is a design scenario, not a reported deployment.

Typographic attacks exploit learned associations between visible text and concepts; the CLIP authors' multimodal-neuron work documents text altering classifications. A general VLM is not guaranteed immune. Image-borne prompt injection is a related but distinct failure: text in the image attempts to become an instruction. Treat image/OCR/transcript content as untrusted data, separate observation from authorized action, validate downstream operations in deterministic policy code, minimize tool permissions and require human approval for consequential actions where appropriate. A system prompt or second text-only model is an additional check, not an authorization boundary.

Six production risks to test are small-detail loss, hallucinated fields, orientation/crop/layout mistakes, typographic/instruction attacks, audio/video sampling loss and cost/privacy blowups. Rate limits, upload decoding limits, permitted formats, provenance, abstention and task-specific golden sets address different risks; “please only claim what you see” alone addresses none reliably.

Recall and stretch: complete answers

1. CLIP versus VLM: standard CLIP ranks aligned image/text embeddings and has no language decoder; a generative VLM conditions text output on visual information through a model-specific interface.
2. Projector: learns a dimension/representation bridge; equal widths alone would not ensure alignment, and other architectures use cross-attention/resamplers.
3. Cost: image representations can add many processed units, but no universal 5–30× multiplier exists. Use the exact model/resolution/accounting contract and usage.
4. Extraction failures: misread small/ambiguous digits and invented missing fields; test OCR/VLM baselines, retain source locations and validate/abstain rather than assuming either is perfect.
5. Typographic attack: visible words can shift learned image/text associations. Include the target label and original/control images; do not promise a specific misclassification or VLM immunity.
6. Thirty frames? Sampling may miss motion/events and omit audio; timestamps, temporal processing and task-aware selection matter. Frame order can still be modeled.
7. Native versus adapter: joint multimodal training and reuse of pretrained components offer different data/compute/interface tradeoffs; no universal quality ordering or private architecture claim.
8. Injection defenses: untrusted image text never grants authority; deterministic permissions, bounded tools, validation and human oversight where needed. Another model's concurrence is not sufficient.
9. Normalization: divide by nonzero norms to make dot product cosine; preserve the learned logit scale for the intended CLIP scoring contract.
10. CNN stretch: zero-shot CLIP uses no downstream fitting labels but extensive pretraining; a supervised classifier needs task data with no fixed per-class count. Fixed heads lack arbitrary new labels; domain-trained or adapted models may win specialized closed-set tasks, so benchmark both.

Explain the contrastive loss, compute 576 patches, and design a receipt-validation failure case. Multimodal retrieval, evaluation, adapters and serving reuse earlier ideas while adding modality-specific evidence and safety requirements.

Optional resources and onward navigation

The original optional pre-read links are retained as navigation, not fact-check evidence or promises about video length, quality or API stability. Use the versioned primary sources in the teaching for consequential claims. Original pacing was a suggested 90-minute session, not a completion guarantee.

• What are Multimodal Models?
• CLIP · Connecting Text and Images
• Build a Vision-Language App with GPT-4V / Claude
• Learning Transferable Visual Models From Natural Language Supervision (CLIP)
• Vision Language Models Explained
• Gemini · A Family of Highly Capable Multimodal Models

Next in the original sequence: S126 · System Design Framework — Reqs, Capacity, HLD, Deep-Dive. Publication and release navigation are adjudicated by the parent.

Hub: The 6-Month Learning Plan. The original 130-session scaffold is retained as navigation, not evidence that every destination has been reviewed.

Review and execution boundary

Reviewed against complete original and pre-edit current teaching with claim-specific primary evidence and independently replayed coverage. The executable CPU program is a bounded teaching test, not a pretrained model, serving, RAG or training benchmark. External model/service recipes remain source-supported and unexecuted. No weights, datasets, GPU, paid API or new service were used. Source publication is not end-to-end application acceptance.