Multimodal — CLIP → LLaVA

From matching pictures to generating grounded text

A text decoder consumes vectors, but feeding it arbitrary image vectors does not confer vision. Three interfaces must agree: the vision encoder must preserve relevant information, the projector must translate useful features into the decoder's input width, and training must teach the decoder to use them. CLIP is a dual-encoder matching model; LLaVA adds a generative language model and visual instruction tuning. Neither describes every proprietary multimodal architecture. GPT-4V and Gemini internals cannot be inferred from a LLaVA diagram.

Prerequisites are tensor shapes, softmax/cross-entropy, ViT patches, causal attention and supervised fine-tuning. This lesson develops the loss, a trainable CPU example, a complete feature-to-caption training procedure and evaluation decisions. The CPU experiment uses generated arrays, not images or a downloaded model. The pretrained/GPU procedure is an unexecuted reference.

The shared-room analogy is useful only up to a point: image and text encoders learn compatible directions, not identical coordinates for every paired sample. A CLIP embedding is a discriminative summary rather than a complete image description. Counting, negation and spatial relationships are empirical weaknesses to test, not properties mathematically forbidden by contrastive learning.

Symmetric contrastive loss from first principles

Let projected image and text matrices have shape (N,d). Normalize each row with a declared epsilon; a zero vector does not become a meaningful unit vector. Let a be a learned log inverse temperature and s=exp(a). Logits are Z=s I T.T. For one matched image-caption pair per row, the target index is its row number.

The first direction asks which caption matches an image; the second asks which image matches a caption. They share logits but have different normalization denominators. Both directions already differentiate through both encoders. Dropping one does not leave the other encoder untrained, and no theorem says its representation must collapse. The original's asserted Flickr30k 8–10-point loss had no supporting experiment and is not retained as a result.

For the row loss the logit derivative is (softmax(Z,row)-identity)/N; the column contribution uses column softmax. Their half-sum is the symmetric derivative. Normalize embeddings to separate direction from norm and learn scale separately. Unnormalized encoders can exploit growing norms on separable pairs, but low training loss alone neither proves this happened nor determines transfer accuracy. Monitor norms, scale, loss and held-out retrieval.

There are N positive cells and N²−N nominal negatives. False negatives arise with duplicated images, paraphrases or multiple correct captions. Larger batches supply more comparisons, not N² independent labelled examples. Use multi-positive labels or remove unintended duplicates when appropriate. Near-duplicate captions can create conflicting supervision; they do not invariably produce zero gradient.

A normalized numeric example

The old three-pair example called [.7,.7] and [.9,.1] normalized although neither has unit norm. With identity image and text vectors and inverse temperature 2, logits are [[2,0],[0,2]]; each directional CE is log(1+exp(-2)), approximately 0.126928. Swapping one side's pair order makes the diagonal wrong and raises the loss by 2.

import numpy as np
I = np.eye(2)
T = np.eye(2)
logits = 2 * I @ T.T
 
def diagonal_ce(z):
    shifted = z-z.max(axis=1, keepdims=True)
    return np.mean(np.log(np.exp(shifted).sum(axis=1))-np.diag(shifted))
 
loss = .5*(diagonal_ce(logits)+diagonal_ce(logits.T))
np.testing.assert_allclose(loss, np.logaddexp(0., -2.))
wrong = .5*(diagonal_ce(logits[:, ::-1])+diagonal_ce(logits[:, ::-1].T))
assert wrong > loss
print('symmetric contrastive loss', loss, 'swapped-pair loss', wrong)

If both captions have the same embedding, the image-to-text candidates are indistinguishable: their probabilities are one half regardless of scale, so row CE is log 2. Raising scale cannot manufacture missing information. Conversely, making scale enormous on an incorrectly ranked pair increases loss and gradients rather than magically improving confidence calibration.

A complete small dual-encoder experiment

The following CPU program implements encoders, projections, stable symmetric loss, a learned bounded inverse temperature, a training loop and shape/gradient checks. The data have a known shared latent cause. It demonstrates optimization of the objective, not CLIP-scale visual learning or zero-shot ImageNet performance. The reference uses PyTorch 2.9 CPU.

import math
import torch
from torch import nn
from torch.nn import functional as F
 
torch.manual_seed(74)
torch.set_num_threads(1)
class DualEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.image = nn.Sequential(nn.Linear(6, 12), nn.Tanh(), nn.Linear(12, 8))
        self.text = nn.Sequential(nn.Linear(5, 12), nn.Tanh(), nn.Linear(12, 8))
        self.log_scale = nn.Parameter(torch.tensor(1.0))
    def forward(self, images, texts):
        i = F.normalize(self.image(images), dim=-1, eps=1e-12)
        t = F.normalize(self.text(texts), dim=-1, eps=1e-12)
        z = self.log_scale.clamp(max=math.log(100)).exp() * (i @ t.T)
        labels = torch.arange(len(i))
        return (F.cross_entropy(z, labels)+F.cross_entropy(z.T, labels))/2, z
 
latent = torch.randn(16, 4)
images = latent @ torch.randn(4, 6)
texts = latent @ torch.randn(4, 5)
model = DualEncoder()
opt = torch.optim.Adam(model.parameters(), lr=.02)
initial = model(images, texts)[0].item()
for step in range(120):
    opt.zero_grad()
    objective, z = model(images, texts)
    objective.backward()
    assert all(p.grad is not None and torch.isfinite(p.grad).all() for p in model.parameters())
    opt.step()
    with torch.no_grad(): model.log_scale.clamp_(max=math.log(100))
final, z = model(images, texts)
assert z.shape == (16, 16) and final.item() < initial * .25
assert (z.argmax(1) == torch.arange(16)).float().mean() >= .9
# Dropping column loss still trains BOTH towers.
opt.zero_grad()
F.cross_entropy(model(images, texts)[1], torch.arange(16)).backward()
assert model.image[0].weight.grad.norm() > 0
assert model.text[0].weight.grad.norm() > 0
print('synthetic contrastive loss', initial, final.item())

eps=1e-12 defines a numerical denominator floor, not a cure for collapsed features: a zero row stays zero and a row smaller than epsilon is not unit length. Track invalid/near-zero rows in real training. Clamp the log scale before exponentiating; exp(1000).clamp(max=100) looks bounded in the forward pass but can produce a NaN gradient through overflow. Projecting the parameter after an update keeps it from remaining far beyond a hard-clamp boundary. The scale limit is a stability choice, not calibrated confidence. The high retrieval score above is on the same 16 pairs used for training; it establishes fit, not generalization.

To study temperature rather than confound it with representation learning, freeze one similarity matrix and compare row entropies for scales 1, 10 and 100. Use both aligned and swapped labels; report loss as well as entropy. To study symmetry, initialize two copies identically, train row-only versus symmetric objectives on the same generated examples, then evaluate both retrieval directions on held-out latents. Do not select the best seed from each condition.

Zero-shot classification and calibrated retrieval

For each candidate class, encode a text prompt such as “a photo of a cat.” Normalize the image and prompt vectors, rank their dot products, and optionally average several normalized template embeddings before renormalizing. Declare how templates are combined. This produces a candidate-relative ranking, not arbitrary caption generation. Softmax over cat/dog must assign its probability mass to those labels even for an image containing neither.

Evaluate balanced class recall, confusion, unknown-object rejection, image quality and prompt-template sensitivity. Fit a rejection rule on validation examples including unknown classes; keep test classes separate. No universal cosine threshold means “same meaning.” For an OCR-heavy form, compare a text-recognition-plus-parser baseline: it can expose exact character confidence and layout errors instead of asking a VLM to infer everything. Use a VLM when relationships between layout, image and text matter and its measured benefit justifies the added ambiguity.

LLaVA interfaces, patch counts and trainable state

For a single unpooled 336×336 image with patch size 14, the grid has 24*24=576 patch tokens, excluding any class token. Original LLaVA and LLaVA-1.5 must not be conflated: the original paper uses a linear projector; the 1.5 paper develops an MLP connector, higher resolution and an expanded instruction mixture. Which hidden vision layer is selected, whether the class token is removed and how special image markers expand are model-specific contracts.

A linear map from 1024 to 4096 features has 1024*4096+4096=4198400 parameters with bias. A 1024→4096→4096 two-layer biased MLP has 20,979,712 parameters. These are interface parameters, not the size or runtime of the whole VLM.

class VisualProjector(nn.Module):
    def __init__(self, vision_width, text_width):
        super().__init__()
        self.layers = nn.Sequential(nn.Linear(vision_width, text_width),
                                    nn.GELU(), nn.Linear(text_width, text_width))
    def forward(self, patches):
        return self.layers(patches)
 
projector = VisualProjector(6, 8)
patches = torch.randn(2, 9, 6)
visual = projector(patches)
assert visual.shape == (2, 9, 8)
visual.square().mean().backward()
assert all(p.grad is not None for p in projector.parameters())
assert 336//14 * (336//14) == 576
assert 1024*4096+4096+4096*4096+4096 == 20979712

Complete feature-alignment and instruction-tuning procedure

This is an unexecuted pretrained/GPU reference. Running it requires appropriate compute, licensed data and model artifacts. Begin with locally available, licensed image-caption records and separately authorized model/tokenizer artifacts. Record checkpoint revisions, image hashes, train/validation/test groups and preprocessing configuration. Split by source image or near-duplicate group before generating multiple questions, so the same image cannot leak across splits.

1. Preflight a single record. Decode an image, apply the vision checkpoint's resize/crop/normalization, and inspect both original and transformed image. Do not blindly stretch aspect ratios. Select documented patch features, discard or retain class tokens deliberately, project to the LM width and insert visual tokens at the specified positions. Verify text/image placeholder count and attention-mask length agree.
2. Build labels. Set labels to −100 for visual inputs, system/user text and padding; retain only the intended assistant caption/answer targets and the model's end token. Shift logits and labels once. Compute summed supervised CE divided by the valid-target count. With a frozen decoder, its operations must still track the input-gradient path to the projector: freezing parameters is not wrapping the decoder forward in no_grad.
3. Stage 1, alignment. Freeze vision and LM weights; optimize the projector on caption data. Use a small overfit batch as a wiring test, then fixed validation captions. Save loss denominators and projector gradients; compare against an image-shuffled control. If shuffling does not matter, the decoder may be exploiting language priors.
4. Stage 2, visual instructions. Continue training the projector and selected LM weights or adapters; keep the vision tower frozen for the baseline. Mix descriptive, short-answer, OCR, spatial, counting and justified-abstention tasks according to the application, not a universal ratio. The original LLaVA reported roughly 595K alignment pairs and 158K instruction examples; LLaVA-1.5 uses a different recipe (558K alignment and 665K instruction examples). They are historical recipe sizes, not required budgets or guarantees.
5. Optimization and recovery. Declare dtype, optimizer, learning rate, batch/accumulation, clipping and updates. Estimate peak activation and KV/weight memory; run a tiny local forward/backward memory check before budgeting the full run. Save config, optimizer, RNG, sampler cursor and artifact hashes. Reload and compare a controlled next step. Never claim a T4 fit or a 90-minute convergence time from parameter count alone.
6. Evaluation and decision. Compare frozen-projector, LM-adapter and optional vision-adaptation variants under the same data and evaluation budget. Test text-only regressions as well as image tasks. Poor OCR is not proof that vision must be unfrozen: image preprocessing, resolution, feature selection, labels and decoder training can each be responsible. Use image perturbations and a linear probe as diagnostics, not a unique causal diagnosis.

A CPU wiring check can isolate the gradient rule without loading an LM. This deliberately tiny frozen causal mixer averages the prefix into each position; it is not a transformer or a visual-language benchmark. It makes earlier visual inputs influence later answer logits, masks non-answer targets, shifts exactly once and divides by supervised targets rather than padded sequence length.

# Depends on the torch/nn/F imports above; no model or data downloads.
torch.manual_seed(740)
connector = nn.Linear(3, 4)
frozen_head = nn.Linear(4, 5)
for parameter in frozen_head.parameters(): parameter.requires_grad_(False)
features = torch.tensor([[[1., 0., -1.], [0., 1., 1.]]])
visual_prefix = connector(features)
text_inputs = torch.zeros(1, 3, 4)
joined = torch.cat([visual_prefix, text_inputs], dim=1)
causal = joined.cumsum(dim=1) / torch.arange(1, 6).view(1, 5, 1)
logits = frozen_head(causal)  # NO no_grad: input-gradient path must survive.
labels = torch.tensor([[-100, -100, -100, 2, 1]])
shifted_targets = labels[:, 1:]
valid = shifted_targets.ne(-100).sum()
assert valid.item() == 2
alignment_loss = F.cross_entropy(logits[:, :-1].reshape(-1, 5),
    shifted_targets.reshape(-1), ignore_index=-100, reduction='sum') / valid
alignment_loss.backward()
assert connector.weight.grad.norm() > 0
assert all(parameter.grad is None for parameter in frozen_head.parameters())
with torch.no_grad(): disconnected = frozen_head(causal)
assert not disconnected.requires_grad  # Negative control: projector cannot learn via this path.
print('supervised targets', valid.item(), 'projector gradient', connector.weight.grad.norm().item())

A real causal decoder supplies the prefix-to-answer attention path; a pointwise head on visual positions alone would not test it. Reject or deliberately skip an all-masked batch rather than dividing by zero. Across variable-length microbatches, sum supervised losses and valid-target counts before forming the token-weighted mean; averaging per-microbatch means gives short examples too much weight. A functioning gradient is only a wiring check: image-shuffled validation and held-out visual tasks test whether the learned connector uses relevant information.

A frozen LM does not change its weights, yet a new visual prefix can still degrade conditional output. Fine-tuning can improve visual use but risks text-only regressions; those are possibilities, not certain outcomes. Unfreezing vision adds memory and data requirements. Domain shift motivates measurement, not automatic full fine-tuning.

Resolution, video and multimodal failure analysis

With eight images in one example, an uncompressed 576-token encoding uses 4608 visual positions. Eight examples each with one image instead use 576 per sequence; batching multiplies total memory but not each sequence's context length. For a 32-layer decoder with 8 KV heads, head width 128 and BF16 cache, each retained token consumes 2*32*8*128*2=131072 bytes before overhead: one 576-token image adds 72 MiB. The vision pass and longer LM prefill are not free.

Any-resolution tiling may combine a global view with crops; four 336-pixel tiles plus a global view have 2880 raw patch tokens before pooling, unpadding and newline conventions. Record actual processor output rather than assuming that arithmetic equals final context consumption. More resolution can help small text but increases cost and may not fix counting. Occlusion, supervision and decoder priors are separate issues. Test count-changing edits, left/right swaps, negation, irrelevant-background changes and text whose answer cannot be inferred from a filename.

For a one-minute video at 2 FPS there are 120 frames. Allocating 32 learned query outputs per frame produces 3840 visual tokens before text/metadata. Add temporal positions or time embeddings and preserve timestamps; simple frame averaging loses event order. Compare sparse frame sampling against a denser reference on events between samples. BLIP-2's Q-Former uses learned queries to extract information from frozen vision features; 32 outputs are an interface choice, not a theorem that spatial detail always survives.

Research extensions without adoption or leaderboard folklore

• SigLIP replaces batch-normalized contrastive softmax with a pairwise sigmoid loss. With labels y=+1 for matches and −1 otherwise, a pair term is softplus(-y*(s*cosine+b)). This avoids a global softmax normalization dependency, but negative selection, pair count and communication still matter; it does not promise excellent training on any small batch.
• SigLIP 2 combines language-image learning with captioning-based pretraining, self-distillation and masked prediction, and develops multilingual/native-aspect-ratio capabilities. It is not evidence that every 2025 VLM adopted that tower. PaliGemma 2's identity cannot be inferred from its release year.
• BLIP-2 bridges frozen vision and language models with a querying transformer. Compare compressed-query versus patch-prefix quality at the same output-token budget. Compression can discard task-relevant information; the useful tradeoff is measured accuracy versus context/memory, not merely 576/32=18.
• LLaVA-NeXT/OneVision study higher-resolution and single-image/multi-image/video transfer. Evaluate whether shared training transfers to held-out video tasks; remove fixed “20–30-point” improvements without the actual task/table.
• Molmo/PixMo emphasize open models/data and human-generated detailed descriptions, including spoken descriptions transcribed for annotation. The lesson is to audit annotation richness, coverage and licensing. It is not a universal claim that a million human examples beats any synthetic corpus or every commercial model.
• Qwen2.5-VL develops dynamic resolution/frame rates, window attention in vision and absolute-time alignment with multimodal positional encoding. Temporal, height and width coordinates serve different roles. A timestamp can be represented without the model reliably understanding every event at that time; test retrieval and ordering independently.

These are source-reviewed mechanisms, not reproduced benchmarks, adoption statistics or private-model disclosures.

Practice and worked answers

1. Write the loss in five operations. Normalize I and T; form exp(a)*I@T.T; construct diagonal labels; compute row and transposed CE; average them. For repeated valid captions, replace the one-positive assumption with an explicitly defined multi-positive objective.
2. Patch arithmetic. 336/14=24 positions per side, 576 patches if no pooling/class token is counted. The projector changes width, not patch count. A tiling processor may change both count and layout.
3. Which weights train? In the baseline alignment stage, only the projector; in visual instruction tuning, projector and LM (or adapters), with frozen vision. Backpropagate through the frozen LM to projected inputs. Variants must name their differences.
4. Why sigmoid rather than softmax? Pairwise terms avoid cross-pair softmax normalization; independent evaluation of terms helps distribution across devices. It does not eliminate all-pairs work when every negative pair is included.
5. When OCR plus a text model? For forms dominated by exact text/fields, compare an OCR/parser pipeline with confidence and layout checks. For charts or spatial relationships, a VLM may offer capabilities the text-only extraction loses. Choose from held-out errors, not fashion.
6. Video Q-Former design. Start with 120 frames×32 queries=3840 tokens, append explicit temporal metadata, reserve text/output budget and test a brief event missed by 2 FPS. Reduce queries only after measuring event recall and temporal order.
7. Two naive-tuning failures. A caption-only mixture may produce verbose answers instead of task-specific outputs; expand supervised task formats and test brief answers. Cropping may erase OCR/count evidence; inspect preprocessing and compare resolution policies. LLaVA-1.5 changed both the connector and data/resolution recipe; do not credit every gain to an unpooled grid or claim counting was solved.

Revisit the loss derivation tomorrow; in a week, explain the gradient path through a frozen decoder and design an image-shuffle control without reading this answer.

Sources and next lesson

Primary papers: CLIP, LLaVA, LLaVA-1.5, SigLIP, SigLIP 2, BLIP-2, OneVision, Molmo, Qwen2.5-VL. Paper results belong to their published configurations. No external vision API, image dataset or pretrained weights were used here.

Previous: latency and throughput · Next: agents and tool use

Optional reading

Explore these supplementary resources after the exercises. Video titles and availability have not been independently verified; use the primary sources above for technical claims.

• CLIP: Connecting Text and Images
• LLaVA — Large Language and Vision Assistant
• CLIP: Learning Transferable Visual Models
• LLaVA project page