Multimodal Models: History and How They Work
A brief history of vision-language models, from CLIP's contrastive dual encoders to PaliGemma's image-prefix decoder, with interactive visualizations.

Want to book this talk?
Why multimodal models
A model that only reads text and a model that only sees images each miss half of what a captioned photo carries. Multimodal models take more than one kind of input, most often images and text, and learn how the two relate. That lets a single model find the image that matches a sentence, caption a photo, or answer a question about a chart.
This talk followed that idea from the first image-text systems to current vision-language models, with two stops in detail: CLIP, which aligns images and text, and PaliGemma, which generates text about images.
A brief history
- 2015, captioning and VQA. Show and Tell paired a CNN image encoder with an LSTM that wrote captions, and the VQA dataset made "answer a question about this image" a benchmark. Each system was trained for one task on hand-labeled data.
- 2019, two-stream transformers. ViLBERT and LXMERT gave images and text their own transformer streams, linked them with cross-attention, and pretrained on captioned images before fine-tuning for each task.
- 2021, contrastive pretraining at web scale. CLIP and ALIGN trained an image encoder and a text encoder on image-text pairs scraped from the web (400 million for CLIP, 1.8 billion for ALIGN), so that matching images and captions land close together in one embedding space.
- 2022, language models learn to see. Flamingo added gated cross-attention layers to a frozen language model. BLIP and CoCa combined contrastive and captioning objectives. PaLI paired a large ViT with an mT5 encoder-decoder and trained it on a mixture of tasks in over 100 languages.
- 2023, cheaper bridges. BLIP-2 connected a frozen image encoder to a frozen language model through a small Q-Former, and SigLIP replaced CLIP's softmax loss with a pairwise sigmoid loss that works at smaller batch sizes.
- 2024, open vision-language models. PaliGemma paired a SigLIP image encoder with the Gemma language model, and Qwen2-VL processed images at their native resolution.
CLIP: aligning images and text
Two encoders, one space
CLIP has two encoders that never see each other's input:
- Image encoder: a ResNet or a Vision Transformer (ViT) that turns an image into a vector.
- Text encoder: a Transformer that turns a caption into a vector.
A learned linear projection maps each output into a shared embedding space, and both vectors are L2-normalized, so the dot product of an image vector and a text vector is their cosine similarity. The walkthrough below follows an image through the ViT-B/32 encoder: splitting it into patches, running the transformer, and projecting the [CLS] output into the shared space.
The contrastive loss
For a batch of N image-caption pairs, CLIP scores all N^2 combinations. Each image is trained to pick out its own caption among the N candidates, and each caption to pick out its own image. With L2-normalized image embeddings I_i and text embeddings T_j, the image-side loss is:
Here \tau is a learned temperature. The text-side loss \mathcal{L}_{\text{text}} swaps the roles and normalizes over images for each caption, and CLIP minimizes the average of the two:
The contrastive loss concept page derives this in more detail.
The shared latent space
Training pulls each image toward its own caption and away from every other caption in the batch. Drag the slider to watch the embeddings sort themselves by concept as training progresses.
Because related images and captions end up close together, CLIP can classify images into categories it was never trained on: embed a prompt such as "a photo of a dog" for each class name and pick the class whose text vector is nearest to the image vector. What CLIP cannot do is write. It scores pairs, but it has no decoder to generate a caption or an answer.
PaliGemma: generating text about images
PaliGemma, released by Google in 2024, is a small open model that captions images, answers questions about them, reads text in them and detects objects, all by generating text. It has three parts:
- SigLIP image encoder (SigLIP-So400m), which turns a 224×224 image into 256 image tokens. At 448 px it produces 1,024 tokens, and at 896 px, 4,096.
- Linear projection, which maps each image token to the embedding size of the language model.
- Gemma-2B decoder, which reads the image tokens followed by a text prompt and generates the answer one token at a time.
The prompt names the task, for example caption en or answer en what is the dog doing. The input sequence is the image tokens, then this prompt (the prefix), then the answer (the suffix). Training is next-token prediction, with the loss computed only on the suffix.
The prefix-LM attention mask
PaliGemma has no separate cross-attention layers. The image tokens sit in the same sequence as the text, so the text reaches the image through the decoder's ordinary self-attention. What sets PaliGemma apart is the mask: the image tokens and the prefix attend to each other in both directions, while the answer is generated causally, one token after another.
| query ↓ key → | img₁ | img₂ | img₃ | img₄ | caption | en | a | dog | runs |
|---|---|---|---|---|---|---|---|---|---|
| img₁ | img₁ attends to img₁ | img₁ attends to img₂ | img₁ attends to img₃ | img₁ attends to img₄ | img₁ attends to caption | img₁ attends to en | img₁ cannot see a | img₁ cannot see dog | img₁ cannot see runs |
| img₂ | img₂ attends to img₁ | img₂ attends to img₂ | img₂ attends to img₃ | img₂ attends to img₄ | img₂ attends to caption | img₂ attends to en | img₂ cannot see a | img₂ cannot see dog | img₂ cannot see runs |
| img₃ | img₃ attends to img₁ | img₃ attends to img₂ | img₃ attends to img₃ | img₃ attends to img₄ | img₃ attends to caption | img₃ attends to en | img₃ cannot see a | img₃ cannot see dog | img₃ cannot see runs |
| img₄ | img₄ attends to img₁ | img₄ attends to img₂ | img₄ attends to img₃ | img₄ attends to img₄ | img₄ attends to caption | img₄ attends to en | img₄ cannot see a | img₄ cannot see dog | img₄ cannot see runs |
| caption | caption attends to img₁ | caption attends to img₂ | caption attends to img₃ | caption attends to img₄ | caption attends to caption | caption attends to en | caption cannot see a | caption cannot see dog | caption cannot see runs |
| en | en attends to img₁ | en attends to img₂ | en attends to img₃ | en attends to img₄ | en attends to caption | en attends to en | en cannot see a | en cannot see dog | en cannot see runs |
| a | a attends to img₁ | a attends to img₂ | a attends to img₃ | a attends to img₄ | a attends to caption | a attends to en | a attends to a | a cannot see dog | a cannot see runs |
| dog | dog attends to img₁ | dog attends to img₂ | dog attends to img₃ | dog attends to img₄ | dog attends to caption | dog attends to en | dog attends to a | dog attends to dog | dog cannot see runs |
| runs | runs attends to img₁ | runs attends to img₂ | runs attends to img₃ | runs attends to img₄ | runs attends to caption | runs attends to en | runs attends to a | runs attends to dog | runs attends to runs |
Full attention over the prefix lets the image tokens take the task prompt into account before any answer token is produced. In the PaliGemma paper's ablations, this prefix-LM setup, with the loss only on the answer, clearly beat a fully causal mask.
CLIP and PaliGemma side by side
Three ways to combine modalities
- Meet only in the loss. Dual encoders such as CLIP, ALIGN and SigLIP never mix the modalities inside the network. Image and text embeddings can be computed and indexed separately, which makes retrieval fast, but these models cannot generate text.
- Cross-attention into a language model. Flamingo adds cross-attention layers to a frozen language model, so that text tokens attend to image features. BLIP-2's Q-Former uses cross-attention to compress image features into a few query tokens that the language model reads.
- Image tokens as a prefix. LLaVA, PaliGemma and Qwen2-VL project image features into the language model's embedding space and place them in the same sequence as the text, so no new attention layers are needed.
Conclusion
CLIP showed that web-scale image-text pairs and a contrastive loss are enough to line up images and language in one embedding space. Generative models such as PaliGemma start from an image encoder trained that way and feed its tokens directly into a language model, so one decoder can caption, answer and read. Many open vision-language models follow the same recipe: a contrastively pretrained image encoder, a projection, and a language model.
Additional Resources
Interested in booking this talk?
I'd love to bring this topic to your event! Get in touch to discuss logistics, timing, and any specific areas you'd like me to focus on.

