Skip to main content
Previously Presented

Multimodal Models: History and How They Work

A brief history of vision-language models, from CLIP's contrastive dual encoders to PaliGemma's image-prefix decoder, with interactive visualizations.

Abhik Sarkar speaking with a microphone beside a projected title slide reading "Understanding Multimodal Models: A Brief History and How They Work" at the Build with AI BangPypers meetup in Bangalore
30 minutes
Bangalore, India
Advanced

Want to book this talk?

Why multimodal models

A model that only reads text and a model that only sees images each miss half of what a captioned photo carries. Multimodal models take more than one kind of input, most often images and text, and learn how the two relate. That lets a single model find the image that matches a sentence, caption a photo, or answer a question about a chart.

This talk followed that idea from the first image-text systems to current vision-language models, with two stops in detail: CLIP, which aligns images and text, and PaliGemma, which generates text about images.

A brief history

  • 2015, captioning and VQA. Show and Tell paired a CNN image encoder with an LSTM that wrote captions, and the VQA dataset made "answer a question about this image" a benchmark. Each system was trained for one task on hand-labeled data.
  • 2019, two-stream transformers. ViLBERT and LXMERT gave images and text their own transformer streams, linked them with cross-attention, and pretrained on captioned images before fine-tuning for each task.
  • 2021, contrastive pretraining at web scale. CLIP and ALIGN trained an image encoder and a text encoder on image-text pairs scraped from the web (400 million for CLIP, 1.8 billion for ALIGN), so that matching images and captions land close together in one embedding space.
  • 2022, language models learn to see. Flamingo added gated cross-attention layers to a frozen language model. BLIP and CoCa combined contrastive and captioning objectives. PaLI paired a large ViT with an mT5 encoder-decoder and trained it on a mixture of tasks in over 100 languages.
  • 2023, cheaper bridges. BLIP-2 connected a frozen image encoder to a frozen language model through a small Q-Former, and SigLIP replaced CLIP's softmax loss with a pairwise sigmoid loss that works at smaller batch sizes.
  • 2024, open vision-language models. PaliGemma paired a SigLIP image encoder with the Gemma language model, and Qwen2-VL processed images at their native resolution.

CLIP: aligning images and text

Two encoders, one space

CLIP has two encoders that never see each other's input:

  • Image encoder: a ResNet or a Vision Transformer (ViT) that turns an image into a vector.
  • Text encoder: a Transformer that turns a caption into a vector.

A learned linear projection maps each output into a shared embedding space, and both vectors are L2-normalized, so the dot product of an image vector and a text vector is their cosine similarity. The walkthrough below follows an image through the ViT-B/32 encoder: splitting it into patches, running the transformer, and projecting the [CLS] output into the shared space.

The contrastive loss

For a batch of N image-caption pairs, CLIP scores all N^2 combinations. Each image is trained to pick out its own caption among the N candidates, and each caption to pick out its own image. With L2-normalized image embeddings I_i and text embeddings T_j, the image-side loss is:

\mathcal{L}_{\text{image}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(I_i^\top T_i / \tau)}{\sum_{j=1}^{N} \exp(I_i^\top T_j / \tau)}

Here \tau is a learned temperature. The text-side loss \mathcal{L}_{\text{text}} swaps the roles and normalizes over images for each caption, and CLIP minimizes the average of the two:

\mathcal{L} = \tfrac{1}{2}\left(\mathcal{L}_{\text{image}} + \mathcal{L}_{\text{text}}\right)

The contrastive loss concept page derives this in more detail.

The shared latent space

Training pulls each image toward its own caption and away from every other caption in the batch. Drag the slider to watch the embeddings sort themselves by concept as training progresses.

Because related images and captions end up close together, CLIP can classify images into categories it was never trained on: embed a prompt such as "a photo of a dog" for each class name and pick the class whose text vector is nearest to the image vector. What CLIP cannot do is write. It scores pairs, but it has no decoder to generate a caption or an answer.

PaliGemma: generating text about images

PaliGemma, released by Google in 2024, is a small open model that captions images, answers questions about them, reads text in them and detects objects, all by generating text. It has three parts:

  • SigLIP image encoder (SigLIP-So400m), which turns a 224×224 image into 256 image tokens. At 448 px it produces 1,024 tokens, and at 896 px, 4,096.
  • Linear projection, which maps each image token to the embedding size of the language model.
  • Gemma-2B decoder, which reads the image tokens followed by a text prompt and generates the answer one token at a time.

The prompt names the task, for example caption en or answer en what is the dog doing. The input sequence is the image tokens, then this prompt (the prefix), then the answer (the suffix). Training is next-token prediction, with the loss computed only on the suffix.

The prefix-LM attention mask

PaliGemma has no separate cross-attention layers. The image tokens sit in the same sequence as the text, so the text reaches the image through the decoder's ordinary self-attention. What sets PaliGemma apart is the mask: the image tokens and the prefix attend to each other in both directions, while the answer is generated causally, one token after another.

query ↓ key →img₁img₂img₃img₄captionenadogruns
img₁img₁ attends to img₁img₁ attends to img₂img₁ attends to img₃img₁ attends to img₄img₁ attends to captionimg₁ attends to enimg₁ cannot see aimg₁ cannot see dogimg₁ cannot see runs
img₂img₂ attends to img₁img₂ attends to img₂img₂ attends to img₃img₂ attends to img₄img₂ attends to captionimg₂ attends to enimg₂ cannot see aimg₂ cannot see dogimg₂ cannot see runs
img₃img₃ attends to img₁img₃ attends to img₂img₃ attends to img₃img₃ attends to img₄img₃ attends to captionimg₃ attends to enimg₃ cannot see aimg₃ cannot see dogimg₃ cannot see runs
img₄img₄ attends to img₁img₄ attends to img₂img₄ attends to img₃img₄ attends to img₄img₄ attends to captionimg₄ attends to enimg₄ cannot see aimg₄ cannot see dogimg₄ cannot see runs
captioncaption attends to img₁caption attends to img₂caption attends to img₃caption attends to img₄caption attends to captioncaption attends to encaption cannot see acaption cannot see dogcaption cannot see runs
enen attends to img₁en attends to img₂en attends to img₃en attends to img₄en attends to captionen attends to enen cannot see aen cannot see dogen cannot see runs
aa attends to img₁a attends to img₂a attends to img₃a attends to img₄a attends to captiona attends to ena attends to aa cannot see doga cannot see runs
dogdog attends to img₁dog attends to img₂dog attends to img₃dog attends to img₄dog attends to captiondog attends to endog attends to adog attends to dogdog cannot see runs
runsruns attends to img₁runs attends to img₂runs attends to img₃runs attends to img₄runs attends to captionruns attends to enruns attends to aruns attends to dogruns attends to runs
Full attention over image + prefixCausal over the answerMasked
PaliGemma’s prefix-LM mask for a toy “caption en” prompt. Rows are queries, columns are keys. The loss is computed only on the answer tokens.

Full attention over the prefix lets the image tokens take the task prompt into account before any answer token is produced. In the PaliGemma paper's ablations, this prefix-LM setup, with the loss only on the answer, clearly beat a fully causal mask.

CLIP and PaliGemma side by side

CLIPContrastive dual encoder, 2021
PaliGemmaImage-prefix decoder, 2024
Output
A similarity score for an image-text pair
Generated text
Architecture
Separate image and text encoders
SigLIP image encoder, linear projection, Gemma decoder
Where modalities meet
Only in the loss, through a dot product
Inside the decoder's self-attention
Training objective
Symmetric contrastive loss
Next-token prediction on the answer
Typical use
Retrieval and zero-shot classification
Captioning, visual question answering, OCR, detection

Three ways to combine modalities

  1. Meet only in the loss. Dual encoders such as CLIP, ALIGN and SigLIP never mix the modalities inside the network. Image and text embeddings can be computed and indexed separately, which makes retrieval fast, but these models cannot generate text.
  2. Cross-attention into a language model. Flamingo adds cross-attention layers to a frozen language model, so that text tokens attend to image features. BLIP-2's Q-Former uses cross-attention to compress image features into a few query tokens that the language model reads.
  3. Image tokens as a prefix. LLaVA, PaliGemma and Qwen2-VL project image features into the language model's embedding space and place them in the same sequence as the text, so no new attention layers are needed.

Conclusion

CLIP showed that web-scale image-text pairs and a contrastive loss are enough to line up images and language in one embedding space. Generative models such as PaliGemma start from an image encoder trained that way and feed its tokens directly into a language model, so one decoder can caption, answer and read. Many open vision-language models follow the same recipe: a contrastively pretrained image encoder, a projection, and a language model.

Share this talk

Additional Resources

Interested in booking this talk?

I'd love to bring this topic to your event! Get in touch to discuss logistics, timing, and any specific areas you'd like me to focus on.

About the Speaker

Abhik Sarkar

Abhik Sarkar

AI researcher and engineer specializing in machine learning systems. Passionate about making complex AI concepts accessible.

Mastodon