TL;DR
CLIP learns a joint embedding space for images and text by training dual encoders on 400 million image-text pairs with a contrastive objective, tackling the vision-language alignment problem at scale. The resulting model can perform zero-shot image classification by comparing image embeddings against text embeddings of class descriptions — no task-specific training data required. On ImageNet, CLIP’s zero-shot accuracy matches a fully supervised ResNet-50, without using the labeled ImageNet training split. Web captions can still name ImageNet classes, and the paper audits possible image overlap rather than claiming those concepts were unseen. The approach shifts the paradigm from fixed-label classification to open-vocabulary visual understanding, and its embeddings have become the backbone of text-to-image generation systems like DALL-E 2 and Stable Diffusion.
The Core Idea: Language as Supervision
Traditional vision models learn from fixed label sets: 1,000 ImageNet classes, 80 COCO categories, and so on. Each new task requires a new labeled dataset. CLIP replaces this with natural language supervision — instead of learning "this image is class 537," the model learns "this image matches the caption 'a golden retriever playing fetch in a park.'" Language carries far richer information than a class index, and it scales naturally: the internet contains billions of image-text pairs that require no manual annotation.
The idea is not new. VirTex (Desai & Johnson, 2021) and ICMLM (Bulent Sariyildiz et al., 2020) explored language supervision, but on small datasets like COCO Captions (around 500K pairs). CLIP’s contribution is demonstrating that scaling this approach to 400 million pairs, combined with a contrastive (rather than generative) objective, produces representations that transfer competitively to dozens of downstream tasks without any fine-tuning.
Contrastive Pre-training Objective
CLIP uses a symmetric contrastive loss that operates over batches of N image-text pairs. Given a batch, the model computes cosine similarities between all N^2 possible image-text combinations, then trains to maximize the similarity of the N correct pairs while minimizing the similarity of the N^2 - N incorrect pairs.
For a batch of N pairs, let I_i and T_j denote the L2-normalized embeddings of image i and text j. The loss for the image side is:
where \text{sim}(I_i, T_j) = I_i^\top T_j is cosine similarity and \tau is a learned temperature parameter. A symmetric text-side loss \mathcal{L}_{\text{text}} is computed analogously, and the total loss is the average of both:
This is an N-way classification problem in both directions: each image must identify its matching text among N candidates, and each text must identify its matching image. The temperature \tau is initialized to 0.07 and learned during training; it controls the sharpness of the softmax distribution and has a measurable impact on downstream zero-shot performance.
In Figure 2, switching from bag-of-words prediction to the contrastive objective improves the rate of zero-shot ImageNet transfer by about 4×. The caption-generation baseline is already about 3× slower than bag-of-words prediction; the reported 4× comparison is not directly against caption generation.
The following matrix is a deterministic teaching illustration, with authored similarities and an animated interpolation rather than an actual training run.
Architecture: Dual Encoders
CLIP consists of two independent encoders that project images and text into a shared embedding space:
Image encoder. The paper evaluates two families: ResNet (modified with attention pooling and anti-aliased rect-2 blur pooling) and Vision Transformer (ViT). The largest model uses ViT-L/14 with input resolution 336x336. The image encoder outputs a single vector by taking the [CLS] token (ViT) or attention-pooled global features (ResNet), then projecting through a learned linear layer to the shared embedding dimension.
Text encoder. The base text Transformer has 12 layers, width 512 and eight attention heads, with GPT-style causal attention. Section 2.4 describes a 49,152-entry BPE vocabulary and sequence cap of 76. For executing the released ViT-B/32, the official tokenizer/model contract is instead 49,408 vocabulary entries and a 77-position context, including SOT and EOT, leaving at most 75 content tokens. The final EOT representation is layer normalized and linearly projected; padding follows EOT. The text encoder learns from scratch alongside the image encoder.
Projection. Both encoders use learned linear maps into one shared embedding space, followed by L2 normalization. Released ViT-B/32 and ViT-B/16 use 512 dimensions; ViT-L/14 uses 768. Other released architectures use other widths, so this is not one fixed CLIP-wide dimension. Similarity is the dot product of unit vectors, which equals cosine similarity.
The dual-encoder design is deliberate: it allows independent encoding of images and text, which means embeddings can be precomputed and cached. This makes retrieval and zero-shot classification fast at inference — the text embeddings for class descriptions only need to be computed once.
Training at Scale: The WIT Dataset
CLIP is trained on WebImageText (WIT), a dataset of 400 million image-text pairs collected from the internet. The dataset is not publicly released, but the paper describes the construction process: they started with 500,000 search queries (derived from Wikipedia article titles and WordNet synsets) and collected up to 20,000 image-text pairs per query from public internet sources.
Training details for the largest ViT-L/14 model: 32 epochs over the 400M dataset, batch size of 32,768, Adam optimizer with decoupled weight decay, cosine learning rate schedule. Training used 256 V100 GPUs for 12 days (ViT-L/14), and the ViT-L/14@336px variant was pre-trained for one additional epoch at 336-pixel resolution. The largest ResNet, RN50x64, took 18 days on 592 V100 GPUs. The large batch size is important — each anchor sees one paired positive and 32,767 in-batch negative candidates in each direction. The training matrix contains 32,768² ordered image-text comparisons.
The paper suggests that WIT’s advantage comes from its size rather than its source. In an appendix ablation, a ResNet-50 trained on the filtered 15M-pair subset of YFCC100M performs similarly on average to one trained on an equally sized subset of WIT (29.6% vs 30.0% zero-shot, 65.5% vs 66.6% linear probe), although individual fine-grained datasets differ by more than 10 points: YFCC does better on Birdsnap and Flowers102, WIT on Stanford Cars. The authors read this as evidence that the approach works with any reasonably filtered collection of image-text pairs, and suspect that WIT’s main advantage over YFCC100M is its much larger size.
Zero-Shot Transfer
At inference, CLIP performs classification without any training examples from the target dataset. The procedure:
- Convert each class name into a natural language prompt, e.g., "A photo of a {class name}."
- Encode all prompts with the text encoder to get class embeddings.
- Encode the test image with the image encoder.
- Predict the class whose text embedding has the highest cosine similarity with the image embedding.
The text encoder supplies normalized weights for a linear classifier, and the image encoder supplies its normalized input. The highest cosine determines the prediction. Softmax values are conditional on the supplied class descriptions and temperature; they are not calibrated probabilities that a class is present.
The icon-based demo below uses authored scores to illustrate the protocol. Genuine released-model scores are available in the implementation inspection later on this page.
Prompt Engineering and Ensembling
Raw class names alone produce suboptimal results. The word "crane" could refer to a bird or a construction machine — context matters. The authors found that wrapping class names in descriptive templates significantly improves accuracy.
Prompt templates. Rather than just encoding "dog," CLIP uses "A photo of a dog" or dataset-specific templates like "a satellite photo of {class}" for satellite imagery datasets. The paper reports that this simple change improves ImageNet accuracy by 1.3 percentage points.
Prompt ensembling. Using multiple templates per class, normalizing each embedding, averaging within the class and normalizing the mean yields further gains. The paper uses 80 templates for ImageNet (e.g., "a bad photo of a {class}," "a sculpture of a {class}," "a photo of the large {class}") and averages the resulting text embeddings. This ensemble improves ImageNet zero-shot accuracy by an additional 3.5 percentage points over the single default prompt, reaching 76.2% top-1 accuracy with ViT-L/14@336.
The following prompt demo is illustrative; its scores are not outputs from a pretrained model.
Implementation: From paired encoders to candidate scores
The theory now becomes two executable paths: a small paired-encoder training step that exposes the objective, and authentic ViT-B/32 inference that exposes exactly which prompts produced each score. Follow the operations before returning to the paper’s large-scale evaluation.
Trace nine operations through executable CPU code. The training program uses a reduced random-weight ViT and causal text transformer, one block per encoder, width 32, a 16-dimensional shared space and synthetic token IDs. It performs a real optimizer step. The separate photo inspection runs the full official pretrained CLIP ViT-B/32; those saved features and logits are genuine model outputs.
Training program · Pinned requirements · Native exporter · Licensed native model and BPE source · Source hashes · Reproduction instructions
python clip_training.py --smoke
Smoke is offline and uses only PyTorch. Authentic export additionally needs the public source ZIP and official checkpoint, with strict hash and architecture checks. It accepts printable ASCII prompts only. The adapter rejects Unicode and controls rather than approximating the unavailable ftfy/regex text normalization. No extra packages are installed. Existing conceptual demos elsewhere on this page remain illustrations; the photo arrays below are the pretrained evidence.
1. Bracket token IDs and keep EOT inside the context
Original paper
The paper brackets BPE text with start/end markers. The release pads to 77 positions with zeros, using vocabulary IDs 49406 and 49407 for SOT and EOT.
Math and tensor shapes
This region handles already-tokenized content; smoke uses synthetic IDs. The genuine exporter separately applies the pinned byte map and BPE merges with printable ASCII cleanup and contraction-first splitting. Numbers are individual digits. Unicode, controls, embedded special tokens and HTML entities that need more than two cleanup passes raise errors; overlong text is never silently truncated.
Smoke [3,8], vocabulary 64; native [10,77], vocabulary 49408
Python implementation
def pad_tokens(contents, context=8, sot=62, eot=63):
"""Bracket integer content IDs; forbid specials inside the content."""
if context < 2 or not 0 <= sot < eot or not contents:
raise ValueError("Require nonempty batch, context >= 2 and SOT < EOT.")
result = torch.zeros(len(contents), context, dtype=torch.long)
for row, content in enumerate(contents):
if len(content) + 2 > context:
raise ValueError("Text exceeds context; no silent truncation.")
if any(type(x) is not int or not 0 <= x < sot for x in content):
raise ValueError(
"Content IDs must be non-special nonnegative ints."
)
sequence = [sot, *content, eot]
result[row, : len(sequence)] = torch.tensor(sequence)
return result2. Project the image CLS representation
Original paper
CLIP adds normalization before the ViT transformer. Final CLS features pass through another normalization and a learned linear projection.
Math and tensor shapes
The reduced encoder uses 8×8 patches, width 32 and one block. Official ViT-B/32 uses 224×224 input, 32×32 patches, 50 tokens, width 768, 12 blocks and a 768×512 projection. Vision attention is bidirectional.
Smoke [B,3,32,32] → [B,17,32] → [B,16]
Python implementation
def forward(self, images):
"""Map [B,3,H,W] through CLS to a learned joint-space projection."""
if images.ndim != 4 or images.shape[1:] != (3, self.size, self.size):
raise ValueError("Unexpected image shape.")
patches = self.patch(images).flatten(2).transpose(1, 2)
cls = self.cls.expand(images.shape[0], 1, -1)
tokens = torch.cat([cls, patches], dim=1) + self.positions
hidden = self.block(self.pre(tokens))
return self.post(hidden[:, 0]) @ self.projection3. Read the causal EOT representation
Original paper
Text features come from the final normalized end-of-text position. The native model uses EOT’s highest vocabulary ID to locate that position with argmax.
Math and tensor shapes
Each query sees only its own and earlier token positions. The example requires exactly one EOT per sequence, then selects it before projection. Padding or altered future content cannot influence its feature. Native ViT-B/32 text has 12 blocks and eight heads; the mechanics profile has one block and four heads.
Smoke [B,8,32] → [B,16]; native [B,77,512] → [B,512]
Python implementation
def hidden(self, tokens):
"""Future token keys receive minus infinity before attention softmax."""
if tokens.ndim != 2 or tokens.shape[1] != self.context:
raise ValueError("Tokens must have the configured context length.")
if (
tokens.dtype != torch.long
or ((tokens < 0) | (tokens > self.eot)).any()
):
raise ValueError("Tokens must be in-vocabulary int64 IDs.")
x = self.embedding(tokens) + self.positions
return self.final(self.block(x, self.mask))
def forward(self, tokens):
"""EOT is the largest ID; require exactly one per sequence."""
if tokens.ndim != 2 or (tokens == self.eot).sum(1).ne(1).any():
raise ValueError("Each sequence must contain exactly one EOT.")
hidden = self.hidden(tokens)
eot_positions = tokens.argmax(dim=-1)
pooled = hidden[torch.arange(tokens.shape[0]), eot_positions]
return pooled @ self.projection4. Normalize both learned embeddings
Original paper
Linear projections connect encoder representations to the joint space. Dividing by each vector’s L2 norm turns their dot product into cosine similarity.
Math and tensor shapes
The program rejects zero or nonfinite embeddings instead of defining an arbitrary cosine for them. Normalization preserves gradient flow into both projections and encoders; it does not freeze them or introduce a nonlinear projection head.
Two [B,d] matrices; each output row has unit norm
Python implementation
def unit_features(features):
"""Normalize each nonzero finite vector; reject undefined cosine inputs."""
if features.ndim != 2 or min(features.shape) < 1:
raise ValueError("Features must be a nonempty matrix.")
norms = features.norm(dim=-1, keepdim=True)
if (
not torch.isfinite(features).all()
or not torch.isfinite(norms).all()
or (norms <= 0).any()
):
raise ValueError("Every embedding must be finite and nonzero.")
return features / norms5. Scale every image-text cosine
Original paper
The learned log-parameter exponentiates into a positive multiplier. The paper caps training scale at 100; the released evaluation forward uses raw exp(logit_scale).
Math and tensor shapes
The mechanics training path caps the multiplier and the parameter after update. Inference uses the released raw scale, including values above 100 if present. Temperature is 1/s; a sharper softmax is not evidence of calibrated certainty.
[B,d] @ [d,C] → [B,C]; paired training uses C=B
Python implementation
def similarity_logits(
image_features, text_features, logit_scale, training=False
):
"""Original training caps scale at 100; released eval uses raw exp(scale)."""
image = unit_features(image_features)
text = unit_features(text_features)
if image.shape[1] != text.shape[1]:
raise ValueError("Image/text embedding dimensions differ.")
if logit_scale.numel() != 1 or not torch.isfinite(logit_scale).all():
raise ValueError("Logit scale must be one finite scalar.")
scale = logit_scale.exp()
if training:
scale = scale.clamp(max=100)
if not torch.isfinite(scale):
raise ValueError("Exponentiated scale is not finite.")
return scale * (image @ text.T)6. Classify paired diagonals in both directions
Original paper
Paper Figure 3 trains each image to find its paired text and each text to find its paired image. The released forward supplies the matrix and its transpose; the loss here implements that pseudocode.
Math and tensor shapes
Batch row i and text row i must be a pair. Both denominators include that positive. There are B−1 negative candidates per anchor. A shared permutation of images and texts preserves the loss; permuting only texts changes the pairing labels.
Square logits [B,B]; labels [0,…,B−1]
Python implementation
def symmetric_loss(logits):
"""The diagonal contains paired positives in both directions."""
if (
logits.ndim != 2
or logits.shape[0] != logits.shape[1]
or not logits.numel()
):
raise ValueError("Paired loss requires a nonempty square logit matrix.")
if not torch.isfinite(logits).all():
raise ValueError("Logits must be finite.")
labels = torch.arange(logits.shape[0], device=logits.device)
image_loss = F.cross_entropy(logits, labels)
text_loss = F.cross_entropy(logits.T, labels)
return (image_loss + text_loss) / 27. Update encoders, projections and scale together
Original paper
Both encoders start from scratch in the original experiment. The paper uses Adam with decoupled weight decay to optimize the paired objective.
Math and tensor shapes
One deterministic CPU step checks execution and gradient reachability. It uses random images and synthetic tokens, and omits WIT data collection, image augmentation, distributed batches, mixed precision and the original schedule. Its numbers demonstrate mechanics, not semantic learning or benchmark reproduction.
Smoke batch 3; gradients reach image/text weights and projections
Python implementation
def train_step(model, optimizer, images, tokens):
"""Differentiate both encoders, both projections and the learned scale."""
model.train()
optimizer.zero_grad(set_to_none=True)
image_features = model.image(images)
text_features = model.text(tokens)
logits = similarity_logits(
image_features, text_features, model.logit_scale, training=True
)
loss = symmetric_loss(logits)
loss.backward()
optimizer.step()
# Keep the parameter itself inside the original training scale bound.
with torch.no_grad():
model.logit_scale.clamp_(max=math.log(100))
return loss.detach(), logits.detach()8. Average normalized prompts within each class
Original paper
The official notebook normalizes each template embedding, averages within a class and normalizes the mean. The resulting prototype can be cached once for all images.
Math and tensor shapes
The photo export uses exactly two stated templates per class. It does not use the paper’s 80-template ImageNet protocol. Averaging raw features would weight prompts by feature magnitude; averaging softmax scores would implement a different ensemble.
Genuine [5 classes,2 templates,512] → [5,512]
Python implementation
def class_prototypes(prompt_features):
"""Normalize prompts, average within each class, then renormalize."""
if prompt_features.ndim != 3 or min(prompt_features.shape) < 1:
raise ValueError("Require [classes, templates, embedding dimension].")
classes, templates, dim = prompt_features.shape
normalized = unit_features(prompt_features.reshape(-1, dim))
averaged = normalized.reshape(classes, templates, dim).mean(1)
return unit_features(averaged)9. Score only the supplied candidate descriptions
Original paper
Text prototypes define a zero-shot linear classifier. The same image embedding can be compared with a new set of descriptions without retraining the encoders.
Math and tensor shapes
The denominator depends on the chosen candidate set C. Removing a candidate leaves retained cosine/logit values unchanged but renormalizes their softmax. These outputs are uncalibrated and cannot show that an omitted class is absent from the image.
[1,512] @ [512,C] → [1,C]
Python implementation
def candidate_scores(image_features, prototypes, logit_scale):
"""Return candidate-conditional, uncalibrated softmax and cosine scores."""
cosine = unit_features(image_features) @ unit_features(prototypes).T
logits = similarity_logits(image_features, prototypes, logit_scale)
return {"cosine": cosine, "logits": logits, "softmax": logits.softmax(-1)}Inspect genuine CLIP candidate scores
Official ViT-B/32 encodes these two licensed photos and ten exact prompts. Class prototypes average two normalized prompt features and are normalized again. Changing the candidate set recalculates only the softmax denominator from saved logits. A high value can occur even when every offered description is a poor fit.
Exact templates: a photo of a {}. · a close-up photo of a {}. Scale: exp(4.605170) = 100.000008; raw released evaluation scale.
Chelsea the cat
| Candidate | Cosine | Logit | Softmax |
|---|---|---|---|
| cat | 0.3023 | 30.231 | 99.53% |
| dog | 0.2464 | 24.640 | 0.37% |
| cup of coffee | 0.2237 | 22.372 | 0.04% |
| bowl of soup | 0.2271 | 22.712 | 0.05% |
| car | 0.2109 | 21.090 | 0.01% |
Photo: Stefan van der Walt, CC0. Pinned photograph · License · Exact input, tokens and full feature/logit arrays
Cup of coffee
| Candidate | Cosine | Logit | Softmax |
|---|---|---|---|
| cat | 0.1944 | 19.442 | 0.00% |
| dog | 0.1861 | 18.607 | 0.00% |
| cup of coffee | 0.3048 | 30.482 | 99.79% |
| bowl of soup | 0.2428 | 24.277 | 0.20% |
| car | 0.1951 | 19.507 | 0.00% |
Photo: Rachel Michetti, CC0. Pinned photograph · License · Exact input, tokens and full feature/logit arrays
Full provenance and numeric record. Native source revision d05afc436d78f1c48dc0dbf8e5980a9d471f35f6; torch 2.12.0+cpu, CPU float32, two threads. Preprocessing resizes the short side to 224 with PIL bicubic, takes a 224×224 center crop, converts to RGB and applies the released CLIP mean/std. Archives preserve every 512-dimensional image/text feature, ten prompt logits and five candidate logits. Direct native forward parity and independent arithmetic checks passed. Two photos do not measure zero-shot accuracy or reproduce the paper’s benchmarks.
Key Results
ImageNet zero-shot. CLIP ViT-L/14@336 achieves 76.2% top-1 zero-shot accuracy on ImageNet, matching the 76.1% of a supervised ResNet-50 trained on 1.28 million labeled ImageNet images. This compares zero-shot transfer without training on ImageNet’s labeled split to a supervised ImageNet model. It does not establish that class names or related images were absent from WIT; Section 5 separately analyzes data overlap.
Broad transfer. CLIP is evaluated across 27 datasets spanning OCR (SVHN, MNIST), fine-grained classification (Stanford Cars, Food-101, Flowers-102), satellite imagery (EuroSAT), medical imaging (PatchCamelyon), action recognition (Kinetics-700, UCF-101), and general recognition (ImageNet, CIFAR-10/100). On 16 of these 27 datasets, zero-shot CLIP outperforms a supervised ResNet-50 trained on ImageNet features with linear probing.
Distribution shift robustness. On ImageNet variants (ImageNet-V2, ImageNet-R, ImageNet-A, ImageNet-Sketch, ObjectNet), CLIP shows notably better robustness than supervised ImageNet models. Section 3.3 compares performance on several shifted distributions after accounting for in-distribution ImageNet accuracy. Zero-shot CLIP shows higher effective robustness than the supervised comparison models. The analysis does not isolate language supervision from training data and compute as the sole cause.
Linear probe efficiency. When a linear classifier is trained on CLIP features (linear probing), performance improves further. CLIP ViT-L/14@336px with linear probing achieves 85.4% on ImageNet, competitive with state-of-the-art supervised models at the time.
Critical Analysis
Strengths.
- Zero-shot generalization. CLIP removes the need for task-specific labeled data. A single pretrained model transfers to dozens of tasks through language alone, fundamentally changing the economics of deploying vision models.
- Natural language as the interface. By using text prompts rather than class indices, CLIP supports open-vocabulary recognition. New classes can be added at inference time without retraining.
- Distribution shift robustness. The broad, web-scale training data produces representations that generalize better across domain shifts than models trained on curated datasets.
- Modular dual-encoder design. Precomputed embeddings enable efficient retrieval at scale, unlike cross-attention architectures that require joint encoding of each image-text pair.
Limitations.
- Counting and spatial reasoning. CLIP struggles with tasks requiring compositional understanding. It cannot reliably distinguish "two dogs and one cat" from "one dog and two cats" because the contrastive objective treats captions as holistic descriptions rather than structured propositions.
- Fine-grained and specialized classification. Section 3.1.5 finds substantial deficits on tasks including Flowers102 and FGVC Aircraft, while other fine-grained tasks favor CLIP. The authors suggest differences in relevant supervision as one explanation; the result does not establish that every specialized task has a 20-point deficit.
- Abstract and systematic reasoning. On MNIST (88% zero-shot), CLIP performs worse than simple logistic regression on raw pixels. Tasks requiring systematic pattern recognition rather than natural image understanding expose the limits of language supervision.
- Data and compute cost. 400 million curated pairs and hundreds of GPU-days of training are not accessible to most researchers. The WIT dataset is not public, limiting reproducibility.
- Social biases. The paper documents that CLIP inherits biases from web-crawled data, including associations between demographic attributes and negative stereotypes. These biases transfer directly to downstream applications.
Impact and Legacy
CLIP’s influence extends well beyond zero-shot classification. Its joint embedding space became a foundational component in generative AI:
- DALL-E 2 (Ramesh et al., 2022) uses CLIP embeddings as the conditioning signal for its diffusion-based image generator, mapping from text to CLIP text embeddings, then from CLIP image embeddings to pixels.
- Stable Diffusion (Rombach et al., 2022) uses CLIP’s text encoder to condition the latent diffusion process, making CLIP’s text representations the language interface for open-source image generation.
- Open-source alternatives. OpenCLIP (Ilharco et al.) reproduced CLIP training on public datasets (LAION-2B, LAION-5B), achieving comparable or better performance and enabling the research community to build on CLIP without access to the proprietary WIT dataset.
- CLIP-guided methods. CLIPSeg (image segmentation), CLIP-NeRF (3D generation), AudioCLIP (audio-visual), and many others extend the CLIP framework to new modalities and tasks.
- Evaluation standard. CLIP’s zero-shot protocol complements linear probing by testing how visual features connect to natural-language class descriptions. The two protocols measure different transfer settings.
The core lesson of CLIP is that scale and supervision source matter more than architectural novelty. The image and text encoders are standard architectures (ViT, GPT-2 Transformer); the innovation is training them jointly on web-scale data with a contrastive objective. This insight — that the right data and training signal can substitute for architectural complexity — has shaped the direction of multimodal AI research since.
Related Reading
- ALIGN — CLIP’s web-scale sibling, showing the same dual-encoder contrastive recipe works on 1.8B noisy alt-text pairs with minimal curation
- SigLIP — replaces CLIP’s softmax contrastive loss with a pairwise sigmoid loss that scales to small batches without a global all-gather
- CoCa — adds a generative captioning objective on top of CLIP-style contrastive learning, training one model for both retrieval and generation
- Attention Is All You Need — the transformer architecture underlying both of CLIP’s encoders
- Vision Transformer — ViT, the image encoder architecture that gives CLIP its best results
- DINO — self-supervised vision transformers that learn visual features without any labels, contrasting with CLIP’s language supervision
- SimCLR — contrastive learning within a single modality (images only), providing context for CLIP’s cross-modal contrastive approach
- BLIP-2 — extends the CLIP paradigm by bridging frozen image encoders with large language models for richer multimodal reasoning
- Latent Diffusion Models — uses CLIP’s text encoder as the conditioning mechanism for Stable Diffusion
- Vision-Language Alignment — compares CLIP’s contrastive joint embedding with the LLM bridges of BLIP-2, LLaVA and Flamingo, and what each trains or freezes