TL;DR
- BLIP introduces a Multimodal mixture of Encoder-Decoder (MED) that shares weights across three modes — unimodal encoder (ITC), image-grounded text encoder (ITM), and image-grounded text decoder (LM) — enabling a single pre-trained model to both score and generate.
- BLIP learns understanding and generation jointly, sharing embeddings, cross-attention, and feed-forward layers while keeping encoder and decoder self-attention separate.
- CapFilt bootstraps noisy web data: a captioner synthesizes fresh captions for web images; a filter removes noisy pairs from both the originals and the synthetic set; the cleaned, augmented corpus retrains the model.
- BLIP was the direct predecessor of BLIP-2, which replaced BLIP's end-to-end training with a frozen-backbone Q-Former.
Why bootstrap the data?
Web image-text pairs are plentiful but noisy. BLIP's CapFilt pipeline creates new captions and filters both the original and generated candidates. To see how those two modules arise, first follow MED's three modes and their joint objectives.
One model, three modes: MED
BLIP's core architecture is the Multimodal mixture of Encoder-Decoder (MED). Its trainable image encoder (ViT) and partially shared text encoder/decoder are optimized end to end. The text model is activated in one of three modes by switching which attention layers are enabled:
Unimodal encoder (ITC loss). Self-attention only; no cross-attention to the image. A [CLS] token aggregates the text. The output is compared to the image's [CLS] token via a contrastive loss, pulling matched image-text pairs together and pushing unmatched pairs apart in the embedding space. This mode learns a contrastive representation.
Image-grounded text encoder (ITM loss). Bidirectional self-attention plus cross-attention to the image encoder's patch features. An [Encode] token produces a multimodal embedding that is fed to a binary classifier predicting whether the image and text are matched. Hard negative mining samples unmatched pairs in proportion to their contrastive similarity, after excluding the paired positive.
Image-grounded text decoder (LM loss). Causal (masked) self-attention plus cross-attention to the image. The model autoregressively predicts caption tokens conditioned on the image, supervised by a language modeling loss. This is the mode used for caption generation at inference.
The unimodal and grounded encoders share their text stack. The decoder shares embeddings, cross-attention, and feed-forward parameters with the encoder, but has separate self-attention parameters. Encoder attention is bidirectional; decoder attention is causal. The joint objective adds ITC, ITM, and LM. Original ITC includes momentum-encoder soft targets and a feature queue; LM shifts next-token targets, ignores pads, and applies label smoothing of 0.1.
Implementation
The three modes differ in both parameter routing and available context. The executable reference makes those distinctions visible before following the pretrained model into data curation and downstream tasks.
Offline seeded CPU mechanics: one width-eight text block, a trainable linear patch encoder, four-dimensional ITC features, and a local eight-slot momentum queue. There are no pretrained weights, BERT tokenizer, learned caption semantics, GPU collectives, or benchmark reproduction. Momentum distillation is included at this reduced scale; distributed all-gather and the original ViT/BERT networks are omitted. All equations below are authored explanations of the cited operations.
1. Share embeddings, cross-attention, and FFN
Original paper
The release excludes /attention from encoder/decoder parameter tying.
Math and tensor shapes
The two paths reference the same SharedText object, while each owns separate self-attention parameters and its associated LayerNorm. Unimodal and grounded encoding use the same encoder. The linear image encoder remains trainable: original BLIP is end-to-end. One block and mean patch pooling replace BERT and ViT CLS readout here.
Tokens [B, L] → hidden [B, L, 8]; patches [B, P, 6] → [B, P, 8]
Python implementation
class SharedText(nn.Module):
"""Embeddings, cross-attention and FFN shared by encoder and decoder."""
def __init__(self, width=8, vocab=16):
super().__init__()
self.embedding = nn.Embedding(vocab, width, padding_idx=PAD)
self.position = nn.Embedding(8, width)
self.cross = nn.MultiheadAttention(width, 2, batch_first=True)
self.cross_norm = nn.LayerNorm(width)
self.ffn = nn.Sequential(
nn.Linear(width, 16), nn.GELU(), nn.Linear(16, width)
)
self.ffn_norm = nn.LayerNorm(width)
class TextPath(nn.Module):
"""Each path owns separate self-attention and its associated LayerNorm."""
def __init__(self, shared, width=8):
super().__init__()
self.shared = shared
self.self_attention = nn.MultiheadAttention(width, 2, batch_first=True)
self.self_norm = nn.LayerNorm(width)
def forward(self, tokens, image=None, causal=False):
validate_tokens(tokens)
shared = self.shared
x = shared.embedding(tokens) + shared.position(
torch.arange(tokens.shape[1])
)
blocked = attention_mask(tokens.shape[1], causal)
attended = self.self_attention(
x,
x,
x,
attn_mask=blocked,
key_padding_mask=tokens == PAD,
need_weights=False,
)[0]
x = self.self_norm(x + attended)
if image is not None:
x = shared.cross_norm(
x + shared.cross(x, image, image, need_weights=False)[0]
)
return shared.ffn_norm(x + shared.ffn(x))
class MED(nn.Module):
"""Tiny trainable image/text model preserving the three parameter routes."""
def __init__(self):
super().__init__()
shared = SharedText()
self.encoder = TextPath(shared)
self.decoder = TextPath(shared)
self.image_encoder = nn.Linear(6, 8)
self.image_projection = nn.Linear(8, 4)
self.text_projection = nn.Linear(8, 4)
self.itm_head = nn.Linear(8, 2)
self.lm_head = nn.Linear(8, 16)
self.temperature = nn.Parameter(torch.tensor(0.07))
def image(self, patches):
if patches.ndim != 3 or patches.shape[1] < 1 or patches.shape[2] != 6:
raise ValueError("patches must have shape [B, P >= 1, 6]")
if not torch.isfinite(patches).all():
raise ValueError("patches must be finite")
return self.image_encoder(patches)2. Route unimodal, grounded, and causal paths
Original paper
Image conditioning and future-token visibility are separate switches.
Math and tensor shapes
True blocks attention in the PyTorch mask. Both encoders are bidirectional; only the grounded encoder receives image keys/values. The decoder uses its separate self-attention with a causal mask and shared cross-attention. Pads are excluded as keys. Start tokens are CLS for ITC, Encode for ITM, and Decode for LM. Image perturbation cannot change unimodal text features; future text cannot affect earlier decoder positions.
Block mask [L, L]; suffix padding key mask [B, L]
Python implementation
def attention_mask(length, causal):
"""True blocks a key; decoder position i cannot attend to j > i."""
if not isinstance(length, int) or length < 1:
raise ValueError("length must be a positive integer")
return (
torch.ones(length, length, dtype=torch.bool).triu(1) if causal else None
)3. Align normalized unimodal features
Original paper
Current online queries compare with detached momentum keys and a queue.
Math and tensor shapes
Bars denote L2 normalization over the four feature dimensions. Each directional loss averages its query rows. Temperature is trainable and clamped to [0.001, 0.5] before the forward. This is a reduced soft-label ITC objective, rather than labeling hard batch-only contrastive CE as the full original loss.
Features [B, 4]; logits and targets [B, B + K]; smoke [3, 11]
Python implementation
def unimodal_features(model, image, tokens):
"""Text receives no image, and both projected representations are normalized."""
text = model.encoder(tokens)
# Mean patches replace the full ViT's learned CLS token in this reference.
image_features = F.normalize(model.image_projection(image.mean(1)), dim=-1)
text_features = F.normalize(model.text_projection(text[:, 0]), dim=-1)
return image_features, text_features
def contrastive_loss(image, text, bank_image, bank_text, targets, temperature):
"""Symmetric soft-label ITC against detached momentum current/queue keys."""
i2t = image @ bank_text.T / temperature
t2i = text @ bank_image.T / temperature
losses = [
-(target * F.log_softmax(logits, dim=1)).sum(1).mean()
for logits, target in zip((i2t, t2i), targets)
]
return (losses[0] + losses[1]) / 2, i2t, t2i4. Detach momentum targets and queued keys
Original paper
EMA happens before the teacher forward; detached keys enter a FIFO queue.
Math and tensor shapes
A copied teacher has no gradients. Its current image/text features precede cloned queue contents in the bank. With alpha = 0.4 and momentum = 0.995, each row mixes an identity positive with teacher probabilities. Enqueue follows ITC construction and precedes ITM/LM, as in the release. The local ring supports arbitrary batch sizes; it omits distributed all-gather and the upstream global-batch divisibility constraint. This reduced teacher copies the entire tiny MED, including unused heads; original EMA covers only the ITC encoder/projection pairs.
Frozen teacher; queue [K, 4], K = 8; target [B, B + K]
Python implementation
class MomentumBank:
"""Frozen EMA teacher and a local FIFO of normalized paired ITC features."""
def __init__(self, model, capacity=8):
if not isinstance(capacity, int) or capacity < 1:
raise ValueError("capacity must be positive")
self.teacher = copy.deepcopy(model).requires_grad_(False)
self.images = F.normalize(torch.randn(capacity, 4), dim=-1)
self.texts = F.normalize(torch.randn(capacity, 4), dim=-1)
self.pointer = 0
@torch.no_grad()
def targets(self, model, patches, tokens, alpha=0.4, momentum=0.995):
if not 0 <= alpha <= 1 or not 0 <= momentum <= 1:
raise ValueError("alpha and momentum must be in [0, 1]")
for teacher, online in zip(
self.teacher.parameters(), model.parameters()
):
teacher.mul_(momentum).add_(online, alpha=1 - momentum)
image, text = unimodal_features(
self.teacher, self.teacher.image(patches), tokens
)
bank_image = torch.cat((image, self.images.clone()))
bank_text = torch.cat((text, self.texts.clone()))
hard = torch.zeros(image.shape[0], bank_text.shape[0])
hard[:, : image.shape[0]].fill_diagonal_(1)
soft_i = (image @ bank_text.T / model.temperature).softmax(1)
soft_t = (text @ bank_image.T / model.temperature).softmax(1)
targets = (
alpha * soft_i + (1 - alpha) * hard,
alpha * soft_t + (1 - alpha) * hard,
)
return bank_image, bank_text, targets, image, text
@torch.no_grad()
def enqueue(self, images, texts):
# Serial local ring supports any batch size; upstream uses all-gather
# and requires queue capacity divisible by the global batch size.
for image, text in zip(images, texts):
self.images[self.pointer] = image.detach()
self.texts[self.pointer] = text.detach()
self.pointer = (self.pointer + 1) % len(self.images)5. Sample confusing non-positive pairs
Original paper
Select negative images from text queries and negative texts from image queries.
Math and tensor shapes
Sampling excludes the paired diagonal after adding the probability floor, which keeps alternative candidates sampleable even when the positive dominates numerically. These draws run without gradients using a seeded local generator. This reference assumes unique pairs within a batch; the diagonal is the declared positive set. B = 1 is rejected because no non-positive remains.
Current-batch logits [B, B] → negative indices [B], B ≥ 2
Python implementation
def sample_negatives(similarity, generator):
"""Sample each row's non-positive with upstream's 1e-4 probability floor."""
if similarity.ndim != 2 or similarity.shape[0] != similarity.shape[1]:
raise ValueError("similarity must be square [B, B]")
if similarity.shape[0] < 2 or not torch.isfinite(similarity).all():
raise ValueError("hard-negative mining needs finite logits and B >= 2")
with torch.no_grad():
weights = similarity.softmax(1) + 1e-4
weights.fill_diagonal_(0)
return torch.multinomial(weights, 1, generator=generator).squeeze(1)6. Classify matched and grounded hard negatives
Original paper
One positive batch and two negative batches supervise a grounded binary head.
Math and tensor shapes
Matched image/text pairs have label 1; mismatched image and mismatched text batches have label 0. Encode replaces CLS in a clone. Bidirectional self-attention and image cross-attention build the pair representation. This is distinct from a cosine similarity score: it is the joint multimodal encoder output that feeds the binary head.
Grounded hidden [3B, L, 8] → Encode logits [3B, 2]
Python implementation
def matching_loss(model, image, tokens, negative_image, negative_text):
"""One matched and two unmatched batches, read at the Encode token."""
grounded_tokens = tokens.clone()
grounded_tokens[:, 0] = ENC
all_tokens = torch.cat(
(grounded_tokens, grounded_tokens, grounded_tokens[negative_text])
)
all_images = torch.cat((image, image[negative_image], image))
hidden = model.encoder(all_tokens, all_images)
logits = model.itm_head(hidden[:, 0])
labels = torch.cat(
(torch.ones(len(tokens)), torch.zeros(2 * len(tokens)))
).long()
return F.cross_entropy(logits, labels), logits7. Predict the next token and exclude pads
Original paper
Decoder logits at i supervise token i + 1; the start token is not a target.
Math and tensor shapes
Replace the first input token by Decode, then shift logits and labels exactly once. Pad labels become −100 and contribute no gradient. The pinned pretrain call uses the default mean reduction over valid targets, with label smoothing 0.1 across the vocabulary. The reference includes arbitrary synthetic token IDs, not real words or generated captions. Each sequence must contain a valid next-token target.
Logits [B, L, 16] → [B(L − 1), 16]; valid targets only
Python implementation
def language_loss(logits, tokens):
"""Shift next-token targets once; ignore suffix pads and smooth labels 0.1."""
validate_tokens(tokens)
if logits.shape != (*tokens.shape, 16) or not torch.isfinite(logits).all():
raise ValueError("finite logits must have shape [B, L, 16]")
labels = tokens[:, 1:].clone().masked_fill(tokens[:, 1:] == PAD, -100)
# The pinned pretrain call uses med.py's default mean reduction.
return F.cross_entropy(
logits[:, :-1].reshape(-1, 16),
labels.reshape(-1),
ignore_index=-100,
label_smoothing=0.1,
)
def decoder_loss(model, image, tokens):
"""Replace CLS by Decode without mutating input; hidden i predicts token i+1."""
decoder_tokens = tokens.clone()
decoder_tokens[:, 0] = DEC
logits = model.lm_head(model.decoder(decoder_tokens, image, causal=True))
return language_loss(logits, tokens)8. Update the end-to-end model with all three losses
Original paper
All three objectives contribute to the online model; the teacher is detached.
Math and tensor shapes
One online image forward supplies features for ITC, ITM, and LM. The optimizer includes the image encoder, both separate self-attentions, the shared text modules, heads, projections, and temperature. A single SGD step makes routing and image-encoder changes observable; the paper uses AdamW at scale. The teacher EMA in the next step will consume the updated online parameters.
Three scalar losses; gradients in image encoder, shared text, both SA paths, and heads
Python implementation
def train_step(model, bank, patches, tokens, optimizer, generator):
"""EMA before teacher forward, enqueue detached keys, optimize three losses."""
validate_tokens(tokens)
if len(tokens) < 2 or len(patches) != len(tokens):
raise ValueError("training needs aligned B >= 2 batches")
optimizer.zero_grad(set_to_none=True)
with torch.no_grad():
model.temperature.clamp_(0.001, 0.5)
image = model.image(
patches
) # One online image forward reused by all losses.
image_f, text_f = unimodal_features(model, image, tokens)
bank_i, bank_t, targets, teacher_i, teacher_t = bank.targets(
model, patches, tokens
)
itc, i2t, t2i = contrastive_loss(
image_f, text_f, bank_i, bank_t, targets, model.temperature
)
bank.enqueue(teacher_i, teacher_t)
negative_image = sample_negatives(t2i[:, : len(tokens)], generator)
negative_text = sample_negatives(i2t[:, : len(tokens)], generator)
itm, _ = matching_loss(model, image, tokens, negative_image, negative_text)
lm = decoder_loss(model, image, tokens)
total = itc + itm + lm
total.backward()
gradients = {
name: float(module.weight.grad.norm())
for name, module in (
("image_encoder", model.image_encoder),
("itm_head", model.itm_head),
("lm_head", model.lm_head),
)
}
optimizer.step()
return {
"itc": float(itc.detach()),
"itm": float(itm.detach()),
"lm": float(lm.detach()),
"total": float(total.detach()),
"gradients": gradients,
"negative_image": negative_image.tolist(),
"negative_text": negative_text.tolist(),
}9. Filter original and supplied generated candidates independently
Original paper
Keep each candidate iff its own binary filter predicts the matched class.
Math and tensor shapes
The random MED never invents captions or semantic filter scores. Supplied fixtures can keep an original only, a generated candidate only, both, or neither. Binary ties select class 0 and are dropped. Actual CapFilt initializes separate captioner and filter copies from MED, then fine-tunes captioner with LM and filter with ITC/ITM on COCO. They do not share parameters during that fine-tuning: the sharing ablation increased confirmation bias. Surviving pairs combine with human-annotated data to pretrain a new model; an image can lose both captions.
Original/generated captions B each; independently supplied logits [B, 2]
Python implementation
def filter_candidates(original, generated, original_scores, generated_scores):
"""Filter both candidate sets independently using supplied [unmatched, match] scores.
These are fixture logits, not semantic predictions of the random MED.
Captioner and filter are separately fine-tuned copies in actual CapFilt.
Ties predict unmatched (class 0), matching the binary-head argmax rule.
"""
if len(original) != len(generated) or not all(
isinstance(s, str) and s for s in [*original, *generated]
):
raise ValueError(
"provide one nonempty original/generated caption per image"
)
kept = []
for label, captions, scores in (
("original", original, original_scores),
("generated", generated, generated_scores),
):
if (
scores.shape != (len(captions), 2)
or not torch.isfinite(scores).all()
):
raise ValueError("candidate scores must be finite [B, 2]")
kept.extend(
(i, label, caption)
for i, caption in enumerate(captions)
if scores[i].argmax().item() == 1
)
return keptRun the same canonical source used for the highlights. Smoke JSON records its SHA-256, seed, shapes, three measured losses, image/head gradient norms, image weight change, queue pointer, detached teacher state, causal future-token check, and independent CapFilt fixture decisions. Finite losses measure mechanics, not caption quality.
blip.pyrequirements-blip.txtPinned historical pretraining sourceSharing and decoupling ablations
python -m pip install -r requirements-blip.txt python blip.py --smoke --seed 7
Bootstrapping noisy data: CapFilt
Web-crawled image-text pairs are cheap and plentiful but noisy — captions are often unrelated to the image, too generic, or entirely wrong. BLIP addresses this with CapFilt, a self-bootstrapping data pipeline built from two separately fine-tuned copies initialized from the pretrained MED model.
The Captioner is the MED decoder fine-tuned on COCO Captions to generate synthetic captions for images. It runs over web images that already have (noisy) associated text, producing a second, model-generated caption for each image.
The Filter is the MED image-text matching head fine-tuned with ITC and ITM on COCO to classify a given image-text pair as matched or not. It removes pairs judged noisy from both the original web corpus and the synthetic captions produced by the captioner.
Each original and synthetic candidate is filtered independently; either, both, or neither can survive. The captioner and filter do not share parameters during their separate COCO fine-tuning. Surviving web pairs are combined with human-annotated pairs to pretrain a new model. This uses existing COCO supervision rather than requiring new manual annotations for the web corpus.
Understanding and generation, unified
The same pretrained BLIP model can be deployed for two qualitatively different tasks through its two grounded modes, with downstream fine-tuning where appropriate:
Understanding — Image-Text Matching (ITM). Given an image and a candidate caption, the model scores how well they match. Internally it activates the image-grounded encoder (mode 2) and reads out the ITM probability from the [Encode] token. This is used for retrieval: re-rank a shortlist of candidates and return the best-scoring pair.
Generation — Image Captioning (LM). Given an image with no text, the model runs the image-grounded decoder (mode 3) autoregressively, emitting caption tokens one at a time until an end-of-sequence token is produced. The caption is grounded in the image features through cross-attention at every decoding step.
No separate model is needed for each task. The pre-training objective is designed so that modes 2 and 3 share cross-attention weights: the same cross-attention that learns to attend to image patches for matching also learns to attend for generation. Shared cross-attention and feed-forward parameters support both objectives, while separate self-attention accommodates their different encoding and decoding roles.
Why it mattered
BLIP was significant for two reasons. First, it unified understanding and generation in a flexible pre-training framework with partially shared weights — CLIP excels at understanding but cannot generate; pure captioning models generate but cannot score. BLIP demonstrated that a single MED architecture can do both, and do both well.
Second, CapFilt showed that the model's own outputs can be used to improve the quality of its training data. The captioner and filter are derived from the same model being trained, then fine-tuned separately on COCO, creating a bootstrapping loop that reuses existing annotations. This principle — using model outputs to curate training data — has become a standard technique in modern multimodal training pipelines.
BLIP set the stage for BLIP-2, which replaced the end-to-end MED pre-training with a frozen ViT backbone bridged to a frozen LLM via a lightweight Q-Former, dramatically reducing training cost while inheriting BLIP's multi-objective training strategy.
Related Reading
- BLIP-2: Efficient Vision-Language Pre-training — the direct successor that freezes BLIP's backbone and bridges it to a large language model with a Q-Former, reducing trainable parameters by an order of magnitude
- CLIP: Learning Transferable Visual Models From Natural Language Supervision — the contrastive pre-training framework BLIP extends with a generation objective and a data-cleaning pipeline
- CoCa — a contemporaneous model that also adds a captioning loss on top of contrastive learning, arriving at a similar unification from a different architecture
