Skip to main content

BLIP: Bootstrapping Language-Image Pre-training

How BLIP unifies vision-language understanding and generation in one model, and bootstraps noisy web data with CapFilt — a captioner that synthesizes captions and a filter that removes noisy image-text pairs.

TL;DR

  • BLIP introduces a Multimodal mixture of Encoder-Decoder (MED) that shares weights across three modes — unimodal encoder (ITC), image-grounded text encoder (ITM), and image-grounded text decoder (LM) — enabling a single pre-trained model to both score and generate.
  • BLIP learns understanding and generation jointly, sharing embeddings, cross-attention, and feed-forward layers while keeping encoder and decoder self-attention separate.
  • CapFilt bootstraps noisy web data: a captioner synthesizes fresh captions for web images; a filter removes noisy pairs from both the originals and the synthetic set; the cleaned, augmented corpus retrains the model.
  • BLIP was the direct predecessor of BLIP-2, which replaced BLIP's end-to-end training with a frozen-backbone Q-Former.

Why bootstrap the data?

Web image-text pairs are plentiful but noisy. BLIP's CapFilt pipeline creates new captions and filters both the original and generated candidates. To see how those two modules arise, first follow MED's three modes and their joint objectives.

One model, three modes: MED

BLIP's core architecture is the Multimodal mixture of Encoder-Decoder (MED). Its trainable image encoder (ViT) and partially shared text encoder/decoder are optimized end to end. The text model is activated in one of three modes by switching which attention layers are enabled:

Unimodal encoder (ITC loss). Self-attention only; no cross-attention to the image. A [CLS] token aggregates the text. The output is compared to the image's [CLS] token via a contrastive loss, pulling matched image-text pairs together and pushing unmatched pairs apart in the embedding space. This mode learns a contrastive representation.

Image-grounded text encoder (ITM loss). Bidirectional self-attention plus cross-attention to the image encoder's patch features. An [Encode] token produces a multimodal embedding that is fed to a binary classifier predicting whether the image and text are matched. Hard negative mining samples unmatched pairs in proportion to their contrastive similarity, after excluding the paired positive.

Image-grounded text decoder (LM loss). Causal (masked) self-attention plus cross-attention to the image. The model autoregressively predicts caption tokens conditioned on the image, supervised by a language modeling loss. This is the mode used for caption generation at inference.

The unimodal and grounded encoders share their text stack. The decoder shares embeddings, cross-attention, and feed-forward parameters with the encoder, but has separate self-attention parameters. Encoder attention is bidirectional; decoder attention is causal. The joint objective adds ITC, ITM, and LM. Original ITC includes momentum-encoder soft targets and a feature queue; LM shifts next-token targets, ignores pads, and applies label smoothing of 0.1.

Implementation

The three modes differ in both parameter routing and available context. The executable reference makes those distinctions visible before following the pretrained model into data curation and downstream tasks.

Offline seeded CPU mechanics: one width-eight text block, a trainable linear patch encoder, four-dimensional ITC features, and a local eight-slot momentum queue. There are no pretrained weights, BERT tokenizer, learned caption semantics, GPU collectives, or benchmark reproduction. Momentum distillation is included at this reduced scale; distributed all-gather and the original ViT/BERT networks are omitted. All equations below are authored explanations of the cited operations.

1. Share embeddings, cross-attention, and FFN

Original paper

Paper excerpt · PDF p. 3Official SA-excluded encoder/decoder tying; §3.2, Figure 2

The release excludes /attention from encoder/decoder parameter tying.

Math and tensor shapes

E_{enc}=E_{dec},\quad CA_{enc}=CA_{dec}
FFN_{enc}=FFN_{dec}
SA_{enc}\ne SA_{dec}

The two paths reference the same SharedText object, while each owns separate self-attention parameters and its associated LayerNorm. Unimodal and grounded encoding use the same encoder. The linear image encoder remains trainable: original BLIP is end-to-end. One block and mean patch pooling replace BERT and ViT CLS readout here.

Tokens [B, L] → hidden [B, L, 8]; patches [B, P, 6] → [B, P, 8]

Python implementation

1. Share embeddings, cross-attention, and FFN
python
class SharedText(nn.Module):
    """Embeddings, cross-attention and FFN shared by encoder and decoder."""

    def __init__(self, width=8, vocab=16):
        super().__init__()
        self.embedding = nn.Embedding(vocab, width, padding_idx=PAD)
        self.position = nn.Embedding(8, width)
        self.cross = nn.MultiheadAttention(width, 2, batch_first=True)
        self.cross_norm = nn.LayerNorm(width)
        self.ffn = nn.Sequential(
            nn.Linear(width, 16), nn.GELU(), nn.Linear(16, width)
        )
        self.ffn_norm = nn.LayerNorm(width)


class TextPath(nn.Module):
    """Each path owns separate self-attention and its associated LayerNorm."""

    def __init__(self, shared, width=8):
        super().__init__()
        self.shared = shared
        self.self_attention = nn.MultiheadAttention(width, 2, batch_first=True)
        self.self_norm = nn.LayerNorm(width)

    def forward(self, tokens, image=None, causal=False):
        validate_tokens(tokens)
        shared = self.shared
        x = shared.embedding(tokens) + shared.position(
            torch.arange(tokens.shape[1])
        )
        blocked = attention_mask(tokens.shape[1], causal)
        attended = self.self_attention(
            x,
            x,
            x,
            attn_mask=blocked,
            key_padding_mask=tokens == PAD,
            need_weights=False,
        )[0]
        x = self.self_norm(x + attended)
        if image is not None:
            x = shared.cross_norm(
                x + shared.cross(x, image, image, need_weights=False)[0]
            )
        return shared.ffn_norm(x + shared.ffn(x))


class MED(nn.Module):
    """Tiny trainable image/text model preserving the three parameter routes."""

    def __init__(self):
        super().__init__()
        shared = SharedText()
        self.encoder = TextPath(shared)
        self.decoder = TextPath(shared)
        self.image_encoder = nn.Linear(6, 8)
        self.image_projection = nn.Linear(8, 4)
        self.text_projection = nn.Linear(8, 4)
        self.itm_head = nn.Linear(8, 2)
        self.lm_head = nn.Linear(8, 16)
        self.temperature = nn.Parameter(torch.tensor(0.07))

    def image(self, patches):
        if patches.ndim != 3 or patches.shape[1] < 1 or patches.shape[2] != 6:
            raise ValueError("patches must have shape [B, P >= 1, 6]")
        if not torch.isfinite(patches).all():
            raise ValueError("patches must be finite")
        return self.image_encoder(patches)

2. Route unimodal, grounded, and causal paths

Original paper

Paper excerpt · PDF p. 3§3.1 · three MED modes; Figure 2

Image conditioning and future-token visibility are separate switches.

Math and tensor shapes

M_{ij}^{dec}=\mathbf1[j>i]
h_u=Enc(T),\quad h_g=Enc(T,I)
h_d=Dec(T,I;M^{dec})

True blocks attention in the PyTorch mask. Both encoders are bidirectional; only the grounded encoder receives image keys/values. The decoder uses its separate self-attention with a causal mask and shared cross-attention. Pads are excluded as keys. Start tokens are CLS for ITC, Encode for ITM, and Decode for LM. Image perturbation cannot change unimodal text features; future text cannot affect earlier decoder positions.

Block mask [L, L]; suffix padding key mask [B, L]

Python implementation

2. Route unimodal, grounded, and causal paths
python
def attention_mask(length, causal):
    """True blocks a key; decoder position i cannot attend to j > i."""
    if not isinstance(length, int) or length < 1:
        raise ValueError("length must be a positive integer")
    return (
        torch.ones(length, length, dtype=torch.bool).triu(1) if causal else None
    )

3. Align normalized unimodal features

Original paper

Paper excerpt · PDF p. 3Official normalized features and symmetric ITC; §3.2

Current online queries compare with detached momentum keys and a queue.

Math and tensor shapes

s^{i2t}_{ij}=\bar v_i^\top\bar t_j/\tau
\ell(s,p)=-\sum_jp_j\log\operatorname{softmax}(s)_j
\mathcal L_{ITC}=\tfrac12(\ell_{i2t}+\ell_{t2i})

Bars denote L2 normalization over the four feature dimensions. Each directional loss averages its query rows. Temperature is trainable and clamped to [0.001, 0.5] before the forward. This is a reduced soft-label ITC objective, rather than labeling hard batch-only contrastive CE as the full original loss.

Features [B, 4]; logits and targets [B, B + K]; smoke [3, 11]

Python implementation

3. Align normalized unimodal features
python
def unimodal_features(model, image, tokens):
    """Text receives no image, and both projected representations are normalized."""
    text = model.encoder(tokens)
    # Mean patches replace the full ViT's learned CLS token in this reference.
    image_features = F.normalize(model.image_projection(image.mean(1)), dim=-1)
    text_features = F.normalize(model.text_projection(text[:, 0]), dim=-1)
    return image_features, text_features


def contrastive_loss(image, text, bank_image, bank_text, targets, temperature):
    """Symmetric soft-label ITC against detached momentum current/queue keys."""
    i2t = image @ bank_text.T / temperature
    t2i = text @ bank_image.T / temperature
    losses = [
        -(target * F.log_softmax(logits, dim=1)).sum(1).mean()
        for logits, target in zip((i2t, t2i), targets)
    ]
    return (losses[0] + losses[1]) / 2, i2t, t2i

4. Detach momentum targets and queued keys

Original paper

Paper excerpt · PDF p. 3Official soft-label targets; EMA and enqueue at lines 225–247

EMA happens before the teacher forward; detached keys enter a FIFO queue.

Math and tensor shapes

\theta_m\leftarrow m\theta_m+(1-m)\theta
p=(1-\alpha)y+\alpha\operatorname{softmax}(s_m)
y_{ij}=\mathbf1[i=j]

A copied teacher has no gradients. Its current image/text features precede cloned queue contents in the bank. With alpha = 0.4 and momentum = 0.995, each row mixes an identity positive with teacher probabilities. Enqueue follows ITC construction and precedes ITM/LM, as in the release. The local ring supports arbitrary batch sizes; it omits distributed all-gather and the upstream global-batch divisibility constraint. This reduced teacher copies the entire tiny MED, including unused heads; original EMA covers only the ITC encoder/projection pairs.

Frozen teacher; queue [K, 4], K = 8; target [B, B + K]

Python implementation

4. Detach momentum targets and queued keys
python
class MomentumBank:
    """Frozen EMA teacher and a local FIFO of normalized paired ITC features."""

    def __init__(self, model, capacity=8):
        if not isinstance(capacity, int) or capacity < 1:
            raise ValueError("capacity must be positive")
        self.teacher = copy.deepcopy(model).requires_grad_(False)
        self.images = F.normalize(torch.randn(capacity, 4), dim=-1)
        self.texts = F.normalize(torch.randn(capacity, 4), dim=-1)
        self.pointer = 0

    @torch.no_grad()
    def targets(self, model, patches, tokens, alpha=0.4, momentum=0.995):
        if not 0 <= alpha <= 1 or not 0 <= momentum <= 1:
            raise ValueError("alpha and momentum must be in [0, 1]")
        for teacher, online in zip(
            self.teacher.parameters(), model.parameters()
        ):
            teacher.mul_(momentum).add_(online, alpha=1 - momentum)
        image, text = unimodal_features(
            self.teacher, self.teacher.image(patches), tokens
        )
        bank_image = torch.cat((image, self.images.clone()))
        bank_text = torch.cat((text, self.texts.clone()))
        hard = torch.zeros(image.shape[0], bank_text.shape[0])
        hard[:, : image.shape[0]].fill_diagonal_(1)
        soft_i = (image @ bank_text.T / model.temperature).softmax(1)
        soft_t = (text @ bank_image.T / model.temperature).softmax(1)
        targets = (
            alpha * soft_i + (1 - alpha) * hard,
            alpha * soft_t + (1 - alpha) * hard,
        )
        return bank_image, bank_text, targets, image, text

    @torch.no_grad()
    def enqueue(self, images, texts):
        # Serial local ring supports any batch size; upstream uses all-gather
        # and requires queue capacity divisible by the global batch size.
        for image, text in zip(images, texts):
            self.images[self.pointer] = image.detach()
            self.texts[self.pointer] = text.detach()
            self.pointer = (self.pointer + 1) % len(self.images)

5. Sample confusing non-positive pairs

Original paper

Paper excerpt · PDF p. 3Official in-batch sampling weights and multinomial draws

Select negative images from text queries and negative texts from image queries.

Math and tensor shapes

w_{ij}=\operatorname{softmax}(s_i)_j+10^{-4}
w_{ii}=0,\quad j\sim\operatorname{Categorical}(w_i)

Sampling excludes the paired diagonal after adding the probability floor, which keeps alternative candidates sampleable even when the positive dominates numerically. These draws run without gradients using a seeded local generator. This reference assumes unique pairs within a batch; the diagonal is the declared positive set. B = 1 is rejected because no non-positive remains.

Current-batch logits [B, B] → negative indices [B], B ≥ 2

Python implementation

5. Sample confusing non-positive pairs
python
def sample_negatives(similarity, generator):
    """Sample each row's non-positive with upstream's 1e-4 probability floor."""
    if similarity.ndim != 2 or similarity.shape[0] != similarity.shape[1]:
        raise ValueError("similarity must be square [B, B]")
    if similarity.shape[0] < 2 or not torch.isfinite(similarity).all():
        raise ValueError("hard-negative mining needs finite logits and B >= 2")
    with torch.no_grad():
        weights = similarity.softmax(1) + 1e-4
        weights.fill_diagonal_(0)
        return torch.multinomial(weights, 1, generator=generator).squeeze(1)

6. Classify matched and grounded hard negatives

Original paper

Paper excerpt · PDF p. 3Official grounded encoder and binary ITM head

One positive batch and two negative batches supervise a grounded binary head.

Math and tensor shapes

z=W_{ITM}h_{Encode}
\mathcal L_{ITM}=CE(z,y)

Matched image/text pairs have label 1; mismatched image and mismatched text batches have label 0. Encode replaces CLS in a clone. Bidirectional self-attention and image cross-attention build the pair representation. This is distinct from a cosine similarity score: it is the joint multimodal encoder output that feeds the binary head.

Grounded hidden [3B, L, 8] → Encode logits [3B, 2]

Python implementation

6. Classify matched and grounded hard negatives
python
def matching_loss(model, image, tokens, negative_image, negative_text):
    """One matched and two unmatched batches, read at the Encode token."""
    grounded_tokens = tokens.clone()
    grounded_tokens[:, 0] = ENC
    all_tokens = torch.cat(
        (grounded_tokens, grounded_tokens, grounded_tokens[negative_text])
    )
    all_images = torch.cat((image, image[negative_image], image))
    hidden = model.encoder(all_tokens, all_images)
    logits = model.itm_head(hidden[:, 0])
    labels = torch.cat(
        (torch.ones(len(tokens)), torch.zeros(2 * len(tokens)))
    ).long()
    return F.cross_entropy(logits, labels), logits

7. Predict the next token and exclude pads

Original paper

Paper excerpt · PDF p. 3Official single target shift, pad ignore, and 0.1 label smoothing

Decoder logits at i supervise token i + 1; the start token is not a target.

Math and tensor shapes

z_{:,0:L-1}\leftrightarrow T_{:,1:L}
q=(1-\epsilon)\operatorname{onehot}(t)+\epsilon/V
\epsilon=0.1,\quad \mathcal L_{LM}=CE(z,q)

Replace the first input token by Decode, then shift logits and labels exactly once. Pad labels become −100 and contribute no gradient. The pinned pretrain call uses the default mean reduction over valid targets, with label smoothing 0.1 across the vocabulary. The reference includes arbitrary synthetic token IDs, not real words or generated captions. Each sequence must contain a valid next-token target.

Logits [B, L, 16] → [B(L − 1), 16]; valid targets only

Python implementation

7. Predict the next token and exclude pads
python
def language_loss(logits, tokens):
    """Shift next-token targets once; ignore suffix pads and smooth labels 0.1."""
    validate_tokens(tokens)
    if logits.shape != (*tokens.shape, 16) or not torch.isfinite(logits).all():
        raise ValueError("finite logits must have shape [B, L, 16]")
    labels = tokens[:, 1:].clone().masked_fill(tokens[:, 1:] == PAD, -100)
    # The pinned pretrain call uses med.py's default mean reduction.
    return F.cross_entropy(
        logits[:, :-1].reshape(-1, 16),
        labels.reshape(-1),
        ignore_index=-100,
        label_smoothing=0.1,
    )


def decoder_loss(model, image, tokens):
    """Replace CLS by Decode without mutating input; hidden i predicts token i+1."""
    decoder_tokens = tokens.clone()
    decoder_tokens[:, 0] = DEC
    logits = model.lm_head(model.decoder(decoder_tokens, image, causal=True))
    return language_loss(logits, tokens)

8. Update the end-to-end model with all three losses

Original paper

Paper excerpt · PDF p. 3§3.2 · joint objectives and shared image forward

All three objectives contribute to the online model; the teacher is detached.

Math and tensor shapes

\mathcal L=\mathcal L_{ITC}+\mathcal L_{ITM}+\mathcal L_{LM}
\theta^+=\theta-\eta\nabla_\theta\mathcal L

One online image forward supplies features for ITC, ITM, and LM. The optimizer includes the image encoder, both separate self-attentions, the shared text modules, heads, projections, and temperature. A single SGD step makes routing and image-encoder changes observable; the paper uses AdamW at scale. The teacher EMA in the next step will consume the updated online parameters.

Three scalar losses; gradients in image encoder, shared text, both SA paths, and heads

Python implementation

8. Update the end-to-end model with all three losses
python
def train_step(model, bank, patches, tokens, optimizer, generator):
    """EMA before teacher forward, enqueue detached keys, optimize three losses."""
    validate_tokens(tokens)
    if len(tokens) < 2 or len(patches) != len(tokens):
        raise ValueError("training needs aligned B >= 2 batches")
    optimizer.zero_grad(set_to_none=True)
    with torch.no_grad():
        model.temperature.clamp_(0.001, 0.5)
    image = model.image(
        patches
    )  # One online image forward reused by all losses.
    image_f, text_f = unimodal_features(model, image, tokens)
    bank_i, bank_t, targets, teacher_i, teacher_t = bank.targets(
        model, patches, tokens
    )
    itc, i2t, t2i = contrastive_loss(
        image_f, text_f, bank_i, bank_t, targets, model.temperature
    )
    bank.enqueue(teacher_i, teacher_t)
    negative_image = sample_negatives(t2i[:, : len(tokens)], generator)
    negative_text = sample_negatives(i2t[:, : len(tokens)], generator)
    itm, _ = matching_loss(model, image, tokens, negative_image, negative_text)
    lm = decoder_loss(model, image, tokens)
    total = itc + itm + lm
    total.backward()
    gradients = {
        name: float(module.weight.grad.norm())
        for name, module in (
            ("image_encoder", model.image_encoder),
            ("itm_head", model.itm_head),
            ("lm_head", model.lm_head),
        )
    }
    optimizer.step()
    return {
        "itc": float(itc.detach()),
        "itm": float(itm.detach()),
        "lm": float(lm.detach()),
        "total": float(total.detach()),
        "gradients": gradients,
        "negative_image": negative_image.tolist(),
        "negative_text": negative_text.tolist(),
    }

9. Filter original and supplied generated candidates independently

Original paper

Paper excerpt · PDF p. 4§3.3 · CapFilt; §4.4 · separately fine-tuned modules

Keep each candidate iff its own binary filter predicts the matched class.

Math and tensor shapes

k(T)=\mathbf1[\arg\max z(I,T)=1]
D_{new}=D_h\cup D_w^{kept}\cup D_s^{kept}

The random MED never invents captions or semantic filter scores. Supplied fixtures can keep an original only, a generated candidate only, both, or neither. Binary ties select class 0 and are dropped. Actual CapFilt initializes separate captioner and filter copies from MED, then fine-tunes captioner with LM and filter with ITC/ITM on COCO. They do not share parameters during that fine-tuning: the sharing ablation increased confirmation bias. Surviving pairs combine with human-annotated data to pretrain a new model; an image can lose both captions.

Original/generated captions B each; independently supplied logits [B, 2]

Python implementation

9. Filter original and supplied generated candidates independently
python
def filter_candidates(original, generated, original_scores, generated_scores):
    """Filter both candidate sets independently using supplied [unmatched, match] scores.

    These are fixture logits, not semantic predictions of the random MED.
    Captioner and filter are separately fine-tuned copies in actual CapFilt.
    Ties predict unmatched (class 0), matching the binary-head argmax rule.
    """
    if len(original) != len(generated) or not all(
        isinstance(s, str) and s for s in [*original, *generated]
    ):
        raise ValueError(
            "provide one nonempty original/generated caption per image"
        )
    kept = []
    for label, captions, scores in (
        ("original", original, original_scores),
        ("generated", generated, generated_scores),
    ):
        if (
            scores.shape != (len(captions), 2)
            or not torch.isfinite(scores).all()
        ):
            raise ValueError("candidate scores must be finite [B, 2]")
        kept.extend(
            (i, label, caption)
            for i, caption in enumerate(captions)
            if scores[i].argmax().item() == 1
        )
    return kept

Run the same canonical source used for the highlights. Smoke JSON records its SHA-256, seed, shapes, three measured losses, image/head gradient norms, image weight change, queue pointer, detached teacher state, causal future-token check, and independent CapFilt fixture decisions. Finite losses measure mechanics, not caption quality.

blip.pyrequirements-blip.txtPinned historical pretraining sourceSharing and decoupling ablations

python -m pip install -r requirements-blip.txt
python blip.py --smoke --seed 7

Bootstrapping noisy data: CapFilt

Web-crawled image-text pairs are cheap and plentiful but noisy — captions are often unrelated to the image, too generic, or entirely wrong. BLIP addresses this with CapFilt, a self-bootstrapping data pipeline built from two separately fine-tuned copies initialized from the pretrained MED model.

The Captioner is the MED decoder fine-tuned on COCO Captions to generate synthetic captions for images. It runs over web images that already have (noisy) associated text, producing a second, model-generated caption for each image.

The Filter is the MED image-text matching head fine-tuned with ITC and ITM on COCO to classify a given image-text pair as matched or not. It removes pairs judged noisy from both the original web corpus and the synthetic captions produced by the captioner.

Each original and synthetic candidate is filtered independently; either, both, or neither can survive. The captioner and filter do not share parameters during their separate COCO fine-tuning. Surviving web pairs are combined with human-annotated pairs to pretrain a new model. This uses existing COCO supervision rather than requiring new manual annotations for the web corpus.

Understanding and generation, unified

The same pretrained BLIP model can be deployed for two qualitatively different tasks through its two grounded modes, with downstream fine-tuning where appropriate:

Understanding — Image-Text Matching (ITM). Given an image and a candidate caption, the model scores how well they match. Internally it activates the image-grounded encoder (mode 2) and reads out the ITM probability from the [Encode] token. This is used for retrieval: re-rank a shortlist of candidates and return the best-scoring pair.

Generation — Image Captioning (LM). Given an image with no text, the model runs the image-grounded decoder (mode 3) autoregressively, emitting caption tokens one at a time until an end-of-sequence token is produced. The caption is grounded in the image features through cross-attention at every decoding step.

No separate model is needed for each task. The pre-training objective is designed so that modes 2 and 3 share cross-attention weights: the same cross-attention that learns to attend to image patches for matching also learns to attend for generation. Shared cross-attention and feed-forward parameters support both objectives, while separate self-attention accommodates their different encoding and decoding roles.

Why it mattered

BLIP was significant for two reasons. First, it unified understanding and generation in a flexible pre-training framework with partially shared weights — CLIP excels at understanding but cannot generate; pure captioning models generate but cannot score. BLIP demonstrated that a single MED architecture can do both, and do both well.

Second, CapFilt showed that the model's own outputs can be used to improve the quality of its training data. The captioner and filter are derived from the same model being trained, then fine-tuned separately on COCO, creating a bootstrapping loop that reuses existing annotations. This principle — using model outputs to curate training data — has become a standard technique in modern multimodal training pipelines.

BLIP set the stage for BLIP-2, which replaced the end-to-end MED pre-training with a frozen ViT backbone bridged to a frozen LLM via a lightweight Q-Former, dramatically reducing training cost while inheriting BLIP's multi-objective training strategy.

If you found this paper review helpful, consider sharing it with others.

Mastodon