Flamingo: a Visual Language Model for Few-Shot Learning

TL;DR

• Flamingo bridges a frozen NFNet vision encoder and a frozen Chinchilla language model by training two bridging components, a Perceiver Resampler and gated cross-attention layers inserted between frozen LM blocks, together with newly added token embeddings.
• The Perceiver Resampler compresses a variable number of image/video feature tokens into 64 latents per visual input, decoupling its output resolution and frame count from the size of each visual representation.
• Gated cross-attention layers (GATED XATTN-DENSE) use separate zero-initialized tanh gates for cross-attention and dense residuals so the augmented model starts identical to the pretrained LM and learns how much each new branch contributes.
• Interleaved image-text sequences with per-image attention masking enable few-shot in-context learning: prepend a handful of image-question-answer examples and Flamingo answers a new query image without any weight updates.

Frozen backbones, learned bridges

Training a vision-language model from scratch requires enormous data and compute. Flamingo takes a different path: keep two powerful pretrained models — a large vision encoder and a large language model — frozen in their original parameters, and train the lightweight components that connect them together with newly added token embeddings. The vision encoder is a Normalizer-Free ResNet (NFNet) pretrained with a contrastive objective on a large image-text dataset. The language model is Chinchilla (or its predecessors, depending on Flamingo variant size), pretrained purely on text.

The core challenge is that image features are spatially dense — a high-resolution image may produce hundreds or thousands of feature vectors — while text-to-vision cross-attention becomes more expensive with more visual keys. Flamingo solves this with the Perceiver Resampler: a small transformer module that holds a fixed set of 64 learned latent queries. Each image or video is processed independently, producing exactly 64 visual tokens regardless of its resolution or frame count. The learned latents provide queries and also join the visual features in the keys and values. With M visual inputs, the resampler produces M × 64 tokens; the attention mask selects which input each text position can read.

Because neither backbone is ever updated, Flamingo's training is efficient: the Perceiver Resampler, the new cross-attention layers and newly added token embeddings need parameter gradients. The reduced reference below uses a fixed synthetic vocabulary and explicitly omits this added-vocabulary training. The frozen LM retains all its language capabilities, and the frozen vision encoder retains all its visual representations. The learned bridges act as a translator between the two modalities.

Gated cross-attention

Getting visual information into the language model requires injecting new cross-attention layers, since the frozen LM's self-attention layers were never designed to attend to visual tokens. Flamingo inserts new GATED XATTN-DENSE blocks between every frozen LM layer (or a subset of layers). Each new block performs cross-attention between the current text hidden states (queries) and the 64 visual latents from the Perceiver Resampler (keys and values), followed by a dense feed-forward network.

The critical design detail is separate tanh gates. Each GATED XATTN-DENSE block multiplies its cross-attention residual by tanh(α_xattn) and its following dense residual by tanh(α_dense). Both learned scalars are initialized to zero. At initialization, tanh(0) = 0, so both new residuals vanish and the model is mathematically equivalent to the original pretrained LM. The paper's ablation found that this initialization improves stability. Gate gradients can be nonzero immediately, while the new branch weights receive zero gradients at exact zero gates.

As training proceeds, the gates can move away from zero in either direction, allowing each new branch to contribute to the residual stream. Different blocks can learn different contributions at different depths. The frozen self-attention and feed-forward parameters in the original LM are never updated, but its forward remains differentiable so the loss can train the bridges.

Implementation

The bridge is easiest to verify by following one interleaved sequence through the resampler, its media mask, two gated residuals, and a frozen text block. The reduced reference below isolates those mechanics on supplied tensors. Its zero-gate and masking checks establish what information can flow; they do not generate answers or reproduce pretrained few-shot behavior.

def __init__(self, width=8, num_latents=4, max_frames=4):
    super().__init__()
    if num_latents < 1 or max_frames < 1:
        raise ValueError("latent and frame counts must be positive")
    self.latents = nn.Parameter(torch.randn(num_latents, width) * 0.02)
    self.temporal = nn.Parameter(torch.randn(max_frames, width) * 0.02)
    self.feature_norm = nn.LayerNorm(width)
    self.latent_norm = nn.LayerNorm(width)
    self.attention = Attention(width)
    self.dense = feed_forward(width)

def forward(self, features):
    """Resample [B, M, F, S, D] to [B, M, L, D], separately per media."""
    batch, media, frames, spatial, width = features.shape
    if frames < 1 or spatial < 1 or frames > self.temporal.shape[0]:
        raise ValueError("require nonempty features within max_frames")
    if media == 0:
        return features.new_empty(batch, 0, self.latents.shape[0], width)
    x = features + self.temporal[:frames][None, None, :, None, :]
    x = self.feature_norm(
        x.reshape(batch * media, frames * spatial, width)
    )
    z = self.latents[None].expand(batch * media, -1, -1)
    context = torch.cat((x, self.latent_norm(z)), dim=1)
    z = z + self.attention(self.latent_norm(z), context)
    z = z + self.dense(z)
    return z.reshape(batch, media, self.latents.shape[0], width)

def media_mask(tokens, num_media, num_latents):
    """Permit only the last image/video marker, including its marker position.

    IMAGE stands for a supplied visual input inserted at that position; 0 in
    phi denotes text before any image. Each input occupies L keys/values.
    """
    if tokens.ndim != 2 or tokens.dtype != torch.long:
        raise ValueError("tokens must be a rank-two int64 tensor")
    if num_media < 0 or num_latents < 1:
        raise ValueError("invalid media or latent count")
    phi = tokens.eq(IMAGE).long().cumsum(dim=1)
    if torch.any(phi > num_media):
        raise ValueError("more image markers than supplied visual inputs")
    key_media = torch.arange(1, num_media + 1, device=tokens.device)
    key_media = key_media.repeat_interleave(num_latents)
    return (phi[:, :, None] == key_media) & (phi[:, :, None] > 0)

def cross_attention(self, text, visual, allowed):
    """Keep pre-image rows visually unchanged, even when the gate opens."""
    batch, media, latents, width = visual.shape
    context = visual.reshape(batch, media * latents, width)
    delta = self.attention(self.norm(text), context, allowed)
    return text + self.alpha_xattn.tanh() * delta

def forward(self, text, visual, allowed):
    """Apply cross-attention, then a separately gated dense residual."""
    text = self.cross_attention(text, visual, allowed)
    return text + self.alpha_dense.tanh() * self.dense(text)

def forward(self, features, tokens):
    """Keep backbone weights frozen, while differentiating through the LM."""
    self.validate(features, tokens)
    allowed = media_mask(
        tokens, features.shape[1], self.resampler.latents.shape[0]
    )
    with torch.no_grad():
        encoded = self.vision(features)
    visual = self.resampler(encoded)
    hidden = self.bridge(self.text_input(tokens), visual, allowed)
    # Do not use no_grad here: loss gradients must reach the bridge gates.
    logits = self.vocabulary(self.text_block(hidden, tokens))
    labels = tokens[:, 1:].clone()
    labels[(labels == PAD) | (labels == IMAGE)] = -100
    nll = F.cross_entropy(
        logits[:, :-1].transpose(1, 2),
        labels,
        ignore_index=-100,
        reduction="none",
    )
    loss = nll.sum(dim=1).mean()
    return {
        "logits": logits,
        "visual": visual,
        "allowed": allowed,
        "loss": loss,
    }

python -m pip install -r requirements-flamingo.txt
python flamingo.py --smoke --seed 7

With the bridge and mask in place, the next step is to change the context rather than the weights. That is where the original pretrained model's few-shot capability enters.

Few-shot in-context learning

The most distinctive capability Flamingo inherits from its frozen LM backbone is in-context learning — the ability to adapt to a new task from a few examples in the prompt, without any gradient updates. Flamingo extends this to the multimodal setting by training on interleaved sequences of images and text drawn from web pages and image-caption datasets.

During training, each sequence consists of interleaved image and text segments. An <image> marker in the text records where a visual input appears; its 64 latents remain separate cross-attention keys and values, rather than being inserted as ordinary LM tokens. The key architectural detail is per-image attention masking: each text token directly cross-attends only to the image or video immediately preceding it. Earlier visual inputs can still influence the token indirectly through causal text self-attention, while future images remain inaccessible.

At inference, few-shot prompting prepends (image, question, answer) demonstrations before the query image and question. The mask grounds each segment's direct visual attention to its own input, and text self-attention carries earlier demonstrations forward. The paper trained with up to five images per interleaved sequence and evaluated up to 32 shots without task-specific weight updates. Its reported zero-shot setup supplied two text-only examples without their images; the schematic below uses an empty demonstration context for illustration. Few-shot adaptation changes the prompt, not the optimizer state.

Why it mattered

Flamingo demonstrated that a frozen-backbone bridging strategy could produce a competitive general-purpose vision-language model. Before Flamingo, the dominant approach was to jointly train vision and language components end-to-end, which was expensive and destroyed the knowledge already encoded in pretrained models. Flamingo showed that the two modalities could be connected cheaply via learned bridges while preserving both backbones intact.

The gated cross-attention design became an influential template: the gate-to-zero initialization trick appears in later models as a standard technique for safely augmenting pretrained transformers. The Perceiver Resampler's idea of compressing variable-length visual features to a fixed token budget is directly echoed in BLIP-2's Q-Former. And Flamingo's interleaved sequence format, with per-image masking, became the standard way to think about multimodal context windows.

LLaVA later showed that instruction tuning on a curated set of visual dialogues — rather than web-scale interleaved sequences — could produce strong multimodal chat and instruction-following behavior with far less data. It kept part of Flamingo's recipe: its vision encoder stays frozen throughout, and its first stage trains only a projection between a frozen encoder and a frozen LLM, but its second stage fine-tunes the LLM together with the projection. Flamingo was the proof of concept that made this bridging direction credible.

Related Reading

• BLIP-2 — bridges a frozen image encoder and a frozen LLM with a Q-Former, a direct descendant of Flamingo's Perceiver Resampler idea, adding a two-stage training procedure for more efficient alignment
• LLaVA: Visual Instruction Tuning — shows that instruction-tuning a vision-language connector on GPT-4-generated visual dialogues achieves strong multimodal reasoning with far less data than Flamingo's web-scale interleaved training
• CLIP — the contrastive vision-language pretraining approach that inspired Flamingo's vision encoder and demonstrated that large-scale image-text pairing produces powerful transferable visual representations
• Attention Is All You Need — the transformer architecture that underlies every component in Flamingo except its NFNet vision encoder: the Perceiver Resampler cross-attention, the gated cross-attention layers, and the frozen LM backbone
• Vision-Language Alignment — compares Flamingo’s gated cross-attention with CLIP, BLIP-2 and LLaVA, and what each trains or freezes