Skip to main content

Multi-Vector Late Interaction

Explore ColBERT and other multi-vector retrieval models that use fine-grained token-level matching for superior search quality.

A bi-encoder crushes a whole document into one vector — fast to index, but a long or many-faceted passage gets averaged into mush. Multi-vector models like ColBERT keep one vector per token and defer the matching to query time. The result sits between a bi-encoder and a cross-encoder: nearly the precision of joint attention, while staying precomputable and searchable.

Interactive ColBERT Visualization

The visualization below is real: every token is a genuine GloVe vector, and the matrix is real cosine similarity. Each query token's strongest match (the ringed cell) is the "max" in MaxSim. Flip to the single-vector view to watch all that detail collapse into one number — and try the long, padded document to see pooling fail where MaxSim holds.

Single vector vs late interaction

Helpful background

Pooling strategies

Compare ways to turn token vectors into one embedding, including padding masks and matching the model’s training setup.

Open full page

A bi-encoder pools each side into one vector and takes a single cosine — every token's contribution is averaged in, so a few relevant terms can be drowned by filler:

s_{\text{single}}(q, d) = \cos\!\big(\bar{E}_q,\; \bar{E}_d\big), \qquad \bar{E}_x = \tfrac{1}{|x|}\textstyle\sum_i E_{x_i}

ColBERT keeps every token vector and scores with MaxSim: each query token takes its best match across the document, and those maxima are summed.

S_{q,d} = \sum_{i \in |q|} \max_{j \in |d|}\; E_{q_i} \cdot E_{d_j}^{\top}

where E_{q_i} is the embedding of query token i and E_{d_j} of document token j. The max is the key: a single strong match counts fully, no matter how long the document — so MaxSim never dilutes.

python
def bi_encoder_score(q_tokens, d_tokens):        # single vector
    q = normalize(mean_pool(encode(q_tokens)))   # one vector
    d = normalize(mean_pool(encode(d_tokens)))   # one vector
    return q @ d                                  # one cosine

def colbert_score(q_tokens, d_tokens):           # late interaction
    Q = normalize(encode(q_tokens))              # one vector per query token
    D = normalize(encode(d_tokens))              # one vector per doc token
    sim = Q @ D.T                                # |q| x |d| matrix
    return sim.max(dim=1).values.sum()           # MaxSim: best per row, summed

The ColBERT encoder

The architecture is deliberately light — a frozen-ish BERT, a linear projection to a small dimension (128), and L2 normalization. The trick is what it doesn't do: it never pools. Query and document each keep a full bag of token vectors.

python
class ColBERT(nn.Module):
    def __init__(self, bert, dim=128):
        super().__init__()
        self.bert, self.proj = bert, nn.Linear(768, dim)

    def encode(self, tokens):
        h = self.proj(self.bert(tokens).last_hidden_state)  # per-token
        return F.normalize(h, dim=-1)                       # keep them all

Because documents are encoded independently of the query, their token vectors are computed once and stored — late interaction is precomputable, unlike a cross-encoder.

Indexing and the cost of fine detail

The power has a price: instead of one vector per document, the index holds one vector per token.

python
# Index: store every token vector, tagged by its document
index = []
for doc_id, doc in enumerate(corpus):
    for tok_vec in colbert.encode(doc):
        index.append((doc_id, tok_vec))   # ~|corpus| x avg_doc_len vectors

# Search: MaxSim against each candidate document's token vectors
def search(query, candidates):
    Q = colbert.encode(query)
    return sorted(candidates, key=lambda d: maxsim(Q, vectors[d]), reverse=True)

That is roughly a 10–100× larger index than a single-vector retriever — the central engineering tension of multi-vector models, and what ColBERTv2 and PLAID spend their cleverness compressing (residual quantization, centroid pruning) back down to single-vector territory.

Where it sits

Single vector (bi-encoder)
Late interaction (ColBERT)
Vectors per doc
One
One per token
Matching
Pooled cosine
Token-level MaxSim
Long documents
Diluted by pooling
Robust — max ignores filler
Index size
Small
Large (10–100×)
Precomputable?
Yes
Yes (unlike a cross-encoder)
Quality
Good
State of the art

It is the natural third point after the cross-encoder vs bi-encoder trade-off: the bi-encoder is cheapest and coarsest, the cross-encoder is most accurate and un-indexable, and late interaction buys most of the accuracy back while staying searchable.

Performance

On MS MARCO passage ranking, late interaction tops single-vector retrievers on both ranking quality and recall:

ModelMRR@10Recall@1000Index Size
BM2518.785.70.5GB
DPR (single)31.295.221GB
ANCE33.095.921GB
ColBERT36.097.0154GB
ColBERTv239.798.425GB

ColBERTv2's headline is the last column: the same late-interaction quality at a fraction of the original index footprint.

Other multi-vector and learned-sparse models

  • Poly-encoder — a lighter middle ground: a handful of learned attention codes per document instead of one-per-token, with a small amount of query-time interaction.
  • SPLADE — projects each token onto the vocabulary to produce a learned sparse vector, getting term-level matching that runs on an inverted index.
  • ColBERTv2 / PLAID — residual compression and centroid-based pruning that make late interaction practical at web scale.

Best practices

  1. Cap document length (ColBERT defaults to ~180 tokens) — the index grows linearly with it.
  2. Use it as a re-ranker first — MaxSim over a BM25 or bi-encoder shortlist captures most of the gain at a fraction of the index cost.
  3. Compress early — residual quantization and centroid pruning (ColBERTv2/PLAID) are what make full-corpus late interaction affordable.
  4. Watch the [Q]/[D] markers — query and document encoders differ by a prepended marker token that meaningfully sharpens matching.

References

  • Khattab & Zaharia "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT"
  • Santhanam et al. "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction"
  • Humeau et al. "Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring"
  • Formal et al. "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking"

If you found this explanation helpful, consider sharing it with others.

Mastodon