Multi-Vector Late Interaction

A bi-encoder crushes a whole document into one vector — fast to index, but a long or many-faceted passage gets averaged into mush. Multi-vector models like ColBERT keep one vector per token and defer the matching to query time. The result sits between a bi-encoder and a cross-encoder: nearly the precision of joint attention, while staying precomputable and searchable.

Interactive ColBERT Visualization

The visualization below is real: every token is a genuine GloVe vector, and the matrix is real cosine similarity. Each query token's strongest match (the ringed cell) is the "max" in MaxSim. Flip to the single-vector view to watch all that detail collapse into one number — and try the long, padded document to see pooling fail where MaxSim holds.

Single vector vs late interaction

A bi-encoder pools each side into one vector and takes a single cosine — every token's contribution is averaged in, so a few relevant terms can be drowned by filler:

ColBERT keeps every token vector and scores with MaxSim: each query token takes its best match across the document, and those maxima are summed.

where E_qi is the embedding of query token i and E_dj of document token j. The max is the key: a single strong match counts fully, no matter how long the document — so MaxSim never dilutes.

def bi_encoder_score(q_tokens, d_tokens):        # single vector
    q = normalize(mean_pool(encode(q_tokens)))   # one vector
    d = normalize(mean_pool(encode(d_tokens)))   # one vector
    return q @ d                                  # one cosine

def colbert_score(q_tokens, d_tokens):           # late interaction
    Q = normalize(encode(q_tokens))              # one vector per query token
    D = normalize(encode(d_tokens))              # one vector per doc token
    sim = Q @ D.T                                # |q| x |d| matrix
    return sim.max(dim=1).values.sum()           # MaxSim: best per row, summed

The ColBERT encoder

The architecture is deliberately light — a frozen-ish BERT, a linear projection to a small dimension (128), and L2 normalization. The trick is what it doesn't do: it never pools. Query and document each keep a full bag of token vectors.

class ColBERT(nn.Module):
    def __init__(self, bert, dim=128):
        super().__init__()
        self.bert, self.proj = bert, nn.Linear(768, dim)

    def encode(self, tokens):
        h = self.proj(self.bert(tokens).last_hidden_state)  # per-token
        return F.normalize(h, dim=-1)                       # keep them all

Because documents are encoded independently of the query, their token vectors are computed once and stored — late interaction is precomputable, unlike a cross-encoder.

Indexing and the cost of fine detail

The power has a price: instead of one vector per document, the index holds one vector per token.

# Index: store every token vector, tagged by its document
index = []
for doc_id, doc in enumerate(corpus):
    for tok_vec in colbert.encode(doc):
        index.append((doc_id, tok_vec))   # ~|corpus| x avg_doc_len vectors

# Search: MaxSim against each candidate document's token vectors
def search(query, candidates):
    Q = colbert.encode(query)
    return sorted(candidates, key=lambda d: maxsim(Q, vectors[d]), reverse=True)

That is roughly a 10–100× larger index than a single-vector retriever — the central engineering tension of multi-vector models, and what ColBERTv2 and PLAID spend their cleverness compressing (residual quantization, centroid pruning) back down to single-vector territory.

Where it sits

It is the natural third point after the cross-encoder vs bi-encoder trade-off: the bi-encoder is cheapest and coarsest, the cross-encoder is most accurate and un-indexable, and late interaction buys most of the accuracy back while staying searchable.

Performance

On MS MARCO passage ranking, late interaction tops single-vector retrievers on both ranking quality and recall:

ColBERTv2's headline is the last column: the same late-interaction quality at a fraction of the original index footprint.

Other multi-vector and learned-sparse models

• Poly-encoder — a lighter middle ground: a handful of learned attention codes per document instead of one-per-token, with a small amount of query-time interaction.
• SPLADE — projects each token onto the vocabulary to produce a learned sparse vector, getting term-level matching that runs on an inverted index.
• ColBERTv2 / PLAID — residual compression and centroid-based pruning that make late interaction practical at web scale.

Best practices

1. Cap document length (ColBERT defaults to ~180 tokens) — the index grows linearly with it.
2. Use it as a re-ranker first — MaxSim over a BM25 or bi-encoder shortlist captures most of the gain at a fraction of the index cost.
3. Compress early — residual quantization and centroid pruning (ColBERTv2/PLAID) are what make full-corpus late interaction affordable.
4. Watch the [Q]/[D] markers — query and document encoders differ by a prepended marker token that meaningfully sharpens matching.

References

• Khattab & Zaharia "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT"
• Santhanam et al. "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction"
• Humeau et al. "Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring"
• Formal et al. "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking"

Related concepts