A bi-encoder crushes a whole document into one vector — fast to index, but a long or many-faceted passage gets averaged into mush. Multi-vector models like ColBERT keep one vector per token and defer the matching to query time. The result sits between a bi-encoder and a cross-encoder: nearly the precision of joint attention, while staying precomputable and searchable.
Interactive ColBERT Visualization
The visualization below is real: every token is a genuine GloVe vector, and the matrix is real cosine similarity. Each query token's strongest match (the ringed cell) is the "max" in MaxSim. Flip to the single-vector view to watch all that detail collapse into one number — and try the long, padded document to see pooling fail where MaxSim holds.
Single vector vs late interaction
Helpful background
Pooling strategies
Compare ways to turn token vectors into one embedding, including padding masks and matching the model’s training setup.
A bi-encoder pools each side into one vector and takes a single cosine — every token's contribution is averaged in, so a few relevant terms can be drowned by filler:
ColBERT keeps every token vector and scores with MaxSim: each query token takes its best match across the document, and those maxima are summed.
where E_{q_i} is the embedding of query token i and E_{d_j} of document token j. The max is the key: a single strong match counts fully, no matter how long the document — so MaxSim never dilutes.
def bi_encoder_score(q_tokens, d_tokens): # single vector
q = normalize(mean_pool(encode(q_tokens))) # one vector
d = normalize(mean_pool(encode(d_tokens))) # one vector
return q @ d # one cosine
def colbert_score(q_tokens, d_tokens): # late interaction
Q = normalize(encode(q_tokens)) # one vector per query token
D = normalize(encode(d_tokens)) # one vector per doc token
sim = Q @ D.T # |q| x |d| matrix
return sim.max(dim=1).values.sum() # MaxSim: best per row, summed
The ColBERT encoder
The architecture is deliberately light — a frozen-ish BERT, a linear projection to a small dimension (128), and L2 normalization. The trick is what it doesn't do: it never pools. Query and document each keep a full bag of token vectors.
class ColBERT(nn.Module):
def __init__(self, bert, dim=128):
super().__init__()
self.bert, self.proj = bert, nn.Linear(768, dim)
def encode(self, tokens):
h = self.proj(self.bert(tokens).last_hidden_state) # per-token
return F.normalize(h, dim=-1) # keep them all
Because documents are encoded independently of the query, their token vectors are computed once and stored — late interaction is precomputable, unlike a cross-encoder.
Indexing and the cost of fine detail
The power has a price: instead of one vector per document, the index holds one vector per token.
# Index: store every token vector, tagged by its document
index = []
for doc_id, doc in enumerate(corpus):
for tok_vec in colbert.encode(doc):
index.append((doc_id, tok_vec)) # ~|corpus| x avg_doc_len vectors
# Search: MaxSim against each candidate document's token vectors
def search(query, candidates):
Q = colbert.encode(query)
return sorted(candidates, key=lambda d: maxsim(Q, vectors[d]), reverse=True)
That is roughly a 10–100× larger index than a single-vector retriever — the central engineering tension of multi-vector models, and what ColBERTv2 and PLAID spend their cleverness compressing (residual quantization, centroid pruning) back down to single-vector territory.
Where it sits
It is the natural third point after the cross-encoder vs bi-encoder trade-off: the bi-encoder is cheapest and coarsest, the cross-encoder is most accurate and un-indexable, and late interaction buys most of the accuracy back while staying searchable.
Performance
On MS MARCO passage ranking, late interaction tops single-vector retrievers on both ranking quality and recall:
| Model | MRR@10 | Recall@1000 | Index Size |
|---|---|---|---|
| BM25 | 18.7 | 85.7 | 0.5GB |
| DPR (single) | 31.2 | 95.2 | 21GB |
| ANCE | 33.0 | 95.9 | 21GB |
| ColBERT | 36.0 | 97.0 | 154GB |
| ColBERTv2 | 39.7 | 98.4 | 25GB |
ColBERTv2's headline is the last column: the same late-interaction quality at a fraction of the original index footprint.
Other multi-vector and learned-sparse models
- Poly-encoder — a lighter middle ground: a handful of learned attention codes per document instead of one-per-token, with a small amount of query-time interaction.
- SPLADE — projects each token onto the vocabulary to produce a learned sparse vector, getting term-level matching that runs on an inverted index.
- ColBERTv2 / PLAID — residual compression and centroid-based pruning that make late interaction practical at web scale.
Best practices
- Cap document length (ColBERT defaults to ~180 tokens) — the index grows linearly with it.
- Use it as a re-ranker first — MaxSim over a BM25 or bi-encoder shortlist captures most of the gain at a fraction of the index cost.
- Compress early — residual quantization and centroid pruning (ColBERTv2/PLAID) are what make full-corpus late interaction affordable.
- Watch the
[Q]/[D]markers — query and document encoders differ by a prepended marker token that meaningfully sharpens matching.
References
- Khattab & Zaharia "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT"
- Santhanam et al. "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction"
- Humeau et al. "Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring"
- Formal et al. "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking"
Related concepts
Master the BM25 algorithm, the probabilistic ranking function powering Elasticsearch and Lucene for keyword-based document retrieval and search systems.
How HNSW, IVF-PQ, and LSH compare for approximate nearest neighbor (ANN) search — recall, latency, memory, build cost, and update characteristics — with Annoy, ScaNN, and DiskANN included for completeness.
Learn how binary embeddings use 1-bit quantization for ultra-compact vector representations, enabling billion-scale similarity search with 32x memory reduction.
Understand the fundamental differences between independent and joint encoding architectures for neural retrieval systems.
How HNSW navigates a layered proximity graph to find nearest neighbors in logarithmic time — the default in-memory index of modern vector databases.
Build hybrid retrieval systems combining BM25 sparse search with dense vector embeddings using reciprocal rank fusion for superior semantic search performance.
