Late Interaction & Multi-Vector Retrieval
Late interaction is a neural retrieval paradigm, exemplified by ColBERT for text and ColPali for visually rich documents, that encodes queries and documents into matrices of token-level or image-patch vectors rather than compressing each passage into a single pooled embedding. Relevance is computed at query time via a lightweight maximum-similarity (MaxSim) operator that sums the best-matching document vector for every query vector. This preserves fine-grained lexical and layout signals that single-vector bi-encoders lose while remaining orders of magnitude faster than full cross-encoder reranking, though it increases index storage footprint and memory requirements.
Origin and context
Information-retrieval architectures traditionally faced a stark trade-off between fast single-vector bi-encoders—which bottleneck an entire passage into one vector—and accurate cross-encoders that jointly attend over every query-document token pair at prohibitive latency. In April 2020 (SIGIR 2020), Omar Khattab and Matei Zaharia introduced ColBERT, delaying query-document interaction until after independent token-level encoding and using PLAID/residual compression in ColBERTv2 (NAACL 2022) to shrink the index. In 2024, Faysse et al. introduced ColPali, applying late interaction directly to Vision-Language Model (VLM) image-patch embeddings of PDF pages, eliminating fragile OCR and layout-chunking pipelines.
Why it matters
Single-vector dense embeddings frequently fail on exact part numbers, rare entities, multi-constraint queries, and complex PDF tables or charts because pooling averages away localized features. Late interaction retains one vector per token or visual patch, allowing each query token to align with its strongest counterpart anywhere in the document. In both text benchmarks (BEIR, LoTTE) and visual document benchmarks (ViDoRe), late-interaction models consistently outperform single-vector bi-encoders on out-of-domain generalization and complex visual layouts while supporting sub-100ms vector-index search.
Example
Instead of running OCR, table-detection, and text chunking on a 10,000-page financial PDF archive, an engineering team renders each page as an image and encodes it with ColPali into 1,030 patch vectors of 128 dimensions stored in Qdrant or Vespa using binary quantization. At query time, the user’s question is encoded into token vectors, scored via MaxSim against candidate page matrices, and the top-3 page images are passed directly to a multimodal LLM for grounded answer generation.
Maturity and evidence
Limits and open questions
Storing 500 to 1,000 vectors per document or page increases raw index size by an order of magnitude compared with single-vector embeddings. Practical deployments require aggressive quantization (int8 or binary vectors), token pooling, or two-stage prefetching to keep RAM and IOPS costs manageable at multi-million-document scale.
Related terms
References
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTKhattab & Zaharia / Stanford University & SIGIR 2020 · 2020-04-27 · class A
- ColPali: Efficient Document Retrieval with Vision Language ModelsFaysse et al. / ICLR 2025 (arXiv) · 2024-07-01 · class A
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late InteractionSanthanam et al. / NAACL 2022 (arXiv) · 2021-12-02 · class A
Last updated: 2026-10-19
This term is also covered in the Skills Atlas as multimodal rag skill.
This term is also covered in the Skills Atlas as dense retrieval skill.
This term is also covered in the Skills Atlas as search re ranking skill.
This term is also covered in the Skills Atlas as information retrieval skill.
This term is also covered in the Skills Atlas as multi vector retrieval skill.
Referencing this analysis in documentation, research, or an engineering write-up? Copy a formatted link or citation below:
Show citation text & canonical URL
[Late Interaction & Multi-Vector Retrieval — AI Glossary — Skills Intelligence](https://www.skillsintelligence.tools/glossary/term/late-interaction-retrieval)Skills Intelligence. (2026). Late Interaction & Multi-Vector Retrieval — AI Glossary. Skills Intelligence AI Glossary. Retrieved from https://www.skillsintelligence.tools/glossary/term/late-interaction-retrieval