Contextual Retrieval & Late Chunking

Traditional RAG ingestion splits documents into fixed 200–500 token passages before encoding. As a result, a chunk stating "The company’s revenue grew by 3% over the previous quarter" loses the entity name and reporting period that appeared on page one. In August 2024, researchers at Jina AI introduced Late Chunking, leveraging 8K-token bi-encoders to run transformer self-attention across the entire document prior to mean-pooling chunk spans. In September 2024, Anthropic introduced Contextual Retrieval, using prompt-cached LLM calls to prepend a 50–100 token document-specific header to every chunk before computing both dense embeddings and BM25 lexical indexes.

Sources: s1, s2, s3

Coreference resolution and implicit document scope are among the most common causes of retrieval misses in enterprise RAG. When chunks are embedded in isolation, queries naming both an entity and a metric fail to match passages that use pronouns or section-inherited context. Contextual retrieval enriches both vector and BM25 lexical search with explicit entity names and dates, while late chunking achieves context conditioning at native embedding inference speed without invoking a generative LLM for every chunk.

Sources: s1, s2

When indexing an SEC 10-K filing, a contextual-retrieval pipeline caches the full filing in the LLM prompt cache and asks the model to generate a two-sentence header for each chunk (for example, "This chunk is from ACME Corp’s Q2 2025 10-K filing discussing European cloud segment operating margins"). That header is prepended to the raw passage before indexing in OpenSearch and pgvector. Alternatively, a late-chunking pipeline feeds the entire 6,000-token section into `jina-embeddings-v3`, extracts the contextualized token embeddings, and pools them by paragraph boundaries.

Sources: s1, s3

Contextual retrieval requires an LLM generation pass per chunk at indexing time; even with prompt caching reducing input token costs, re-indexing millions of rapidly changing documents adds nontrivial ingestion latency and cost. Late chunking avoids generative LLM cost but is bounded by the maximum context window of the embedding model and does not enrich lexical BM25 indexes with explicit keywords.

Sources: s2, s3