Retrieval-Augmented Generation & Knowledge Systems

Agentic RAG gives an agent control over when and how to retrieve information before answering. Instead of always running one fixed search, the system can choose a source, refine a query or retrieve again when evidence is insufficient, with explicit limits on the actions and resources available to it.

Multimodal RAG retrieves evidence from more than one media type, such as text, images, audio or video, and supplies usable evidence to a generative model. The engineering challenge is preserving the information carried by each medium while linking retrieval results to their source location and the question being answered.

Query optimization for retrieval transforms a user's request into search inputs that are more likely to find relevant evidence. It can resolve references, expand terminology, decompose a question or generate a hypothetical passage, while preserving the original intent and testing whether the transformation actually improves retrieval.

Self-reflective RAG adds decisions that assess retrieved evidence or generated answers and use that assessment to retrieve again, revise or stop. Self-RAG and Corrective RAG are specific research approaches within this space; ordinary reflection instructions alone do not reproduce their trained mechanisms or evaluation claims.

Embedding models map inputs such as words, passages or images to numerical vectors whose geometry supports a task. For search, useful representations place relevant queries and documents in compatible regions; selecting or adapting an embedding model therefore requires evaluation of relevance, language coverage and the intended comparison function.

Neo4j is a graph database that represents data as nodes, relationships and properties and supports queries over those connections. In AI applications, the skill includes modeling entities and relationships, writing reliable graph queries and deciding when explicit graph structure adds value beyond document or vector retrieval.

AI grounding connects an answer to evidence the system can inspect, while citations identify where particular claims are supported. The skill combines evidence selection, claim attribution and source presentation, so a reader can distinguish a supported statement from an inference or a claim for which the available material is insufficient.

Document AI turns document content and structure into information that software can use. It combines capabilities such as text recognition, layout analysis, field extraction and validation, with attention to tables, reading order and source locations rather than assuming every document is a clean sequence of plain text.

Document chunking divides source material into units that can be indexed, retrieved and supplied to a model. Good boundaries preserve enough local meaning to answer a question while keeping units selective, traceable and small enough for the retrieval and generation system that will use them.

GraphRAG uses a graph representation of entities and relationships to help retrieve or organize evidence for generation. The name includes a specific Microsoft research approach as well as broader graph-assisted designs; a useful description should state which graph, retrieval mechanism and evidence aggregation the implementation actually uses.

Knowledge graphs represent entities and their relationships using explicit identities and meaningful relation types. They help applications connect facts across sources, query those connections and preserve provenance; their value depends on the quality of the model and evidence, rather than simply displaying data as nodes and edges.

Visual document retrieval searches documents using representations of rendered pages or page regions, preserving information in layout, charts and images. It can complement text extraction when the relevant evidence is visual, but retrieving the correct page and accurately interpreting that page remain separate tasks to evaluate.

Retrieval-augmented generation supplies a generative model with external evidence found for the current request. The application indexes or searches a collection, selects relevant material and uses it during answer generation, making knowledge updates and source attribution possible without relying only on information encoded in the model's weights.

Hybrid search combines retrieval signals, commonly lexical matching and dense-vector similarity, to find candidates that either method might miss alone. The skill includes defining the combination rule and evaluating the resulting ranking, with particular attention to exact terms, semantic paraphrases and the behavior of filters.

Search reranking applies a second scoring or ordering step to an initial candidate set. It spends more detailed computation on a smaller collection of results, using signals such as query–document interaction or business rules to improve the order presented to a user or supplied to a generative model.

FAISS is a library for efficient similarity search and clustering over dense vectors. It supplies exact and approximate index structures that applications can use to find nearby embeddings, while leaving document storage, permissions, update workflows and the meaning of those embeddings to the surrounding system.

Vector databases store vector representations alongside identifiers and often metadata, providing similarity search and lifecycle operations for an application. They support retrieval infrastructure; deciding what to embed, which similarity means relevance and how retrieved evidence should be used remains a separate modeling and application responsibility.

pgvector is a PostgreSQL extension that adds vector data types and similarity-search operations to a relational database. It lets an application keep embeddings near its existing records and query them with SQL, while requiring deliberate index selection, filtering and performance evaluation as the collection and workload grow.

Sentence-Transformers is a library for using and training embedding and related text-matching models. It supports representations for sentences and passages as well as cross-encoder scoring, making it useful for retrieval experiments where model choice, input formatting and task-specific evaluation matter more than a generic notion of semantic similarity.

Graph databases store and query connected data with explicit relationships as a first-class part of the data model. The skill involves choosing a graph representation, writing bounded traversals and maintaining consistency, so an application can answer relationship-oriented questions without repeatedly reconstructing connections from unrelated records.

Azure Document Intelligence is a Microsoft service for extracting text, layout and structured information from documents. Using it well involves choosing supported models, connecting extracted fields to page evidence and validating results against the application's requirements, especially when document layouts or image quality differ from the configuration examples.

Contextual retrieval adds explanatory context to individual chunks before indexing them so that isolated passages remain meaningful during search. A specific Anthropic approach uses generated chunk context for both embeddings and lexical indexing; the broader practice is useful only when the added context accurately preserves the source's identity and meaning.

BM25 is a lexical relevance scoring method that ranks documents using query-term matches, term rarity and document-length normalization. It is a strong baseline for text retrieval and the lexical component of many hybrid systems, especially when exact identifiers or specialist terms matter more than broad semantic similarity.

Dense retrieval searches by comparing learned vector representations of queries and documents. It can find relevant passages with different wording from the query, but its quality depends on the encoder, training objective and similarity function; nearby vectors are candidates for relevance rather than verified answers.

OpenSearch is a search and analytics engine that can support lexical, vector and hybrid retrieval in an application. The skill involves index design, query construction and operational management, with attention to how text analysis, vector configuration and filters jointly determine which records can be returned and how they are ranked.

Chroma is a retrieval database and toolkit for storing embeddings, documents and metadata and querying related records. It can simplify application prototypes and deployed retrieval workflows, but useful search still depends on the selected embedding model, collection design, filters and explicit handling of source versions and access.

LanceDB is a database for vector and multimodal retrieval built around columnar data storage. The skill combines table and schema design, embedding management and search configuration, particularly where vectors need to remain connected to structured fields or media references that an application can filter, inspect and update.

Metadata filtering restricts retrieval using structured fields such as product, date, tenant or access category. It helps search obey explicit constraints that embeddings may not preserve, but correct behavior depends on the filter semantics, where filtering occurs and whether the application enforces the underlying authorization policy.

Milvus is a vector database designed to store and search embeddings with structured metadata and scalable indexing infrastructure. Competence involves choosing a schema and index, managing data ingestion and lifecycle, and validating filtered retrieval under the actual workload rather than treating scale-oriented architecture as evidence of relevance.

Pinecone is a managed retrieval service for indexing and querying vector-based representations with associated metadata. The skill involves defining record and namespace organization, selecting supported search behavior and measuring quality, latency and lifecycle costs, while keeping application authorization and source correctness explicit outside the service abstraction.

Qdrant is a vector database that stores vector representations with structured payloads and provides similarity search and filtering. The skill includes configuring collections and indexes, designing payload fields and evaluating recall under selective queries, particularly where metadata and semantic search must work together without weakening explicit constraints.

Vector indexing organizes embeddings so similarity search can find candidates without comparing every stored vector on each query. Exact and approximate structures trade memory, build effort, update behavior and recall; the skill is selecting and tuning that trade-off with measurements tied to the application's relevance requirements.

Weaviate is a database for vector-oriented and hybrid retrieval with structured objects and metadata. Using it requires deliberate schema, representation and query choices, along with testing of filtering, updates and source resolution; built-in integration features do not replace an application's relevance and access-control requirements.

Cross-encoder reranking scores each query and candidate document together, allowing the model to inspect their interaction before reordering results. It is a second-stage retrieval technique that can distinguish subtle relevance differences, while trading per-candidate computation for a ranking that independent embedding similarity may not provide.

Multi-vector retrieval represents a source item with several vectors and defines how their matches produce an item-level result. The vectors may represent chunks or derived views, or tokens and patches used in late interaction; the aggregation rule is central because multiple vectors alone do not define a retrieval method.