RAG Faithfulness Evaluation

An evaluator divides an answer into statements and assesses whether the available context supports each one, often using an entailment model or language-model judge. A score may aggregate supported statements, but details such as compound claims, uncertainty and denominator selection affect the result. The supplied context is the reference, not every fact the evaluator happens to know. A faithful answer can still be wrong if the context is outdated or false, and an unfaithful claim can happen to be true but absent from the permitted evidence. These distinctions make faithfulness one dimension of RAG quality rather than overall correctness.

The practitioner defines atomic claim rules, allowed evidence and how partial or ambiguous support is labeled. They validate decomposition and support judgments against expert annotations, including numerical differences and missing qualifiers. Useful artifacts include claim–passage pairs, judge configuration, aggregation rules and error analysis. The evaluator receives the context actually used during generation, not a broader candidate set that could retroactively support the answer. Separate checks assess source reliability, answer completeness and relevance. Changes in the judge or decomposition prompt require renewed calibration before scores are compared.

A judge can overlook subtle contradictions, split claims inconsistently or reward answers with fewer substantive statements. Aggregation can conceal one critical unsupported claim among many easy supported ones. Missing evidence is also different from explicit contradiction. Quality reports should preserve claim-level judgments and state how uncertainty is handled. Faithfulness is most useful when calibrated and combined with independent checks of source validity and task coverage, without pretending that contextual agreement establishes universal truth.