BM25

BM25 belongs to the probabilistic information retrieval tradition. Its score gives more weight to terms that are informative across the collection, reduces the marginal contribution of repeated occurrences and adjusts for document length. Parameters control aspects of term-frequency saturation and length normalization. Text analysis also matters: tokenization, stemming and field handling determine which terms match before scoring. BM25 differs from dense retrieval because it primarily uses lexical overlap rather than learned vector proximity. Its numeric scores are ranking signals within a query and index configuration, not direct probabilities of relevance.

The practitioner configures text analysis and searchable fields, establishes relevance labels and tunes scoring only where evaluation justifies it. Tests include identifiers, abbreviations, spelling variants and queries whose relevant document uses different wording. Useful artifacts include the index mapping, query strategy and measured ranking baseline. Comparing BM25 with dense or hybrid search helps identify whether errors come from vocabulary mismatch or candidate ranking. Field weights and filters are documented so a result's position can be explained without attributing every effect to the BM25 formula.

Lexical matching can miss paraphrases, synonyms and cross-language relevance. Scores are also affected by corpus statistics and document preparation, so thresholds do not transfer automatically between indexes. Aggressive stemming can merge distinct identifiers, while repeated terms can still distort ranking under unsuitable configuration. Evaluation should include the actual query vocabulary and field structure. BM25's simplicity and usefulness do not make it a universal solution or justify skipping a retrieval quality test.