BM25 is the default ranking function in Lucene and Elasticsearch, but most engineers never tune its parameters. This post unpacks the algorithm’s internals, practical configuration levers, and the production trade-offs that separate adequate search from great search.
Hands-on build guide for a hybrid sparse-dense retriever (BM25 + embedding reranking) with a from-scratch HNSW vector index. Real runnable code, architecture diagrams, and a roadmap to senior-level.
Step-by-step build of a real hybrid retrieval system that combines learned BM25 with dense embeddings, fuses scores, and streams answers — the kind of project that makes hiring managers stop scrolling.
BM25 (Best Matching 25) is a probabilistic ranking function that powers modern search engines by scoring document relevance based on query terms, term frequency saturation, inverse document frequency, and document length normalization. As an information retrieval engineer, you’ll use BM25 for precise lexical matching in applications like Elasticsearch, Azure Search, and custom retrievers—outperforming TF-IDF while complementing semantic embeddings in hybrid systems.[1][3][4] This zero-to-hero tutorial takes you from basics to production-ready implementation, pitfalls, tuning, and strategic decisions on when to choose BM25 over vectors or hybrids. ...