Hybrid search for RAG: what BM25 actually bought us

Hybrid search, dense embeddings plus a keyword index like BM25, is the standard advice when a RAG system is missing things it should find. It is good advice. It is also usually given without a number attached, which makes it impossible to tell whether it is the fix for your problem or a fix for somebody else’s.

So we measured it. The setup, the corpus and every count below come from our RAG failure audit, where we broke a default RAG stack on purpose and recorded what each configuration change bought.

The comparison

Two configurations, identical except for the retriever:

• C, the baseline: recursive chunking at 1,000 characters with 200 overlap, k=8, dense retrieval only.
• M, the same chunking and the same k, with BM25 added.

Both run over the same corpus of 625 documents and 275 questions, 25 per probe, fixed seed 7, embedder mxbai-embed-large, generator qwen2.5:7b.

The result: it moved 2 of 10 probes

probe	dense only	with BM25	change
multihop	2 of 25	7 of 25	+5
paraphrase	23 of 25	25 of 25	+2
table_split	0 of 25	0 of 25	0

The two that moved are multihop and paraphrase. Multihop went from 2 of 25 to 7 of 25, which is the largest single change anywhere in the comparison. Paraphrase went from 23 of 25 to 25 of 25.

The part that matters more: 7 probes had no room to move

boundary, control, distractor, duplicate crowding, negation, summarisation and superseded were already at 25 out of 25 on the dense retriever alone. Hybrid cannot improve a probe that is already retrieving everything, so those 7 rows carry no information about whether BM25 helps. They are not evidence that it does not.

That leaves 3 probes where the comparison could say anything at all, and 2 of those 3 moved.

This is the thing that gets lost when hybrid search is reported as a single percentage. Average the ten probes together and BM25 looks like a small improvement, because seven of the ten rows are pinned at the ceiling and drag the average toward “no change”. Look at the rows with headroom and it is doing real work on exactly two kinds of question.

And one that had room and did not move

table split retrieves 0 of 25 without BM25 and 0 of 25 with it. That probe splits a fact across a table boundary, so the answer is not in any single chunk under either retriever. No amount of better ranking fixes a document that was cut in the wrong place. That is a chunking problem wearing a retrieval problem’s clothes, and hybrid search is the wrong tool for it.

The caveat that limits all of this

Every one of the 275 questions in this corpus contains an identifier that also appears in the documents, and it is selective: the median question’s identifier appears in 2 of 768 chunks, and for 125 questions it appears in exactly 1.

That hands a keyword index an anchor. So the multihop gain is not a clean demonstration that BM25 understands multi-hop questions; it is consistent with BM25 having a rare token to lock onto that the dense retriever was diluting. The corpus inflates every configuration this way, not the hybrid one specifically, but it is exactly the property your corpus may not share.

What to take from this

1. Hybrid search is a targeted fix, not a general upgrade. In this run it moved questions that need two documents, and questions phrased differently from the source text, on a corpus where rare identifiers gave a keyword index leverage. If your questions do not carry anchors like that, expect less.
2. Check for saturation before you measure. A probe already at ceiling cannot show an improvement, and including it in an average makes any change look smaller than it is.
3. If the fact is split across chunks, fix the chunking. Retrieval cannot return a sentence that does not exist in any chunk.

The full audit, including which failure modes fired under which configuration and what each change cost, is in the RAG failure audit. The results file and the code that produced it are in the archive linked there, so every number on this page can be recomputed.

Working out which of these applies to a specific stack is what our AI automation audit does.