RAG chunk size and overlap: what each one actually bought

Chunk size and overlap are the two knobs everyone turns first, and the advice attached to them is almost always qualitative: smaller chunks for precision, overlap so you do not cut an answer in half.

We isolated them. One probe, 25 questions, each with the answer planted so that a naive splitter cuts across it. Figures from our RAG failure audit.

The instrument

A span is intact when the gold sentence appears whole inside a single chunk. That check is deterministic and model-free: it is recomputed from the corpus generator, not judged.

config	splitter	span intact	retrieved
A	fixed 1,000, overlap 0	0/25	0/25
A_prime	recursive 1,000, overlap 0	25/25	19/25
B	recursive 1,000, overlap 200	25/25	24/25

Config A is the one worth staring at. It preserved 0/25 spans and retrieved 0/25. A fixed-size splitter with no overlap cut every single answer in half, and no amount of retrieval tuning was going to recover a sentence that exists in no chunk.

The splitter did the work

Moving from fixed to recursive splitting, with overlap held at 0, gained 19 answers and lost 0.

All 19 of those gains were by preservation: the span went from split across chunks to intact inside one. Zero were by ranking. The retriever did not get better. The documents stopped being destroyed before it saw them.

Overlap gained less, and not for the usual reason

Adding 200 overlap on top gained 6 more and lost 1.

Here is the part that contradicts the standard explanation. Of those 6 gains, 0 were by preservation and 6 were by ranking.

Overlap is sold as insurance against cutting an answer in half. On this corpus it preserved nothing further, because the recursive splitter had already made every span intact: 25/25. There was nothing left to preserve. What overlap actually did was change what the chunks looked like, which changed how they ranked.

That is a real benefit. It is a different benefit from the one on the label, and it will not transfer to a corpus where your splitter is still cutting spans.

The header effect

One more thing the run recorded. At config B, 7/25 answers arrived in a chunk that also contained the section header, against 0/25 without overlap.

Overlap pulled the preceding header into the answer’s chunk. A chunk carrying “Section 4.2 Maintenance intervals” plus the sentence retrieves better than the bare sentence, because the header carries the query’s vocabulary. That is the ranking mechanism, concretely.

What to do

1. Fix the splitter before you tune anything else. A fixed-size splitter with no overlap destroyed 25/25 answers here, and nothing downstream can recover them.
2. Do not credit overlap with preservation until you have measured preservation separately. Count spans intact, not just answers retrieved. They are different questions and only one of them is about your splitter.
3. Consider deliberately including headers rather than getting them by accident through overlap. If the mechanism is header vocabulary, do that on purpose.

The full audit and the code archive are on the RAG failure audit page. What all this costs in tokens is in RAG cost per query.