A RAG benchmark is only worth running if it can fail

A benchmark that always passes is not measuring your system. It is measuring whether you wrote a hard enough question, and it answers no.

This sounds obvious. We published a case study, found two of its probes were measuring artefacts rather than the failure they claimed, and withdrew it the same day. The rule we adopted afterwards, and the counts that would have caught it, come from our RAG failure audit.

The rule

A probe may not be reported until a test proves it is armed.

The test has to show two things: the failure mechanism fires under the naive configuration, and it does not fire under the corrected one. Both halves matter. A probe that fires everywhere is broken in the other direction: it is not measuring the thing you changed.

Write that test before you write the probe. If you cannot make it fail on purpose, you do not have a probe, you have a question.

What it catches, concretely

The boundary probe plants an answer so a naive splitter cuts across it. Under a fixed-size splitter with no overlap, the span survived intact in 0/25 documents and retrieval returned 0/25. Under a recursive splitter with overlap it survived in 25/25 and retrieval returned 24/25.

That is an armed probe. The mechanism visibly fires on one side and visibly does not on the other, and the check that decides it, whether the gold sentence appears whole inside a single chunk, is deterministic and recomputable from the corpus without a model.

Now imagine the same probe reported only at the corrected configuration. It scores 24/25, and you would conclude the system handles split answers well. What it actually shows is that the failure was never posed.

The count that keeps you honest

Report, for every failure probe, whether it fired at all. In this run 8 of the 9 did. duplicate crowding never did under any configuration.

That gets reported as not-fired rather than as a pass. The probe is armed and tested; this corpus did not trigger it. Rolling its perfect score into an average would report the system as safer than it was measured to be, when what was actually measured is a property of the corpus.

Controls are the opposite case and must not be pooled with these. A control is supposed not to fire, and a control that fires means the instrument is broken.

A checklist

• Every probe has a test proving it fires under the naive config and not under the corrected one.
• The check that decides pass or fail is deterministic, and does not call a model.
• Probes that did not fire are reported as not-fired, never as passes.
• Controls are counted apart from failure probes, in both directions.
• Raw counts beside every rate, so a reader can see the denominator.

The related failure, a comparison where most probes were already at ceiling and so could not move, is covered in four ways a good score misleads. The method this rule sits inside is in how to evaluate a RAG stack, and the run itself is on the RAG failure audit page.