RAG evaluation: how to test a stack that fails quietly

Most RAG evaluation writing is a list of metrics. Faithfulness, context precision, answer relevance, all with a formula attached. The metrics are fine. The problem is that they assume the hard part is scoring, and the hard part is not scoring.

The hard part is making the failure happen in the first place, and knowing you did.

This is the method we used in our RAG failure audit, where we deliberately broke a default stack across 11 probes, 9 of them failure probes and 2 of them controls (control and unanswerable), over 275 questions and 625 documents.

Step 1: run the controls, and report them first

Controls are the questions the system must get right. Plain retrieval, answer in one chunk, no trap.

In this run the control probe had 0 retrieval failures across 175 attempts and 0 generation failures across 100.

If that had not held, nothing else would have meant anything. A stack that cannot answer an easy question is not being tested by a hard one, it is just failing.

Report controls before findings, not in an appendix. A result whose controls are buried at the bottom is asking you to take the instrument on trust.

Step 2: prove each probe is armed

This is the step that everyone skips, and it is the one that invalidated two of our own probes before we caught it.

A probe may not be reported until a test proves it fires. The test has to show the failure mechanism actually triggering under the naive configuration and not triggering under the corrected one. Without that, a probe that scores 100% is ambiguous between “the system handled it” and “the probe never posed the problem”.

In this run, 8 of the 9 failure probes fired: boundary, distractor, multihop, superseded, summarisation, table split, paraphrase and negation. duplicate crowding never did under any configuration, and that is reported rather than hidden. It is armed and tested; this corpus simply did not trigger it, which is a negative result about the corpus rather than a score for the system.

Step 3: separate retrieval failure from generation failure

These need different fixes and they get conflated constantly.

The boundary probe, which splits an answer across a chunk edge, records 33 retrieval failures out of 175 attempts and 0 generation attempts. When retrieval never returns the right chunk, the generator is not the problem and no prompt change will help.

Count them apart. A blended score cannot tell you which half of the pipeline to work on.

Step 4: grade deterministically, never with an LLM judge

An LLM judge introduces a second model’s failure modes into your measurement of the first model’s failure modes, and you now cannot tell them apart. Every grade in this run is a deterministic comparison against a known answer.

That constrains the questions you can ask. It is worth it: the results are reproducible from the archive, by anyone, without a model.

Step 5: report what you could not establish

Of 7 probes graded for severity, the classes came out as:

class	probes
none	control, distractor and unanswerable
low	negation and table split
undetermined	summarisation and superseded

undetermined is a real result and the most common one in honest evaluation. It means the comparison the class needs could not be made, usually because there were not two groups to compare. “We could not establish this” and “this does not happen” are different findings, and collapsing them is how evaluations end up flattering.

The short version

1. Controls first, always.
2. No probe reported until a test proves it is armed.
3. Retrieval failures and generation failures counted apart.
4. Deterministic grading only.
5. Raw counts beside every rate, and undetermined reported as a result.

The full audit, the results file and the code archive are on the RAG failure audit page.

Related reading: what BM25 actually bought on the same corpus.

If you would rather have this run against your own stack than against ours, that is our AI automation audit.