RAG evaluation metrics: four ways a good score misleads
Saturation, partial credit, abstention counted as a catch, and detectability without power. Four ways a RAG metric reports a result it did not earn.
A RAG metric can be computed correctly, from real data, and still tell you something false. Not through fraud, through structure. Here are four we hit while measuring a default stack in our RAG failure audit, each with the number that exposed it.
1. Saturation makes a real improvement look small
When you compare two retrievers by averaging across probes, any probe already scoring at ceiling contributes a zero to the average no matter how good the new retriever is.
Adding BM25 to a dense retriever moved 2 of 10 probes. It looks like a weak result until you notice that 7 of the 10 were already at maximum on the baseline. Those rows could not move. They were not evidence of anything, and averaging them in dragged the reported improvement toward zero.
The fix: report per probe, and say which probes had headroom.
2. Partial credit invents safety
Token-overlap scoring gives credit for getting part of a span. For a lot of tasks that is right. For anything where the whole span is the point, it is a way of reporting success you did not have.
The clearest version of this is in our PII work, where direct-identifier mentions score 17.0% under an all-or-nothing rule (274/1,612) and 62.5% when any overlap counts. Same run, same detections. The gap is purely a scoring choice.
Note the unit: those are annotated mentions, not entities. Entity recall is a separate and stricter question, because an entity is only redacted when every one of its mentions is. Getting the unit wrong is its own way for a correct number to mislead.
Ask which one a benchmark used. If it does not say, assume the more generous one and ask for the counts.
3. Abstention counted as a catch
We measured whether a cheap guardrail, checking that a cited chunk contains the value
the answer asserts, catches failures. The rollup at config
B, over 53 failures:
| outcome | count |
|---|---|
| caught by the citation check | 0 |
| caught by a missing citation | 0 |
| the system had already abstained | 17 |
| slipped through | 36 |
17 of those failures were on questions the system had already declined to answer. It is tempting to book those as the guardrail working. They are not: the guardrail had nothing to catch, because nothing was asserted.
Only 0 is the check doing its job.
A metric that pools abstentions with catches reports a guardrail that is not there. Count them apart, and label the metric by what it consulted: anything that reads the answer key is an evaluation, not a guardrail.
4. Detectability reported without power
The most seductive one. You look for a signal that separates failures from successes, find one in a cell, and report a threshold.
Across 77 probe-configuration cells for the primary embedder, 1 was separable, 4 were not, and 72 were underpowered: too few failures or too few passes to decide either way. That is 93.5% of the grid.
If you report only the separable cell, you are reporting the one place a threshold appeared out of 77 chances, without saying how many chances there were. That is a multiple-comparisons problem wearing a finding’s clothes.
The honest report says: one cell separated, 4 did not, and most of the grid could not be judged.
The common thread
Each of these reports a number that is arithmetically correct and answers a different question from the one you asked. The defence is the same every time:
- Raw counts beside every rate. 0.0% of 53 reads differently from “0%”.
- Say what the denominator was, including the cells you could not judge.
- Name what the metric consulted. Ground truth, or only what production sees.
The method these came out of is in how to evaluate a RAG stack, and the full run is on the RAG failure audit page.