We broke a default RAG stack on purpose

Most RAG systems are not evaluated against the ways they actually fail. They are demoed on questions whose answers sit conveniently inside one chunk, and shipped.

So we built a stack the way the tutorials say to, planted a set of known faults in a corpus we control, and measured what survived. Everything below comes from one recorded run on a local machine. The code and the raw per-instance records are in the archive; every figure on this page is rendered from the results file, not typed in by hand. Its sha256 is served beside it at /code/rag-failure-audit.sha256, which lets you check that your download arrived intact, and that is all a digest can tell you here: the packing script is not in the archive, so you cannot repack the tree yourself and compare. That the packed tree is the one this page renders from is asserted by this site’s test suite, which the deploy runs before it ships anything, not by something you can verify from the download alone. The digest is for the archive currently served, so check your copy against it when you download. It changes when the archive changes, not when the site is redeployed: a copy that matched when you fetched it and fails against a later digest is out of date. A copy that never matched is damaged.

Read the controls first

Before any failure number means anything, the instrument has to be shown working. Controls are ordinary questions over the same corpus: the answer sits inside one chunk, nothing competes with it.

Probe	Layer	A	A_prime	B	C	C_prime	D	M
control	control	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25

They pass at every configuration in the matrix, and on the 4 configurations where generation ran the answer was correct every time. So when a probe below fails, it is the planted fault doing it.

Two limits travel with that. Controls were run under mxbai-embed-large only, so nothing here certifies nomic-embed-text, and the table that uses it is marked uncertified for that reason. And they show the instrument works on questions of this shape, which is the claim they are here to support and the only one they support.

What broke

Probe	Layer	A	A_prime	B	C	C_prime	D	M
boundary	architectural	0%0/25	76%19/25	96%24/25	100%25/25	96%24/25	100%25/25	100%25/25
distractor	architectural	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25
multihop	architectural	0%0/25	0%0/25	0%0/25	8%2/25	8%2/25	0%0/25	28%7/25
superseded	architectural	96%24/25	96%24/25	96%24/25	100%25/25	100%25/25	100%25/25	100%25/25
summarisation	architectural	0%0/25	0%0/25	0%0/25	100%25/25	100%25/25	88%22/25	100%25/25
table_split	architectural	0%0/25	0%0/25	0%0/25	0%0/25	0%0/25	0%0/25	0%0/25
duplicate_crowding	architectural	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25
unanswerable	architectural	n/a	n/a	n/a	n/a	n/a	n/a	n/a

Read these numbers as an upper bound, for a reason built into the corpus. All 275 of the 275 questions carry a phrase distinctive enough to act as a lookup key, whether an area code or a team name. The sharpest of those keys narrows the index to 1 chunk out of 768; the median question narrows to 2, and the least selective to 7. Any retriever, lexical or dense, can shortlist to a handful of candidates before meaning enters into it, which is closer to a key-value fetch than to search. Real corpora rarely hand you that.

At the sharpest end, 125 of the 275 questions have an anchor occurring in a single chunk, spread across 5 probes. Most are section codes; one probe’s are team names, which no identifier-shaped pattern would match. Selectivity rather than the shape of the token is what is measured here.

Hybrid retrieval is the one mitigation in this matrix, so it is worth isolating. Comparing C with M, which differ only in BM25 and hold k=8 fixed, hybrid moves 2 of the 10 probes with gold to retrieve, every one above plus the two embedding-bound ones, leaving out unanswerable, which has none. It moves multihop (2 to 7 of 25) and paraphrase (23 to 25 of 25).

The reason it moves so few is headroom, not identifiers. 7 probes already sit at the ceiling under C, so hybrid has nothing left to win on them, and a probe at the ceiling tells you nothing either way. Only 3 have room to move, and they do two different things: paraphrase and multihop gain, as above, while table_split stays at 0, because its answer row is severed from the header naming its columns under every chunking in the matrix.

Of the two that gain, multihop’s is the less surprising once the caveat above is taken seriously: its team name occupies a single chunk, and a token that rare is precisely what BM25 is for. That is the identifier leverage working, not a clean demonstration that hybrid retrieval understands multi-hop questions.

The leverage caveat above applies to every column here, not just to this one, because it is a property of the corpus rather than of the retriever.

Two probes indict the embedding model rather than the architecture, so they are reported separately and on both embedders. Without that, a miss here is a claim about one model rather than about retrieval.

Probe	Layer	A	A_prime	B	C	C_prime	D	M
paraphrase	embedding	80%20/25	84%21/25	84%21/25	92%23/25	92%23/25	88%22/25	100%25/25
negation	embedding	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25	100%25/25

Probe	Layer	A	A_prime	B	C	C_prime	D	M
paraphrase	embedding	36%9/25	36%9/25	36%9/25	44%11/25	44%11/25	36%9/25	100%25/25
negation	embedding	48%12/25	48%12/25	48%12/25	88%22/25	84%21/25	92%23/25	100%25/25

Negation fails twice, and the two failures belong to different components. The pair states one item in two stores with opposite polarity, so the two planted sentences differ by an opaque store code and the word “not”. The documents around them differ as documents do, in their headers and their filler; the sentences are what compete. Under mxbai-embed-large the negated chunk is retrieved 25/25 at every configuration, which is what licenses the generator-side discussion later on this page. Under nomic-embed-text at k=3 it arrives 12 times in 25, and in 9 of the 13 misses the affirmative twin is what arrived instead. On that embedder the probe fails before the model is reached at all.

The perfect column is weaker evidence than it looks. Counting the instances where the ordering was actually observed, so the count does not depend on k, mxbai-embed-large ranks the affirmative above the negated chunk in 8 of 25 at B and 14 of 25 at D. The second embedder does it in 12 of the 21 instances at B where either chunk was retrieved at all; the other 4 retrieved neither and so ordered nothing, which is not a pass and is not counted as one.

So it is not that one embedder represents polarity and the other does not, but the two are not failing the same way either. At B the primary embedder puts the correctly negated chunk first in 17 of 25, and the affirmative first in 8. The second puts the affirmative first in 12 of the 21 it ordered at all. One of them is unreliable on polarity; the other is actually biased against it. Saying both prefer the wrong chunk would be a directional claim, and it is false for the primary embedder.

The consequence differs more than the ordering does. At k=3 the primary embedder retrieves the negated chunk 25/25 however it ranks the pair, and both chunks in 24, so its ordering errors cost nothing that this probe can see. On the second embedder the same misordering pushes the negated chunk out of the context altogether, which is why the probe reads as a retrieval failure there and as a question about the generator here.

Hybrid retrieval is the one configuration where the wrong-polarity ordering all but disappears: out of 25 instances it survives in 1 under the primary embedder and 0 under the second. That is not evidence that BM25 understands negation. The store code is a rare exact token and the two documents carry different ones, so this is the same identifier leverage flagged above, sidestepping polarity rather than resolving it. A corpus whose contradictory passages are not conveniently labelled would not get this.

Three results are worth pulling out.

Fixed-size chunking destroyed every planted span. Under config A the sentence answering the question was cut in half by the chunk boundary in every single case: the gold sentence survives whole inside a chunk in 0/25 documents, so retrieval had nothing to find and the evidence reached the model in 0/25 cases. No value of k recovers evidence that is not in the index.

The splitter is the entire fix, and the overlap is not. This is the one place where the measurement contradicts the usual advice, so it is worth separating two things the headline number hides. A span can fail to reach the model because it was destroyed by a boundary, or because it survived intact and lost the top-k contest. Only the first is what “add overlap to fix boundary truncation” claims to solve. Which one applies is decidable without a model: chunk the corpus and ask whether the gold sentence appears whole inside any single chunk.

Switching to a recursive splitter, still with zero overlap, restores the span in 25/25 documents. All 19 of its retrieval gains are preservation, and 0 come from ranking.

That 25/25 is guaranteed by the corpus, and the guarantee is worth stating plainly, because it is half of what the bolded claim above rests on. The page already says the span is planted across character 1000, the chunk size under test, which is why the fixed splitter cuts every one. The other half is the shape of the text around it. The span is the last sentence of a paragraph roughly one chunk long, it begins a fresh sentence because the filler before it is forced to end on one, and its own padding has every sentence terminator stripped. So when the recursive splitter has to break that paragraph, it falls through \n\n and \n to ". ", and every cut point at that level lies before the span and none inside it. The fixed splitter has no notion of any of this and cuts at the character offset, which the header has pushed into the middle of the span. Both numbers follow from how the instrument was built.

The span is not planted as its own paragraph. It shares one with the filler that positions it, which is what puts it across the boundary in the first place. Adding the 200-character overlap on top preserves nothing further, because there is nothing left to preserve: every one of its 6 gains is a ranking change rather than a repair, and it also loses 1 document it had previously retrieved.

So the honest version is narrower than the advice, and narrower again once the construction is on the table. What this corpus can support is that on text whose answer sentences are paragraph-sized and free of internal sentence breaks, a recursive splitter preserves them and an overlap adds nothing to that. Real documents are not reliably like that: an answer spanning two sentences, or one longer than the chunk, has a cut point inside it and the recursive splitter will use it. This measurement says nothing about those, and a reader whose corpus looks like that should expect the overlap to matter more than it did here.

What is not built in, and is the part worth carrying away: the overlap preserved nothing further, every one of its gains was a ranking change rather than a repair, and it lost a document. Those were free to come out differently and did not.

What it does do is not random. The planted spans deliberately vary in length, and the movement is entirely stratified by that. Retrieved without overlap, then with it: 71 to 80-char spans 1/7 then 7/7; 141-char spans 6/6 then 6/6; 261-char spans 6/6 then 6/6; 421-char spans 6/6 then 5/6.

Every span the overlap rescued was 71 to 80 characters long, and the one it cost was 421. Those are measured lengths, not the class labels: the labels are 60 characters, which is the bucket a span falls in and not the length of any span in it.

The reason is checkable. Each document opens with a header line naming the section and its subject. Without overlap that header shares a chunk with the answer in 0 of 25 cases: it is always stranded in the preceding chunk. With a 200-character overlap it reaches forward into the answer’s chunk in 7, and those are exactly the shortest spans.

What decides that is not proximity but what the preceding chunk happens to contain, since an overlap carries that chunk’s tail forward and nothing else. For the shortest spans, 71 to 80 characters, the head paragraph fits in a chunk of its own, 35 to 41 characters, narrower than the overlap, so the whole header travels. For every longer class the chunk before the answer runs to 887 to 892, 828 to 835, 745 to 753 characters, whose carried tail is filler.

Config D rules the mechanism out as a necessary one. It reaches 25/25 on the boundary probe with the header in the answer’s chunk 0 times, so the probe can succeed with the header nowhere near the answer. The header is what rescues the short spans at k=3; it is not what the probe needs in general.

D cannot say what replaces it. It is recursive 512/128 against C’s 1000/200, so it moves chunk size and overlap together and a difference between them belongs to neither. That makes D a configuration rather than a control, so the missing cell was run rather than argued about. C_prime is A_prime’s chunking (1000/0) at k=8: against A_prime it isolates k, and against C it isolates the overlap.

It settles it. With no overlap at k=3 the probe retrieves 19/25. Widening k alone, the header still never reaching the answer’s chunk in any of them, brings it to 24 of 25. Adding the overlap on top of that brings it to 25, which is all of them. So the wider k is what carries this probe once the header is out of the picture, and at k=8 the entire overlap-and-header mechanism is worth 1 instance in 25.

Put the four cells together and the two levers are substitutes rather than complements. From 19, adding the overlap alone gives 24, adding k alone gives 24, and adding both gives 25. Whichever you reach for second is worth about one document. The usual advice pairs them as though they compound.

Proximity is not what decides it: those answers sit at character 917 to 927 in chunks of 997 to 1000 characters, at the far end rather than the near one, and they are the furthest of any class from their own header.

So overlap is not repairing a severed sentence here. It is dragging a heading forward into the chunk that needs it, and it does that only when the chunk behind the answer is small enough to travel whole.

Summarisation-shaped questions fail twice, and widening k only fixes the first failure, and only at one chunk size. “List every exception that applies here” needs 5 separate documents. At k=3 the stack cannot physically see them all, and retrieval is 0/25. Widening to k=8 fixes that completely: at C every one of the 25 instances has all 5 items in context. Config D holds k=8 and halves the chunk, and retrieval drops back to 22/25: a wider k is not sufficient on its own, because smaller chunks spread the same 5 items across more of them.

The answer is still wrong in 24 of those 25 cases: the evidence arrives and the model does not use it. So the failure does not clear when you fix retrieval, it relocates, and the config change that looks like the fix buys you a complete context and a wrong answer.

Adding hybrid retrieval on top looks like it helps, 1 correct at C against 4 at M, and we nearly reported it that way. The two sets of correct answers share 0 instances between them: the one the plain configuration gets right is one the hybrid gets wrong. 5 different questions are answered correctly by one setting or the other and none by both, which is movement around a floor rather than a gain.

Multi-hop questions fail on the hop you would not suspect. Each instance plants two sentences: one maps the team in the question to a zone, the other states the answer for that zone. The question averages 8.44 content words. The answer sentence shares 5.44 of them. The mapping sentence shares 2: the word “team” and the team’s own number, which is the only part of it a retriever could use to tell one instance from another.

The one that goes missing is the mapping, and at k=8 the split is stark: the answer sentence is retrieved 25/25 times and the mapping 2/25. Hybrid retrieval helps the mapping specifically, 7/25 at M, because a team name is a rare exact token and that is what BM25 is good for.

That split is a property of the wider k, though, not of the probe, and the default configurations are less tidy. At B (k=3) the answer sentence itself arrives only 16 times in 25, and the mapping 0 times; A and A_prime sit at 17 and 16 on the answer hop. So at the settings most stacks ship with, the failure is not the elegant one where the answer is found and its justification is lost. It is that neither hop reliably arrives, and the elegant version only becomes visible once k is wide enough to guarantee the easy hop.

So dense retrieval fetches the passage that looks like the question and drops the one that connects the question to it. The answer arrives without the evidence that it is the right answer, and nothing downstream can tell the difference.

The failures that matter are the quiet ones

Retrieval numbers only tell you what reached the model. What the model then does with it is where the risk lives.

Probe	Layer	A	A_prime	B	C	C_prime	D	M
control	control	100%25/25	not run	100%25/25	100%25/25	not run	not run	100%25/25
distractor	architectural	100%25/25	not run	100%25/25	96%24/25	not run	not run	96%24/25
negation	embedding	92%23/25	not run	92%23/25	60%15/25	not run	not run	52%13/25
superseded	architectural	92%23/25	not run	96%24/25	88%22/25	not run	not run	96%24/25
summarisation	architectural	0%0/25	not run	0%0/25	4%1/25	not run	not run	16%4/25
table_split	architectural	0%0/25	not run	0%0/25	0%0/25	not run	not run	0%0/25
unanswerable	architectural	100%25/25	not run	100%25/25	100%25/25	not run	not run	100%25/25

Two things surprised us, and they point in opposite directions.

The stack behaved well when evidence was missing. Given an unanswerable question, it abstained every time rather than inventing an answer. The prompt does carry an explicit NOT_IN_CONTEXT instruction, and it is tempting to credit it, but the matrix contains no run without that instruction, so what the instruction is worth here is untested.

Widening k changed what one probe’s failures look like, in both directions. Under mxbai-embed-large, negation retrieves its gold at 25/25 at every configuration, so the evidence is never missing and what follows is about the generator alone. That holds on this embedder only: under nomic-embed-text the same probe is a retrieval failure at k=3, as above, and generation never ran on it. Correct answers still fall from 23 at k=3 to 15 at k=8, and to 13 once hybrid retrieval is added, while abstentions rise from 2 to 10 to 12.

A wider k therefore costs 10 of 25 answers on this probe, and it is worth being precise about what it buys, because the previous version of this page got it wrong. It reported one confident wrong answer at k=3 and none at k=8, quoted that answer, and concluded the stack was safer and less often right at the same time. The answer was graded wrong by us, not given wrongly by the model: the grading defect is described below.

Corrected, the trap is taken 0 times at k=3 and 0 times under hybrid, which is to say never, at any configuration that generated. There is no safety gain to weigh against the 10 lost answers. What widening k bought on this probe was abstention rather than accuracy: the model stopped answering, and it was not answering wrongly to begin with.

We have no mechanism to offer for it. The obvious one, that more context makes the model more cautious, does not survive the third point: C and M both hold k=8 and differ only in which chunks arrive, yet abstentions still rise between them, and the per-instance flips run both ways.

Two probes were never caught out, for two quite different reasons. We originally reported them together as “armed, and simply not triggered”. Measuring the ranking margin shows that was one sentence covering two unlike things.

The near-duplicate distractor fires and is absorbed. Ranked by dense similarity under B, its trap chunk outranks the gold chunk in 7 of 25 instances, by margins of -0.0696 to -0.0036. In the other 18 the gold chunk leads, by 0.0125 to 0.1009. An earlier version of this sentence gave the range as -0.0696 to 0.1009, which is the range over all 25 and cannot describe the contested ones: an instance where the trap wins has a negative margin by construction. So the discrimination genuinely fails, and costs nothing here only because k is greater than one, so both chunks are fetched and “did the evidence arrive” is satisfied even when the wrong one won. Retrieving a single chunk would have missed the gold one in those 7, which follows from the recorded ranking; whether the answer would then have been wrong is untested, because no single-chunk configuration was run.

It is not quite true that it never failed. The trap was taken 0 times in 100 generated answers, but 2 replies came back empty, at k=8 and under hybrid, and the grader counts those as incorrect.

duplicate_crowding was never in contention. The target outranks all 6 near-identical copies in every instance, the closest margin being 0.0117.

That margin is partly built in, and the probe is weaker than it looks. The copies are near-identical to each other, which is the crowding mechanism and is deliberate, but they restate a different sentence than the question asks about: across 25 instances they carry 4.8 of the answer’s 9.3 content words on average, or 52%, against 10.3 of 12.3 (84%) for the distractor’s trap, which is the same sentence with the site qualifier and the value changed and every other token identical. So nothing here competed for the query, and the comfortable margin is not evidence that the stack resists crowding by content that would. What it rules out is narrower: on this corpus, crowding never produced a contest at all. Both figures are dense-similarity margins under B; the hybrid configuration ranks on a blend and is not what these describe. Crowding never begins, so calling it armed overstates it. Note also that generation never ran on this probe, so “never failed” here is a statement about retrieval only.

Empty replies are worth their own line, because nothing at runtime flags them. Across the run there are 19 of them, 12 on table_split, 5 on superseded, 2 on distractor. An empty answer is neither an abstention nor a wrong value: the grader marks it incorrect and records it as neither, so it reads as a confident miss in any table that separates the two. That matters most for table_split, where they are a large share of what looks like silent failure.

It behaved badly when evidence was present but incomplete. Asked to list every exception applying to a site, it never abstained and never once produced the whole list, across all 25 attempts. The failures split into two shapes. 11 replies named some of the required items and stopped. This one gave 3 of 5, in full:

respirator cartridges, emergency lighting circuits, eyewash stations

No hedge, no indication that anything was missing. The other 14 did something we had not anticipated, and it is the more interesting failure: they enumerated the labels they had been asked for while naming none of their content.

exception 4, exception 2, exception 5

That is a fluent, well-formed, entirely contentless answer, and no downstream check that looks for a list would notice. Both replies are rendered from the results file rather than typed.

That is the distinction the severity classes below are built on: a system that refuses is recoverable, a system that answers confidently and incompletely is not.

Severity, and why it is not a score

An earlier draft of this audit scored severity as failure rate multiplied by “silence” multiplied by “recoverability”. That number would have been indefensible: it multiplies a corpus-dependent ratio by two invented scales and produces a figure with no units. So severity here is an ordinal class, assigned mechanically from two things we actually measured.

Detectability. For each probe under each configuration we asked whether any threshold on retrieval similarity separates the failing queries from the passing ones. Read that as written: the threshold is found by sweeping the scores of the very instances it is then scored against, and the sweep is told which ones failed. There is no holdout. So this measures whether the two classes can be told apart after the fact, which is a diagnostic. It does not establish that a threshold could have been set in advance, and nothing here is a check the stack could run.

Across all 91 probe-by-configuration cells in the run, a threshold worked in exactly one: boundary under A_prime, where a cut-off at 0.83 catches 5 of 6 failures at a 5% false-positive rate, 1 of 19.

The rest split by embedder, and they have to be read separately. Under mxbai-embed-large, whose controls pass, 4 cells had enough failures and enough successes to look properly and nothing separated them, while 72 could not be evaluated at all. The second embedder contributes 14 more cells, of which 9 are measurable and none separable.

Those 14 are inside the 91 above, and that is a choice worth defending rather than hiding. They are uncertified: that embedder runs only the two embedding-bound probes, so it has no controls, and by this page’s first rule nothing certifies the instrument that produced them. Folding uncertified cells into a negative result can only strengthen it, which is why they are counted here and would not be counted in a positive one. Excluding them leaves 1 separable of 77: the same cell and the same conclusion. They are not kept out of the headline, and the count above says so.

That most cells cannot be evaluated at all is the honest headline rather than a footnote. Once the sibling confound was removed from the corpus, most probes stopped failing often enough to have two classes to compare, so “no threshold was found” and “there was nothing to look at” are different results and are counted apart.

Where we could look, the sample means in 2 of the 4 measurable cells under mxbai-embed-large came out with the failing queries scoring higher than the passing ones. That denominator is the certified embedder’s cells only, which matters two paragraphs after counting 9 more measurable cells under the other one: the inversion test was never run on those. It is tempting to report that as a finding, that the signal does not merely fail but points the wrong way. It does not support that.

Those two gaps are 0.0032 and 0.0080 on classes of 10 to 15 points. A two-sided permutation test on the difference of means (20,000 reshuffles, seed 17, so the p-values recompute from the archive) puts them at p = 0.81 and p = 0.25. That is what chance produces at this size, and 0 of them survives testing.

Of the 5 cells with both classes large enough to test, 1 is distinguishable from chance: boundary under A_prime, where failures score lower, which is the useful direction and is the same cell where a threshold worked. Two more cells sit just the wrong side of the line, both pointing the useful way: multihop at p = 0.0621 and paraphrase at p = 0.0506. Neither clears the threshold and both are reported as not distinguishable, but a summary that leaves them out would overstate how empty the picture is.

So the honest reading is not that the signal is inverted. It is that there is very little signal, the one place it clearly appears is the one cell where the two classes were separable at all, and two more places hint at it without reaching significance. Calling that cell already usable would claim a deployable threshold on the strength of 6 failures and 19 passes that the threshold was chosen from.

Recovery. We then implemented the obvious cheap guardrail, checking that a chunk the model cited actually contains the value its answer asserts, and measured whether it caught anything. The check uses only the model’s output and the retrieved chunks, never the answer key, because a guardrail that consults ground truth is not a guardrail.

Across the 53 failures at B on the probes that generated an answer, it caught 0. Not a low number: none. Every apparent catch was an abstention, 17 of them, and an abstention is a failure that has already announced itself.

That is this configuration. It is not the whole run, and the difference matters enough that it is set out per configuration at the end of this page: the check catches 1 at M and flags a further 19 across C and M for citing nothing at all.

The remaining 36 slipped through, and they split into two kinds that are worth keeping apart. In 19 the check ran and was satisfied: a cited chunk genuinely did contain the value the answer asserted. Those break down as 14 summarisation, 1 superseded, 4 table_split, so the majority are not a misread table row but a list answer naming exception labels rather than their content, which the check confirms because the label number is right there in the cited chunk. In the other 17 there was nothing to check at all, because the answer asserted no value the test could look for, so it passed by default. The second kind is the more troubling: a guardrail that is silent because it found nothing wrong is indistinguishable, in the log, from one that is silent because it never ran.

One instance shows what that costs. Asked which interval the current policy sets, with a superseded policy for the same site also in the corpus, the model answered:

Every 28 days.

The current policy says 38. It answered 28, the superseded figure, and cited sup-018-old#0, which is the outdated document. The citation check passed, because that chunk really does contain the number the answer asserts.

Every signal a production system has says this reply is sound: it is specific, it carries a citation, and the citation resolves to a chunk containing the value. Nothing short of knowing which policy supersedes which catches it, and vector search has no recency channel to know that with.

The finding is therefore stronger and less comfortable than the one it replaces: the standard citation check caught 0 of the 53 failures at B. It looks useful only against failures that had already announced themselves, and those it gets no credit for, because an abstention is visible without it. Against the quiet ones it offers nothing. An earlier version of this sentence said it earned its keep on the loud ones, which credits it with catches that were abstentions.

What this supports, and what it does not

Supported. In a default-configured local RAG stack, with the components, prompt and corpus published in the archive, each failure mode above either triggers reproducibly or is shown not to trigger here. duplicate_crowding never fired at all, and unanswerable never produced the fabrication it exists to provoke: the stack abstained on every one. Those are results, and they are the reason this paragraph no longer says every mode is triggerable.

What can be said about signals is narrower than it sounds. Of the 77 probe-by-configuration cells under the certified embedder, 72 have too few failures or too few successes to compare at all; for the 5 that do not, we can say whether a retrieval-score threshold separated them after the fact, which is a diagnostic and not a signal the stack could act on. The cheap check ran only where an answer was generated, which is 4 of the 7 configurations and 7 of the 11 probes.

What holds without qualification is the last part: how the counts moved under single-variable configuration changes.

The components are named, not pinned. model_digests in the results file records 3 of 3 models as unavailable: this run replayed entirely from cache and the models had been removed from the host by the time it ran, so the harness could not ask for their digests. It records that rather than an empty object, which is what it used to do and which read as though no models were involved. A reader reproducing this pulls the models by name, and gets whatever version is current.

The conclusions this section replaced were the one place on the page that survived every review unedited, while the probe-level findings above them were retracted one by one. They went on asserting what those retractions had removed.

Not supported. Any claim about how often these happen in the wild. The corpus is a purpose-built instrument, so the rates are properties of it, not prevalence estimates. What the instrument does support is that each rate is out of that many observations, and we measured that rather than assuming it. Across the 10 probes with answer-bearing documents, the closest pair of instances inside any probe overlaps 0.77 against a 0.8 threshold, once the tokens a probe’s own documents all share are removed, so no probe is one observation reported twenty-five times. Across probes the closest pair is 0.58, against 1.00 for two probes built from the same documents, so negation and summarisation are two instruments rather than one under two names. The 1 probe left out is unanswerable, whose instances name no answer-bearing document because there is no answer to hold one; its 25 questions are still distinct. The transferable results are the config deltas, the detectability findings, and what the cheap citation check did and did not catch, which is worth stating per configuration rather than in one word.

At B it caught 0 of the 53 failures there. That is the number discussed above and it is one of the 2 configurations, of 4, where the check looks least useful; A is the other: at C and M the same check is not idle. It flags 12 failures at C and 7 at M for carrying no citation at all, and at M it catches 1 outright: an answer that was neither empty nor an abstention, citing a chunk that does not contain the value it asserts. That is the check doing exactly the job it was added for, once in 60 failures.

Neither “the check caught nothing” nor “B is the configuration where it caught nothing” is what the run says. The rollup at B is one of the 2 configurations where the count is zero, not the only one, so a result stated at one configuration does not transfer to the others by itself.

Three limits worth stating plainly.

The corpus hands every retriever a shortlist, for the reason given above the first table, so read the retrieval columns as an upper bound rather than a forecast.

The hybrid configuration M is not a clean sweep: it still leaves multihop at 7/25 and table_split at 0/25. The two are stuck for different reasons, and neither is a dead end.

Multi-hop is not out of reach: hybrid retrieval is what lifts its mapping sentence from 2 to 7 of 25, so the query does reach it, just rarely. We did not vary the lexical weight, widen k past 8, or try a second retrieval pass, so “unfixable” is not something this matrix can support.

table_split is stuck for a different reason. Its answer row and the header naming its columns land in the same chunk in at most 0 of 25 instances under any of the 7 configurations tested. Widen the chunk to 4000 characters and they reunite 25 times, so the severance is caused by chunk size and nothing else.

We assumed that made it fixable, and it does not. Running the probe at that chunk size outside the matrix, retrieval reaches 25/25 and the answer is still wrong in 23 of 25, with 0 abstentions.

The transcripts are worth reading rather than summarising. Of those 23 wrong answers, 22 name the asset the question asked about and 23 give a number. The model is not ignoring the reunited table; it is reading it and taking the value from the wrong row. That relocation is the same one seen with summarisation, and a worse version of it: the retrieval problem is real, solvable, and once solved the failure becomes harder to spot rather than easier.

4 of the 23 do something worse still. Somewhere before giving a number, they say the asked asset is not in the tables at all, and then supply a number from a different asset’s row. That count is a case-insensitive match over the regular expressions listed in wrong_denying_the_record_patterns in the results file. A denial worded differently is missed, and a match is not proof on its own, so all 4 were read: each denies the record for the asset the question asked about.

Only 1 generator was tested, qwen2.5:7b, a small local model. Nothing here predicts how a frontier model would behave on the same prompts. 2 embedders were run, mxbai-embed-large and nomic-embed-text, but the second only on the probes that indict the embedding model. That was a budget decision and it has a price this page pays twice: no controls were run under it, so by the rule at the top nothing certifies its instrument, its 14 detectability cells are reported apart and could only ever strengthen a negative result, and the table using it is marked uncertified. Running it everywhere would have bought exactly that certification, which is the thing skipping it cost.

low is the only class here that is neither none nor undetermined, and it means different things for the two probes that hold it. For table_split it is a property of one configuration rather than of the probe. It retrieves nothing anywhere, so its evidence never changes, and yet how it fails moves a great deal. At B it abstains 15 times and returns 6 empty replies against 4 confident wrong answers, which is what earns the low grade. At k=8 the same probe abstains 4 times with 4 empties and answers confidently and wrongly 17 times, and under hybrid 17 times again. Nothing about the evidence changed; the failure simply got quieter, and a grade assigned at one configuration would have read as a fact about the probe.

For negation the same grade is stable rather than configuration-bound: it is low at every configuration that generated, because the trap is taken nowhere and its failures are abstentions. Two probes share a class for opposite reasons, which is why the class is reported per configuration and not as a property of a probe.

That is the mirror of the negation result above, on the same lever. Widening k made negation louder and table_split quieter, and we cannot explain either.

And the severity classes are mostly a report of what could not be established. The classes are assigned mechanically. What follows is one entry per branch of the assigning function, in the order the branches are tested. The function emits that list into the results file; the copy below is checked against it string for string by a test, rather than rendered from it, because the renderer that evaluates these pages cannot take a list expression. undetermined appears three times because there are three distinct reasons a cell reaches it, and they are listed apart because collapsing them into one bullet drops a branch: a reader applying the shortened rules in this order reaches critical on a cell that is graded undetermined. Each cell in the results file records which branch assigned it, so any class here can be checked against the rule that produced it:

• none No failures under this configuration. Not a claim that none exist elsewhere.
• undetermined Generation never ran on this cell, so there is no answer to classify.
• undetermined The controls for this embedder were never run, so by the first rule on this page nothing certifies the instrument that produced the cell.
• low A loud failure. At least 80% of the failures abstained, errored, or returned nothing, rather than answering confidently and wrongly.
• medium Wrong, but the failures and successes are separable in this sample by a retrieval-score threshold. That threshold is fitted on the same instances it is scored against, with no holdout, so this is a diagnostic rather than a deployable check.
• high Wrong and with no separating signal, but the cheap check flagged most of them: either the chunk they cited does not contain the value they assert, or they cite nothing at all. Abstentions are excluded, because a failure that announces itself is not one the check saved you from.
• undetermined Wrong, quiet and uncaught, but one of the two classes is too small for that to mean anything: fewer than 5 failures or fewer than 5 successes, so ‘no signal was found’ and ‘there was nothing to look at’ cannot be told apart. Tested after the three classes above, which is why a cell can be wrong, quiet and uncaught and still not reach critical.
• critical Wrong, confident, no separating signal, not caught by the cheap check, and both classes large enough for those to mean something. Nothing in this run reached this class.

Of the 7 probes where an answer was generated at B: 3 none (control, distractor, unanswerable); 2 low (negation, table_split); 2 undetermined (summarisation, superseded).

Nothing reaches critical, so there is no ordering to show. summarisation is the case that looks most like the critical class, wrong on every one of its 25 attempts with no abstentions and nothing catching it, but the class needs a comparison against passing cases and there are none: 0 of 25 were correct. “We could not establish this” is the result, and it is duller than the failure it looks like.

Whether any of this holds for your stack is exactly what this experiment cannot tell you, because it depends on your corpus, your chunker and your queries. That is what the automation audit is for.

What we got wrong

This page was published and withdrawn on its first day, then went through repeated adversarial reviews before this version, each one reading the results file rather than this page. Every number on it recomputed correctly in every one of them. Everything else did not, and the failures fall into kinds rather than one kind, which an earlier version of this section got wrong by claiming they were all explanations gone astray.

The count of reviews is deliberately not given. It was written as a number, went stale at the next review, and stayed wrong through several more, which is a small example of the thing this section is about: a figure typed once and never revisited.

We asserted things that were not true. A model reply was quoted “in full” that appears in no generation record, left behind when the numbers around it were regenerated. The distractor probe was described abstaining rather than reporting the wrong figure, behaviour that never occurred in any run. A second generator was claimed although only one was ever pulled. The cheap citation check was said to reliably catch the missing-evidence failures when it caught none of them. A severity ordering was promised as a transferable result while no severity was rendered anywhere. The harness was described as refusing to time a contended host before it did so. The rerun command named configurations the published run had not used, so following it would have produced a file disagreeing with this page. One probe’s lookup key was called the sharpest in the corpus when a large minority of questions tie at the same selectivity. And the rendered report inside the archive twice made claims this page retracts, once because it was stale and once because the generator had never been updated after the finding changed. The single confident wrong answer this page reported, and quoted, was our grading error rather than the model’s: the reply named the store the question asked about and said the item was not permitted there, and the grader looked for “not permitted” as one phrase, missed it across the store name, and matched the trailing “permitted” instead. It was the only trap this probe ever recorded, and a claim about widening k buying safety rested on it. An earlier version of that retraction called it the only trap taken in the run, which is a different and false claim: the run records 29 on summarisation and 2 on superseded, and one of the latter is quoted on this page as an exhibit.

We explained results with mechanisms we had not checked. The overlap finding went through five accounts before this one: that overlap repaired a severed span, when the span was already whole; that it gave the span a second, shorter chunk, when there is one chunk either way and overlap makes it longer; that the movement was noise, when it is stratified entirely by span length; that header propagation was refuted, by a check that could not tell the difference because the planted sentence names the section code itself; and that the answer sits near a boundary, when it sits at the far end of its chunk. Multi-hop was blamed on the hop that always arrives rather than the one that goes missing. Summarisation was written up as barely working because k was too small, which is the comfortable reading, since k is a one-line change and the real answer is not.

We built measures that could not see what they were asked about. Identifier leverage matched tokens that looked like identifiers instead of measuring how far a key narrows the index, so a probe whose key is as sharp as any in the corpus was reported as having none at all. The multi-hop overlap count tokenised on letters only, so the discriminating part of a team name was invisible to the very measure used to explain it. The list classifier matched one phrasing rather than asking whether any required item was named.

We let single numbers stand for several unlike things. A guardrail’s “caught” total bundled abstentions, which it gets no credit for, with real catches, of which there were none. “Never fired” covered both a probe whose mechanism fires and is absorbed and one never in contention. Severity graded cells where no answer had been generated at all. The detectability headline counted one embedder while claiming to cover the matrix.

And the published artefacts drifted from the page. The downloadable archive twice shipped material from the run withdrawn on day one, the second time a rendered report asserting severities this page retracts. It also carried the name of an unrelated project sharing the machine. Raw model output was missing from the record, so a reader could not check why any verdict was assigned.

Two of these, the reply quoted in full and the abstention that never happened, are the reason quotes are now selected by rule from the run rather than typed. An earlier version of this section omitted both, along with the second-generator claim, while keeping the tidier stories about faulty reasoning. That omission was not deliberate and is worth naming anyway: a section about your own errors is written by the same judgement that produced them.

Rerun it

The archive contains the harness, the corpus generator, the graders and the full per-instance records. It needs a local Ollama.

The results file this page renders was written on 2026-09-03, which is the date at the top of this entry: the entry follows the run rather than the other way round, and a test refuses a publication date earlier than the run it reports. Note what that timestamp does and does not mean. It is when the file was written, and every model call in it came from cache, so the answers themselves were computed earlier. How long it takes depends on your hardware, and we do not quote a figure. The harness withholds elapsed_seconds when it detects another workload sharing the GPU, or when the run was served from cache, and one of those has applied to every run we have made, so we have no honest timing to give you. What “detects” depends on is spelled out below.

tar -xzf rag-failure-audit.tar.gz && cd lab
uv sync --group dev && uv run pytest        # verify the instrument first
uv run python -m ragaudit.run --n 25 --gen-configs A B C M

elapsed_seconds in the published results file is null, and that is the refusal firing: every model call was served from cache, so this was a replay. The check runs before and after the matrix, and a run contended at either end publishes no timing at all, when it is armed. It matches a process pattern taken from an environment variable, and the archive ships that variable unset: naming the workload it watches for would disclose what else this machine runs, which is the one thing this site will not publish. So a rerun detects nothing unless you configure it, and conditions.contention_detection_configured records which kind of run produced any given file. On this one it is true.

There is a second reason not to have trusted it. Model calls are cached, and the published run was replayed from a warm cache after a crash lost the original write: 14,597 calls served from cache and 0 computed. The answers and scores are the ones the models actually produced, and a rerun on your machine will be slow the first time and near-instant after.

The tests are not decoration. One of them asserts that the planted span really does straddle the chunk boundary, because an early version of this corpus placed it just past the split and silently measured nothing. Another asserts that probe instances are not near-duplicates of each other, because the first full run was invalidated by exactly that: retrieval was returning arbitrary sibling documents, and the boundary probe was quietly measuring whether the embedder could tell one identifier from another. Both bugs were found by looking at what the stack actually retrieved rather than at the summary table.