Tag
10posts tagged Evaluation. All posts
A confidence score that routes documents well can still be uncorrelated with correctness. Measured on receipts: accuracy at 1.0 was lower than at 0.0.
We added BM25 to a dense retriever across ten probes. It moved two of them, and seven were already saturated. What that means before you build it.
Stock Presidio measured on real court judgments and synthetic PII: entity recall, the partial-redaction trap, and what the defaults do not catch.
Scanned PDFs passed every document-level quality check we threw at them while whole pages extracted as one character. Where the loss actually hides.
Raising the Presidio score threshold reads like a precision knob. Measured across a full sweep it is a cliff, and names fall off it in one step.
A benchmark that cannot fail is measuring whether you wrote a hard question. The armed-probe rule, and the counts that show whether yours is armed.
Changing the splitter recovered 19 of 25 answers. Adding overlap on top recovered 6 more, and none of them for the reason overlap is sold on.
A method for evaluating RAG that survives a real corpus: armed probes, controls first, and retrieval failures counted apart from generation failures.
Saturation, partial credit, abstention counted as a catch, and detectability without power. Four ways a RAG metric reports a result it did not earn.
We built the cheap guardrail, checking a cited chunk holds the value the answer asserts, then measured it against the answer failures of four configs.