AI document extraction: the confidence score is not calibrated

Every document extraction product ships a confidence score, and every buyer is told to route on it: auto-approve above the threshold, send the rest to a human. It is a good pattern. It is also usually sold with an implied claim that nobody checks, which is that the score tracks correctness.

We measured that claim on receipts. Figures from our confidence-routed extraction study.

Controls first

Clean US-format receipts, the case the pipeline must handle trivially, went first. The extractor got 30 of 30 fields right with 0 magnitude errors. A router measured on a pipeline that cannot read a plain receipt is measuring its own noise.

The routing works

454 of 600

field-pairs were auto-approved and 146 went to review. The error rate among the ones sent to a human is 89.0%, against 46.5% among the ones shipped.

So the split is doing real work. The queue a person sees is much denser in errors than the pile that shipped. That is the whole job of a router and it is being done.

The score is still not monotonic with correctness

Now look at accuracy by confidence bin:

confidence	accuracy
= 0.0	173/349 (49.6%)
[0.5, 0.67]	1/50 (2.0%)
(0.67, 1.0)	0/24
= 1.0	85/177 (48.0%)

Accuracy at maximum confidence is 48.0%. Accuracy at zero confidence is 49.6%, which is higher.

A score that is not ordered with respect to correctness is not a probability, whatever the API calls it. You cannot read “0.9” as “90% likely right”, and you cannot compare two documents by their scores.

Why routing works anyway

Because routing does not need calibration. It needs the errors to be concentrated somewhere, and here they are: the score is a grounding check, and grounding correlates with a specific failure, the model inventing a value that is not in the document.

The score concentrates on a handful of values. The two commonest carry 526 of 600 pairs. It is closer to a set of flags than a scale.

The failure it does catch

34 factor-of-1000 errors were recorded, the kind where a decimal separator is read as a thousands separator or the reverse. One the detector logged, on the subtotal field: |28.182 - 28182.0| > 0.01. 34 of 34 were routed to review and 0 shipped. Containment on that class was complete in this run.

That is worth having. A magnitude error in a finance pipeline is the error that costs money, and it is contained not because the score is calibrated but because the grounding check happens to fire on exactly that shape of mistake.

The failure it does not

Of the auto-approved errors, 96 of 254 nulls were wrong: the model returned nothing where the document had a value. Grounding cannot fault a null, because there is no asserted value to check against the text.

So the honest description of this router is narrower than “it catches inventions”. It contained one specific class completely, the magnitude errors above, and it is blind to omissions by construction. In between, plenty still shipped: 115 of 200 auto-approved filled values were wrong as well. A router that concentrates errors is not a router that removes them.

What to take from this

1. Do not read a confidence score as a probability unless someone has shown you a reliability curve. Ask for accuracy by bin.
2. A router can concentrate errors while the score behind it is uncalibrated. Those are separate questions and they get conflated. Concentrating is not removing: 46.5% of what shipped here was still wrong, so whether that is deployable depends entirely on what the wrong ones cost you.
3. Ask which failure the score is blind to. Every grounding-based score is blind to the omission, and omissions are silent by definition.

The full study, including the precision-coverage curve and what a rerun with a different scoring function did, is in confidence-routed extraction.

Deciding what a review queue should actually contain is part of our agentic workflow automation work.