A useful review router with an uncalibrated confidence score

Controls before findings

Clean US-format receipts, the case the pipeline must handle trivially, are reported first and are the first thing to check. The extractor got 30 of 30 fields right with 0 factor-of-1000 errors. If those had not passed, nothing below would mean anything: a router measured on a pipeline that cannot read a plain receipt is measuring its own noise.

Beside them sits the locale probe, which is a control in the other direction. The same Indonesian-format documents run twice, once under the naive numeric prompt and once under the corrected one, so the failure mode reported later has a before and an after on identical inputs rather than an assertion.

• Naive prompt: 18 of 25 correct, 6 factor-of-1000 errors.
• Strings prompt: 25 of 25 correct, 0 factor-of-1000 errors.

The probe is weaker than its count suggests, and here is how much. Its 5 documents fall into 2 distinct line structures once the numbers and names are discarded, the larger holding 3 of them, so 3 carry no shape the others do not. The single word PROBO appears in

5 of 5

titles: one naming family, as well as one small set of shapes. The 6 controls are 3 structures on the same measure. Vocabulary overlap says the opposite, and that is the point: token similarity across the controls peaks at 0.23, because the words in each slot differ while the slots do not, so a measure that reads the nouns cannot see this. The archive ships the documents under runs/control_docs. The before-and-after is still a genuine pair on identical inputs, which is what the comparison rests on, but a rerun should widen the shapes before the probe is asked to carry more than that.

Every figure on this page is rendered from the results files in the archive below, not typed in by hand.

The claim worth making

A document extraction pipeline scores each field’s confidence and routes low-scoring fields to a human review queue. The tempting story is “we built a router and it works”. The measured story is more specific and more useful: the binary auto/review split does real work while the confidence score behind it is not calibrated, and knowing which half to trust is what makes the system deployable.

The pipeline extracts receipt fields with a local vision model, scores each field by whether its value is actually printed in the document text (grounding) and optionally by agreement across repeated samples, and flags fields below a threshold. Grading against ground truth is deterministic throughout: currency comparison within a fixed absolute tolerance, multiset F1 over line items, no LLM judge anywhere. The grader is croute.grading, deterministic, independent of the source eval harness.

What the measurement showed

The committed run is cord_v2_qwen7b_vision_v2 (committed predictions.jsonl, 100 docs), scored as one (document, scored field) pair; 6 scored fields per document, giving 600 pairs of which 400 have a value in the document at all.

• The score is not monotonic with correctness. By the bins the run recorded, [0.5, 0.67]: 1/50 correct and (0.67, 1.0): 0/24 correct, against = 0.0: 173/349 at the bottom of the scale and = 1.0: 85/177 at the top. The middle of the scale is far worse than the bottom.
• The binary split still separates. Auto-approved fields were wrong 211 of 454 (46.5%); fields routed to review were wrong 130 of 146 (89.0%). Restricted to fields where the document really carries a value, the split is 186 of 271 (68.6%) against 119 of 129 (92.2%).
• The failure mode that matters most was routed, every instance the detector saw. Indonesian receipts group thousands with a dot where an English reader expects a decimal point. A model that emits such a total as a JSON number has it collapsed by the JSON parser before any downstream code can react, a silent factor-of-1000 error: the run’s first recorded instance is field total_price of document cord_test_0005, graded |31.0 - 31000.0| > 0.01. The run contained 34 errors the detector scored as this shape and the router sent 34 of 34 to review, by two different routes rather than one: 32 carry the half-credit confidence the grounding scorer assigns an ambiguous numeric match, and 2 scored 0 and were routed as ungrounded. The detector takes the ratio on absolute values, so a magnitude error that also flipped sign counts here too; one of the 2 is that case. A decimal point need not be involved at all: a collapsed dotted thousands group and a sign error produce the same magnitude error.

The confidence score concentrates on a handful of values: the four commonest carry 578 of 600 pairs. So the precision/coverage curve, which operating threshold ships how much at what error rate, offers very little room.

Every factor-of-1000 error the detector recorded scored at or below 0.5. Above that, across the 16 operating points the curve holds, coverage runs from 177/600 (29.5%) to 210/600 (35.0%), the error rate never drops below 92/177 (52.0%), and 0 of these errors ship. At the threshold itself coverage is 251/600 (41.8%), which is better, and 32 of these errors ship with it. That figure is the point at the threshold, not the top of the safe range: quoting it as the range makes the usable band look wider than the run supports. The near-flat curve is the honest cost of a score built from grounding alone.

Two things that containment does not say, and both are easy to assume. It is a property of this run’s score, which is grounding on its own; the rerun below blends grounding with agreement, and on that scale 28 field-pairs take values from 0.65 upward that sit above 0.5 without being certain, so a threshold chosen here does not carry over to the configuration this page recommends. And it is a property of the errors the detector recorded: the rule is a ratio test on two parsed numbers, so a collapse it cannot parse, or one where the document’s value was never annotated, is not among the 34 it recorded.

The fix that took a prompt, not a model

The factor-of-1000 collapse is two-part, and the fix starts before parsing: numeric fields are requested from the model as strings, copied exactly as printed, and a structural parser then commits to a reading (a separator followed by groups of three digits to the end is a thousands grouping). The locale probe reported above is that comparison: the naive prompt produced 6 factor-of-1000 errors in 25 fields and the strings prompt produced 0 in 25, on the same documents with the same model. A pinned unit test proves the mechanism fires under the naive configuration and not under the corrected one.

Rerunning the ported pipeline over the first 50 CORD documents as cord_strings_agreement3 (croute pipeline, numbers-as-strings, 3 samples). That subset is not a sampling decision. The results file records the reason: Documents are the first N of the CORD test split in id order; the run was truncated under host contention if N < 100. The source-run comparison below is restricted to the same documents. Here N is 50. The run also recorded no manifest, so 0 of its 4 configuration fields are attested by the artefact; the label above is what the measurement script was told, not what the pipeline recorded. The comparison below is against the same documents, which answers the truncation and not the missing configuration: if the run did not use the settings its label claims, the predictions themselves differ and pairing the documents does not recover that. So the headline rerun is half the corpus, compared fairly on that half, under a configuration the run did not write down.

What the comparison shows: 0 factor-of-1000 errors against 16 for the original run on the same documents, 145 of 300 field pairs correct against 134 of 300, and a confidence score with 17 distinct values instead of 12. The score still is not calibrated ((0.67, 1.0): 3/27 correct, worse than either endpoint at = 0.0: 5/24 and = 1.0: 52/91), which is the finding: agreement sampling makes the score finer, not truthful, and the binary router is what carries the system.

How the shared severity criteria grade it

The same analysis this site applies everywhere was handed production-only signals, the field confidence as the runtime score and grounding as the deployable check. It graded all 4 analyses critical: 2 runs, each on two populations, rather than 4 separate runs.

On the committed source run that is 600 instances, of which 341 failed, and the check caught 240 of them at 174 of 259 correct fields wrongly flagged: a false-positive rate of 67.2%. That figure is inflated by fields the model correctly left empty, which score zero and so look flagged. Restricted to fields the document actually fills, the same check flags

10 of 95

(10.5%) and catches 95 of 196. Both populations are reported because publishing only the first would bake the artefact into the headline. The detectability measurement found no threshold that separates the two groups, which the run records as a separability of false. That measurement consults ground truth to decide which fields failed, so it is a diagnostic and not a check the pipeline could run.

What the unit is, and how many of them there are

Every rate on this page is over one population, and it is worth naming before the rates start.

one (document, scored field) pair; 6 scored fields per document

600 pairs were scored in the source run. 400 of them have a gold value to be right or wrong against; the remaining 200 are fields the document does not carry, where the correct answer is nothing.

That split matters more than it looks. A pair with no gold value can only be got wrong by inventing something, and a pair with one can be got wrong in two directions. Where a figure below is restricted to the gold-carrying subset, it says so.

Grading is croute.grading, deterministic, independent of the source eval harness. No model judges another model’s output anywhere on this page.

The locale probe, which is a control in the other direction

The negative control asks whether the pipeline breaks on an easy document. The locale probe asks whether it breaks on a document that is only unusual, and it runs the same documents twice under two prompts.

	naive numeric prompt	numbers as strings
fields graded	25	25
correct	18/25 (72.0%)	25/25 (100.0%)
factor-of-1000 errors	6	0

Same documents, same model, same fields. The only change is how the prompt asks for numbers, and it takes the magnitude errors from 6 to 0 of the same 25 fields.

This is what an armed control looks like: the failure fires under one condition and not under the other, on identical inputs, so the mechanism is demonstrated rather than asserted. Without the naive column, the corrected column is just a pipeline that happened to work.

The documents themselves, field by field, under the naive prompt:

document	field	predicted	gold	correct	reason recorded
probe_warung	total_price	31000	31000	true	none
probe_warung	cash_paid	50000	50000	true	none
probe_warung	change	19000	19000	true	none
probe_warung	subtotal	31000	null	false	predicted 31000.0 but gold is empty
probe_warung	tax	null	null	true	both empty
probe_toko	subtotal	30000	30000	true	none
probe_toko	tax	3000	3000	true	none
probe_toko	total_price	33000	33000	true	none
probe_toko	cash_paid	null	null	true	both empty
probe_toko	change	null	null	true	both empty
probe_kios	total_price	12500	12500	true	none
probe_kios	cash_paid	20000	20000	true	none
probe_kios	change	7500	7500	true	none
probe_kios	subtotal	null	null	true	both empty
probe_kios	tax	null	null	true	both empty
probe_resto	subtotal	20.909	20909	false	|20.909 - 20909.0| > 0.01
probe_resto	tax	2.091	2091	false	|2.091 - 2091.0| > 0.01
probe_resto	total_price	23	23000	false	|23.0 - 23000.0| > 0.01
probe_resto	cash_paid	null	null	true	both empty
probe_resto	change	null	null	true	both empty
probe_depot	total_price	46.75	46750	false	|46.75 - 46750.0| > 0.01
probe_depot	cash_paid	100	100000	false	|100.0 - 100000.0| > 0.01
probe_depot	change	53.25	53250	false	|53.25 - 53250.0| > 0.01
probe_depot	subtotal	null	null	true	both empty
probe_depot	tax	null	null	true	both empty

The confidence score, value by value

The score is not a scale. It is a small set of values with very different populations behind them.

confidence	pairs
0.0	349
1.0	177
0.5	41
0.75	11
0.6667	7
0.625	2
0.9333	2
0.7	1
0.825	1
0.8333	1
0.875	1
0.9083	1
0.9091	1
0.92	1
0.975	1
0.9833	1
0.9875	1
0.9944	1

18 distinct values across 600 pairs, and the four commonest carry 578 of 600 of them. Anything between the two poles is a long tail of single pairs.

A score shaped like that cannot support the operations a score is usually put to. You cannot rank two documents by it when most of them share a value, and you cannot read a threshold off it when moving the threshold by a hundredth steps over a cliff.

The precision-coverage curve, in full

One row per operating threshold: how much ships, how much of what ships is wrong, and how many factor-of-1000 errors go with it.

threshold	auto-approved	wrong among approved	factor-of-1000 errors shipped
0	600/600 (100.0%)	341/600 (56.8%)	34
0.5	251/600 (41.8%)	165/251 (65.7%)	32
0.625	210/600 (35.0%)	125/210 (59.5%)	0
0.6667	208/600 (34.7%)	123/208 (59.1%)	0
0.7	201/600 (33.5%)	116/201 (57.7%)	0
0.75	200/600 (33.3%)	115/200 (57.5%)	0
0.825	189/600 (31.5%)	104/189 (55.0%)	0
0.8333	188/600 (31.3%)	103/188 (54.8%)	0
0.875	187/600 (31.2%)	102/187 (54.5%)	0
0.9083	186/600 (31.0%)	101/186 (54.3%)	0
0.9091	185/600 (30.8%)	100/185 (54.0%)	0
0.92	184/600 (30.7%)	99/184 (53.8%)	0
0.9333	183/600 (30.5%)	98/183 (53.5%)	0
0.975	181/600 (30.2%)	96/181 (53.0%)	0
0.9833	180/600 (30.0%)	95/180 (52.8%)	0
0.9875	179/600 (29.8%)	94/179 (52.5%)	0
0.9944	178/600 (29.7%)	93/178 (52.3%)	0
1	177/600 (29.5%)	92/177 (52.0%)	0

The run records 18 thresholds evaluated and 18 distinct operating points, with degeneracy false. That check exists because a curve whose points collapse onto each other looks like a curve and offers no choice, and a page publishing one without saying so is offering a dial that does not turn.

What the table shows is worse than degeneracy. The curve turns, and everywhere it turns the error rate among approved pairs stays in the same band. Buying coverage costs accuracy at roughly the rate you would get by approving at random within this population.

The rerun, and what changed

The fix is a prompt change, not a model change, and it was rerun as its own configuration rather than argued about.

	source run	rerun, same documents
documents	50	50
pairs	300	300
correct	134/300 (44.7%)	145/300 (48.3%)
auto-approved	101/225 wrong (44.9%)	101/231 wrong (43.7%)
routed to review	65/75 wrong (86.7%)	54/69 wrong (78.3%)
factor-of-1000 errors	16	0

Documents are the first N of the CORD test split in id order; the run was truncated under host contention if N < 100. The source-run comparison below is restricted to the same documents.

The comparison is restricted to the documents both runs cover, which is the only way to compare them: the rerun was truncated under host contention, and pairing on the union would let the extra documents in the longer run move the difference.

The rerun’s score is a different scale from the source run’s, and that is the thing most likely to be misread here. The source run’s confidence is grounding alone. The rerun blends grounding with agreement across 3 sampled runs, producing 12 distinct grounding values underneath it. A threshold chosen on one does not carry to the other.

It cost 60,288 input tokens and 23,372 output tokens to produce, which is the price of sampling the same document three times.

What the rerun did not record

The rerun’s manifest was never written. The run file carries the excerpt, and it is empty:

field	recorded
numbers_as_strings	null
self_consistency_samples	null
model	null
vision_model	null

So the configuration this page names for the rerun, numbers as strings with 3 samples, is read off the run’s label rather than off an attestation inside it. That is weaker than the source run, where the settings are recorded, and it is stated here rather than quietly presented as equivalent evidence.

This is also why the comparison above is restricted to shared documents and not presented as a clean before and after: an unrecorded configuration and a truncated run are two separate problems, and pairing the documents answers only the second.

Severity across every run graded

run	instances	failures	caught by the deployable check	false positives	class
source_vision_v2	600	341	240/341 (70.4%)	174	critical
source_vision_v2_filled_only	291	196	95/196 (48.5%)	10	critical
croute_strings_agreement3	300	155	102/155 (65.8%)	90	critical
croute_strings_agreement3_filled_only	139	76	23/76 (30.3%)	8	critical

Signals handed to the shared analysis are production-only: the field confidence as the runtime score, grounding >= 0.9 as the deployable citation check. Severity criteria are the lab's published ones.

Two of those four rows are not separate runs. The filled-only entries are a second population of the same run, the pairs where the document carried a value, and they are listed separately so that a reader counting rows does not count them as independent evidence.

Accuracy by confidence bin, and the subset that removes the null problem

The bins above are over every pair. A large share of those have no gold value, where the correct answer is nothing, and a model that returns nothing everywhere would score well on them. So the same bins are recomputed over the gold-carrying subset alone.

confidence	all pairs	gold-carrying pairs only
= 0.0	173/349 (49.6%)	9/177 (5.1%)
(0.0, 0.5)	no pairs	no pairs
[0.5, 0.67]	1/50 (2.0%)	1/47 (2.1%)
(0.67, 1.0)	0/24 (0.0%)	0/24 (0.0%)
= 1.0	85/177 (48.0%)	85/152 (55.9%)

Restricting to gold-carrying pairs does not rescue the ordering. It makes it worse: the bins that looked mediocre over everything look worse once the pairs that could only be failed by inventing something are removed.

The same two views for the rerun:

confidence	all pairs	gold-carrying pairs only
= 0.0	5/24 (20.8%)	0/19 (0.0%)
(0.0, 0.5)	85/157 (54.1%)	8/76 (10.5%)
[0.5, 0.67]	0/1 (0.0%)	0/1 (0.0%)
(0.67, 1.0)	3/27 (11.1%)	3/26 (11.5%)
= 1.0	52/91 (57.1%)	52/75 (69.3%)

Where the auto-approved errors come from

An error that ships is not one kind of thing. Splitting the auto-approved failures by whether the model returned a value or returned nothing:

auto-approved pairs	count	wrong
a value was returned	200	115/200 (57.5%)
nothing was returned	254	96/254 (37.8%)

The two halves need different fixes and only one of them is visible to a grounding check. A wrong filled value asserts something that can be looked for in the document. A wrong null asserts nothing, so there is nothing to ground, and no confidence score built on grounding can distinguish it from a correct null.

That is the structural limit of this router stated as a count rather than as an opinion.

Every factor-of-1000 error the detector recorded

document	field	confidence	comparison the detector logged	routed to review
cord_test_0005	total_price	0.5	|31.0 - 31000.0| > 0.01	true
cord_test_0005	subtotal	0	|28.182 - 28182.0| > 0.01	true
cord_test_0005	cash_paid	0.5	|31.0 - 31000.0| > 0.01	true
cord_test_0015	total_price	0.5	|63.0 - 63000.0| > 0.01	true
cord_test_0015	cash_paid	0.5	|70.0 - 70000.0| > 0.01	true
cord_test_0015	change	0.5	|7.0 - 7000.0| > 0.01	true
cord_test_0037	cash_paid	0.5	|50.0 - 50000.0| > 0.01	true
cord_test_0037	change	0.5	|30.0 - 30000.0| > 0.01	true
cord_test_0042	total_price	0.5	|16.5 - 16500.0| > 0.01	true
cord_test_0042	subtotal	0.5	|15.0 - 15000.0| > 0.01	true
cord_test_0042	tax	0.5	|1.5 - 1500.0| > 0.01	true
cord_test_0042	cash_paid	0.5	|50.0 - 50000.0| > 0.01	true
cord_test_0042	change	0.5	|33.5 - 33500.0| > 0.01	true
cord_test_0044	total_price	0.5	|30.0 - 30000.0| > 0.01	true
cord_test_0044	cash_paid	0.5	|50.0 - 50000.0| > 0.01	true
cord_test_0044	change	0.5	|20.0 - 20000.0| > 0.01	true
cord_test_0052	cash_paid	0.5	|100.0 - 100000.0| > 0.01	true
cord_test_0052	change	0.5	|60.0 - 60000.0| > 0.01	true
cord_test_0064	total_price	0.5	|16.0 - 16000.0| > 0.01	true
cord_test_0064	cash_paid	0.5	|50.0 - 50000.0| > 0.01	true
cord_test_0064	change	0.5	|34.0 - 34000.0| > 0.01	true
cord_test_0073	total_price	0.5	|22.0 - 22000.0| > 0.01	true
cord_test_0073	cash_paid	0.5	|25.0 - 25000.0| > 0.01	true
cord_test_0073	change	0.5	|3.0 - 3000.0| > 0.01	true
cord_test_0079	total_price	0.5	|22.0 - 22000.0| > 0.01	true
cord_test_0080	total_price	0.5	|25.0 - 25000.0| > 0.01	true
cord_test_0080	subtotal	0.5	|25.0 - 25000.0| > 0.01	true
cord_test_0080	cash_paid	0.5	|25.0 - 25000.0| > 0.01	true
cord_test_0083	total_price	0.5	|23.0 - 23000.0| > 0.01	true
cord_test_0083	subtotal	0.5	|20.909 - 20909.0| > 0.01	true
cord_test_0083	tax	0.5	|2.091 - 2091.0| > 0.01	true
cord_test_0083	cash_paid	0.5	|25.0 - 25000.0| > 0.01	true
cord_test_0083	change	0.5	|2.0 - 2000.0| > 0.01	true
cord_test_0092	change	0	|-10.0 - 10000.0| > 0.01	true

34 instances, 34 routed to review, 0 shipped. The confidences they carry top out at 0.5, which is why a threshold above that value contains the class completely.

Be precise about what that does and does not say. It is a property of this run’s score, and the rerun blends grounding with agreement on a different scale, so the threshold does not carry across. And it is a property of the errors the detector recorded: the rule is a ratio test on two parsed numbers, so a collapse it cannot parse, or one on a field the document never annotated, is not among the 34 it found.

The rerun’s own curve

threshold	auto-approved	wrong among approved	factor-of-1000 errors shipped
0	300/300 (100.0%)	155/300 (51.7%)	0
0.2	276/300 (92.0%)	136/276 (49.3%)	0
0.3	267/300 (89.0%)	130/267 (48.7%)	0
0.65	119/300 (39.7%)	64/119 (53.8%)	0
0.725	118/300 (39.3%)	63/118 (53.4%)	0
0.744	116/300 (38.7%)	61/116 (52.6%)	0
0.7833	115/300 (38.3%)	60/115 (52.2%)	0
0.8125	113/300 (37.7%)	58/113 (51.3%)	0
0.8183	112/300 (37.3%)	57/112 (50.9%)	0
0.825	111/300 (37.0%)	56/111 (50.4%)	0
0.83	109/300 (36.3%)	54/109 (49.5%)	0
0.8364	108/300 (36.0%)	53/108 (49.1%)	0
0.8425	107/300 (35.7%)	52/107 (48.6%)	0
0.8833	106/300 (35.3%)	51/106 (48.1%)	0
0.9	105/300 (35.0%)	50/105 (47.6%)	0
0.9767	92/300 (30.7%)	40/92 (43.5%)	0
1	91/300 (30.3%)	39/91 (42.9%)	0

Its curve records 17 thresholds and 17 distinct operating points, degeneracy false.

Compare the threshold column against the source run’s. They do not line up, and they are not meant to: the two scores are built from different components, so a number that means “grounded and agreed” here means only “grounded” there. Publishing both curves is the only honest way to show that, and quoting a single threshold across the two would be the error this section exists to prevent.

Reproducing this

The measurements file this page reads is a copy, produced by scripts/sync_measurements.mjs from:

file	sha256
projects/p3_confidence_routing/results/controls.json	7a5a2a1a93bfeaa7641ecccfcf1ede3e65b7a1e171bddaf96b29d1bddc374653
projects/p3_confidence_routing/results/source_run_verification.json	0ddd9fd907acc6cacb244b4bf038de221faccd443407d9b81c9c47e443b038fb
projects/p3_confidence_routing/results/own_run_measurement.json	060ffa299eb9876bcd0217842f4ca6645268045e6f75db4185aa3860ebc418f1
projects/p3_confidence_routing/results/severity_analysis.json	ba63571dcf777d3754e3c5b2c00454d2d182b849f83c95db02a5e7a4ae0f33c2
projects/p3_confidence_routing/results/corpus_independence.json	22ec39b157b822c0a4728e52b5479e1e1b3e42ac2939ba8b4e5f7988d5ddf261

Nothing is held out of the site copy. Nothing is omitted: this payload is its sources in full. Assembled from several results files, one per key listed in parts, because a page imports exactly one measurements file. Keys listed in merged are not from the run: they are properties of the committed corpus, computed without a model or ground truth, and are merged here so the page can state them without them being mistaken for measurements of the run.

Corpus independence

Each line is classified by leading keyword into header, date, item, subtotal, tax, total, cash or change; two documents with the same sequence are one layout with the nouns swapped. Lexical overlap was measured first and peaks at 0.23 across the controls, which would have called this family independent: it reads the nouns, and the nouns are the part that differs.

family	documents	distinct structures	largest group	redundant	documents sharing the commonest name template
control	6	3	3	3	4/6
locale_probe	5	2	3	3	4/5

The locale probe is the one to watch: it is a small family built from one template on purpose, because the probe isolates number format rather than document variety. A denominator from it is not 5 independent observations, and the page does not treat it as though it were.

Why this matters for deployment

• A router’s binary decision can be trusted before its score can. Publishing the precision/coverage curve, not just the review rate, is what shows the difference.
• 96 of 211 of the auto-approved errors were silent nulls, fields the model left empty where the document had a value. Grounding cannot flag an empty field, so review tooling has to treat “confidently absent” as its own category.
• Latency was deliberately not reported: the measurement host was shared with another workload, and the runner refuses to record timings under contention rather than publishing a number that measures the neighbour’s traffic. The controls file records that the host was contended when this run started.