OCR accuracy: the document-level check that measures nothing

If your pipeline reads PDFs, you have a quality check somewhere that decides whether a document is machine-readable. Usually it is characters per page against a threshold.

We measured what that check catches on 11 real construction PDFs, including 2 older scans carrying vendor OCR text layers. The answer is that it catches nothing, and the loss it misses is real. Figures from our construction document pipeline.

Every document passed. Including both scans.

11 of 11 PDFs passed a chars-per-page intake check.

We expected the scanned documents’ OCR layers to look visibly degraded at document level. They did not. Their document-level non-word rate sits inside the range the born-digital controls produce. A document-level quality check has nothing to separate here, and a pipeline resting on one would report a clean intake.

That is a null result and it is the most useful thing in this post: the check most teams have is measuring nothing.

The damage is per page

Across 1,234 pages:

• 22 extract effectively no text at all.
• 10 fall beyond the worst page any born-digital control produces, against a ceiling of 0.4 calibrated on 303 control pages from 8 documents, and never on the documents being judged.

Pages inside the scanned specifications extract as a single character while the document they sit in has a healthy average. Downstream, those pages simply do not exist. Nothing in the pipeline says so. A retrieval system will answer questions about that specification and quietly never see one of its pages.

Move the check from the document to the page and 3/11 documents get flagged for reprocessing.

The part we got wrong first

Having flagged those pages, the obvious next step is to re-OCR them and recover the text. So we did, at 300 dpi, on 6 of the flagged pages.

On those pages it recovered nothing. 0 of the 6 returned more characters than the embedded layer already held. The non-word rate moved by at most 0.0317.

6 pages is a probe, not a survey: they are one empty page and one degraded page from each flagged document, chosen to cover both categories rather than sampled. The claim it supports is that re-OCR is not automatically the fix, not that it never recovers anything on this corpus.

The empty pages were empty because they are blank separators, not because the OCR failed on them. Fresh OCR at higher resolution finds nothing on a blank page, which is the correct answer and not a useful one.

This is worth stating plainly because the intuitive story, “flagged pages are pages whose text can be recovered”, is the story we started with and the data does not support it. The gate finds pages worth looking at. It does not follow that reprocessing recovers anything.

What to actually do

1. Measure per page, not per document. A document average is the wrong unit: it hides exactly the pages that failed, because the pages that worked outnumber them.
2. Calibrate the threshold on documents you know are clean, never on the ones you are judging. Otherwise the threshold moves to accommodate the failure.
3. Separate “this page has no text” from “this page has bad text”. They have different causes and only one of them has a fix.
4. Check whether reprocessing helps before you budget for it. On the pages we re-ran it returned nothing, which is enough to stop treating a re-OCR stage as a given and start treating it as something to measure.

The pipeline measured at every handoff, intake through to the surviving structured record, is in the construction document case study, with the manifest and code archive attached.

If you need to know how much of your document corpus is silently not reaching your model, that is what our AI automation audit measures.