A construction document pipeline, measured at every handoff

Document-AI writeups measure one thing: the extraction step, on documents already converted to clean text, with the answer known to be in them. A construction firm’s pipeline has three other places to lose a document, and nobody publishes those.

So we measured every handoff on documents we had the right to use: intake (is the document machine-readable at all), extraction (a local model pulling the fields a bid team needs), routing (what goes to a human, decided only from signals production has) and the structured record that survives.

One thing to be clear about before any number below. The handoffs are measured on two corpora, not one, and no single document travels all four. Intake is measured on the 11 specification PDFs, because those are the only documents here that can fail it: a tender notice arrives from an API as structured JSON and is machine-readable by construction. Extraction, routing and the surviving record are measured on the 65 notices, because those are the only documents here with a field-level ground truth to be right or wrong against. A specification PDF has no buyer or deadline to extract. That split is forced by what public construction documents are, and reporting it as one continuous funnel would be inventing a path no document took.

Every figure on this page is rendered from the results file and the document manifest in the archive below, not typed in by hand. Both are hashed in the measurements file, and the split matters: the corpus counts and the regulation and classification numbers naming the documents come from the manifest, the measurements from the results.

The documents, and why sourcing was the hard part

Real construction documents are mostly things you have no right to republish: buyer-uploaded tender packs carry architects’ and consultants’ copyright, planning portal documents are the applicants’, building codes are the code bodies’. We used only sources whose licence we could verify, and recorded provenance (URL, publisher, licence, retrieval date, hash) per document in the manifest:

• 65 UK construction tender and award notices (CPV division 45) from the Contracts Finder and Find a Tender OCDS APIs. Open Government Licence v3.0, declared by the API at fetch time and recorded per document in the manifest, which is what a reader can check from the archive. The prose is written on the buyer’s side rather than generated for this test; the structured release beside it is the deterministic ground truth. Every rate below is over these 65, and they are not 65 independent observations: 3 are award notices from one buyer under one framework, sharing a covering letter and differing only in the school and the works. They overlap 0.86 against a 0.8 threshold, so the corpus carries about 63 documents’ worth of independent evidence. Nothing was deleted to tidy this up: repeated framework boilerplate is what public procurement data is like, and a corpus with the awkward rows removed would be less representative and the rates less honest. The comparison is on the extracted text the model is given, and the closest pair below the threshold is 0.75.
• 11 public-domain US construction PDFs: current UFGS specifications, OSHA 29 CFR 1926 sections, and FHWA documents including 2 older scans carrying vendor OCR layers. These carry the intake measurement.

Rejected, with reasons logged: SAM.gov (machine access is login-gated), the notice HTML pages and attachments (they refuse non-browser clients; the API is the offered channel), loc.gov scans (anti-bot wall), ICC codes (copyright), council planning documents (applicant copyright). If we could not obtain a document class honestly, the page says so rather than quietly substituting.

First handoff: intake, where the quiet loss lives

All 11 PDFs pass the simplest intake check there is, a chars-per-page threshold: 11 of 11 accepted, including both scans. We expected the scans’ OCR text layers to be visibly degraded at document level. They were not, and that null result is pinned by a test: the scans’ document-level non-word rate sits inside the born-digital control range. A document-level quality check measures nothing here.

The real damage is per page. Across 1234 pages, 22 extract effectively no text and 10 fall beyond the worst page any born-digital control produces (ceiling 0.4, calibrated on 303 control pages and never on the probe documents). Pages in the scanned specifications extract as a single character inside documents whose averages look healthy. Downstream, those pages simply do not exist, and nothing says so. The demos below show fresh OCR recovers nothing from them, which is what a blank separator looks like rather than what a figure page looks like.

The corrected gate is page-level and flags 3/11 documents for reprocessing. Re-OCR at 300 dpi says what the flagged pages actually are. The 6 pages re-run are one empty page and one degraded page from each flagged document. The 3 empty ones held no more than 36 characters each, and fresh OCR returned no more characters than the embedded layer already held, 2 of them none at all. The demos record character counts and non-word rates, not a text comparison, so that is the whole of what they show. The 3 degraded ones already held from 1,246 characters upward, and re-OCR returned fewer on every one: 0 of the 6 pages recovered more than the embedded layer had, and the non-word rate moved by at most 0.0317 across them. The gate finds pages worth looking at; it does not follow that re-processing recovers anything.

Second handoff: extraction, where the text does not contain the answers

The bid-triage fields are buyer, value, currency, deadline, contract start and end. The structured release knows them. The notice prose, which is what a text pipeline gets, mostly does not:

field	in the structured record	stated in the notice text
buyer name	65/65	19/65
contract value	57/65	2/65
tender deadline	58/65	3/65
contract start	44/65	0/65
contract currency	57/65	5/65
contract end	44/65	0/65

All 6 rows are shown, because the totals below are built from them: the 29 field-slots the notice text actually states are this column’s sum, and a table missing a row is a table a reader cannot check.

This is the single largest loss in the pipeline and it happens before any model runs. The values live in form fields the buyer typed into the portal, not in the prose. An extraction pipeline over notice text is mostly being asked to extract things that are not there, so the load-bearing skill is abstention.

We measured that directly, two prompt conditions over the same 65 documents with a local model (qwen2.5:7b, temperature 0, seed 7): a naive prompt that just asks for the fields, and a guarded one told to return null for anything not stated.

Controls first: of the 29 document-field pairs whose value sits verbatim in a short text (the extractions that should be trivial), the naive prompt got 26 right and the guarded prompt only 10. Then the other side: on field slots genuinely absent from the text, the naive prompt asserted a value anyway 42 times out of 296; the guarded prompt 6 out of 296.

Those denominators exclude a third group, and it is worth saying so because excluding it flatters both numbers. 65 field slots have no ground truth in the structured release at all, so the corpus cannot say whether an assertion on them is right or invented, and they sit in no rate on this page. The model asserted values on 8 of them under the naive prompt and 7 under the guarded one, across contract_end_date, contract_start_date, deadline_date, value_amount and value_currency. Excluding unadjudicable slots is the defensible choice; not counting them anywhere was an undisclosed one.

That pair of numbers is the honest shape of prompt guarding: the guard buys its low fabrication rate by abstaining, including on values that are right there in the text. A pipeline evaluated only on its fabrication rate, or only on its extraction rate, would look finished; measured on both, neither configuration is fit to ship as-is. No notice states every field in prose, so document-level controls do not exist in this corpus; the control claim is scoped to what is actually trivial.

Third handoff: routing, from signals production actually has

The router sees the extraction output and the document text. It never sees the ground truth; a test asserts the module cannot even reach it. Signals: schema validity, verbatim grounding of every asserted value in the text, required-field completeness, date sanity.

Under the guarded condition it auto-approved 0/65 and routed 65/65 to a human. That queue is not model failure: it is the coverage gap above flowing downstream. A record without a value and a deadline cannot be auto-approved, and the text never contained them. The pipeline is honest about what it cannot do, at the price of a review queue.

The grounding check is the same deployable guardrail as in our RAG audit: the asserted value must appear in the source text. What it cannot catch, by construction, is an asserted value that is in the text but is the wrong figure for the field. Under the naive condition,

28 of 36

failing documents asserted at least one value the grounding check could not fault: a real string from the document, fully grounded. Grounding cannot flag those, which is the scoped claim and the true one.

What it is not is invisible to everything, which is a wider claim than the run supports: under the naive condition 0 of 65 documents were auto-approved and 65 of 65 went to a human, mostly for being incomplete rather than for anything to do with grounding. None of these failures reached production in this run. A different corpus, where more notices carried complete records, would not have that accident to rely on.

The code that assigns these classes is not in this project’s archive: it is the shared analysis module from the RAG audit, and this project imports it from that tree rather than vendoring a copy that could drift. It ships in that project’s archive instead, so the criteria are readable, just not from this download. The classes and the reason codes each condition was given are in the results file here.

The shared severity assignment calls the naive condition critical. It calls the guarded condition critical as well, for the same recorded reason (wrong_quiet_uncaught), on 2 documents rather than 28 of them. So the guards cut the count sharply and leave the class where it was. Both conditions are named here for that reason: giving the class for the naive condition alone would read as though the guards had changed it.

What survives end to end

Guarded condition, and the funnel starts at extraction rather than at intake, for the reason given at the top: these are the notices, and the PDFs measured earlier never enter it. 65 documents in: 65 produced a valid structured record, 0 were auto-approved, and 0 of those were fully correct against the ground truth the router never saw.

Latency is not reported: the host ran other workloads throughout, the runner detects contention and refuses to record wall-clock numbers under it.

The model, and the settings it ran under

Extraction is a local model with everything pinned, so the run can be repeated:

setting	value
model	qwen2.5:7b
temperature	0
seed	7
context window	8,192
max new tokens	400
GPU layers	0
threads	16

num_gpu at 0 is not a typo. This run was CPU-only, because the machine’s GPU was carrying another workload, and that is also why no timing figure appears anywhere on this page.

Every document, with its source:

source	documents
contracts_finder	45
find_a_tender	20
wbdg	5
govinfo	3
rosap	3

The specification PDFs, named, because a reader checking the intake result needs to know what kind of document produced it:

document
UFGS 26 20 00 Interior Distribution System
UFGS 04 20 00 Masonry
29 CFR 1926.451 Scaffolds: general requirements
29 CFR 1926.501 Fall protection: duty to have
29 CFR 1926.651 Excavations: specific requirements
FP-96 Standard Specifications for Construction of Roads and Bridges
FHWA-RD-93-032 Guide Design Specification for Bridge Temporary Works (scan)
FHWA-RD-93-031 Guide Standard Specification for Bridge Temporary Works (scan)
UFGS 03 30 00 Cast-in-Place Concrete
UFGS 05 12 00 Structural Steel
UFGS 07 92 00 Joint Sealants

Licences in use across the corpus: Open Government Licence v3.0 and US government work, public domain (17 USC 105). The notices are CPV division 45, construction work, across 37 distinct CPV codes. The regulations are 29 CFR part 1,926.

Intake, document by document

The counts above are a summary. This is every PDF, with what the intake check saw:

document	pages	chars/page	nonword	worst page	classification	producer says scanner	flagged
cfr_1926_451	11	4,265	0.2517	0.3271	born_digital	false	false
cfr_1926_501	3	4,390	0.2295	0.2500	born_digital	false	false
cfr_1926_651	4	4,270	0.2181	0.2482	born_digital	false	false
ufgs_03_30_00	93	2,266	0.2421	0.3958	born_digital	false	false
ufgs_04_20_00	67	2,393	0.2387	0.3826	born_digital	false	false
ufgs_05_12_00	25	2,183	0.2759	0.4000	born_digital	false	false
ufgs_07_92_00	18	1,869	0.2515	0.3690	born_digital	false	false
ufgs_26_20_00	82	2,330	0.2477	0.3672	born_digital	false	false
rosap_fp96	827	1,570	0.2148	0.5833	born_digital	false	true
rosap_temp_works_design	90	2,232	0.2343	0.4730	scanned_ocr_layer	true	true
rosap_temp_works_spec	14	2,022	0.2269	0.4432	scanned_ocr_layer	true	true

Read the nonword column against the page ceiling of 0.4. The document-level rates cluster: the scans do not stand out from the born-digital controls at this level, which is the null result the summary above states. The worst page column is where they separate, and it is the column a document-level check never computes.

The producer string is worth its own column. It is metadata the file carries about the tool that wrote it, and it is the cheapest signal available: a scanner or OCR product in that field is a strong hint before any text is extracted. It is a hint rather than a test, which is why the run records it beside the measured rates instead of routing on it.

Re-OCR, page by page:

document	page	dpi	embedded chars	fresh OCR chars	embedded nonword	fresh nonword
rosap_fp96	16	300	36	35	0.3333	0.3333
rosap_fp96	3	300	1,246	1,185	0.4600	0.4653
rosap_temp_works_design	12	300	1	0	1.0000	1.0000
rosap_temp_works_design	6	300	2,230	1,907	0.4706	0.4389
rosap_temp_works_spec	14	300	1	0	1.0000	1.0000
rosap_temp_works_spec	5	300	2,253	1,930	0.4432	0.4396

Extraction, field by field, outcome by outcome

The table earlier shows how often each field is in the structured record and how often the prose states it. This is what the model then did about that, under both prompts.

Guarded:

field	asserted unsupported	correct abstain	correct extract	miss abstain	ungraded no gt	wrong value
buyer_name	0	46	5	14	0	0
value_amount	0	55	1	1	8	0
value_currency	0	52	1	4	8	0
deadline_date	1	54	3	0	7	0
contract_start_date	3	41	0	0	21	0
contract_end_date	2	42	0	0	21	0

Naive:

field	asserted unsupported	correct abstain	correct extract	miss abstain	ungraded no gt	wrong value
buyer_name	32	14	17	1	0	1
value_amount	1	54	2	0	8	0
value_currency	0	52	4	1	8	0
deadline_date	2	53	3	0	7	0
contract_start_date	4	40	0	0	21	0
contract_end_date	3	41	0	0	21	0

The category names are the ones the grader assigns, and they are worth reading carefully because two of them are successes and two are failures:

• correct abstain: the field is not in the text and the model said nothing. Right answer.
• correct extract: the field is in the text and the model pulled it. Right answer.
• miss abstain: the field is in the text and the model said nothing. Over-abstention.
• asserted unsupported: the field is not in the text and the model produced a value anyway. Fabrication.

Between the two prompts, the last category moves from 42/296 (14.2%) of absent field slots to 6/296 (2.0%). That is the single number the prompt change buys, and the table beside it shows what it costs.

Over-abstention, which is what the guard costs

The guarded prompt does not simply fabricate less. It says less.

condition	control pairs passed
naive prompt	26/29 (89.7%)
guarded prompt	10/29 (34.5%)

29 control pairs, where the value is in the text and the right move is to pull it. The naive prompt passes 26/29 (89.7%). The guarded prompt passes 10/29 (34.5%).

So the tradeoff is not fabrication against nothing. It is fabrication against over-abstention, and a prompt that moves one moves the other. A pipeline choosing the guarded prompt is choosing to send more documents to a human rather than fewer wrong values to a database, which is usually right and is never free.

Fields with no ground truth

field slots with no ground truth in the structured release, excluded from the fabrication denominator because the corpus cannot say whether an assertion is right

condition	slots with no ground truth	model asserted anyway	fields
naive	65	8/65 (12.3%)	contract_end_date, contract_start_date, deadline_date, value_amount, value_currency
guarded	65	7/65 (10.8%)	contract_end_date, contract_start_date, value_amount, value_currency

Those slots are excluded from the fabrication denominator, which is the conservative choice: including them would let an assertion the corpus cannot judge count as a fabrication, and the rate would look worse than the evidence supports.

Routing, and why nothing was auto-approved

condition	documents	auto-approved	routed to a human	incomplete	ungrounded
naive	65	0/65 (0.0%)	65/65 (100.0%)	64	9
guarded	65	0/65 (0.0%)	65/65 (100.0%)	65	7

Under both conditions every document went to a human, and the reason is almost always the same one: the record is incomplete. Grounding barely features. That ordering is the finding: a routing rule built only on grounding would have had almost nothing to do here, because completeness fails first and fails on nearly everything.

The funnel, and why it starts where it does

65 notices enter the guarded funnel. 65/65 (100.0%) produced a structured record that validates, 0/65 (0.0%) were auto-approved, and 0, out of none auto-approved of those were fully correct against the ground truth the router never saw.

The funnel starts at extraction rather than at intake, and that is a deliberate restriction rather than an oversight. The intake measurement is on the specification PDFs, which have no buyer or deadline to extract; the extraction and routing measurements are on the notices, which arrive as structured JSON and cannot fail an intake check. Joining them into one funnel would draw a path no document travelled.

Where the failures go, under each prompt

outcome	naive	guarded
failing documents	36	21
loud empty or invalid	0/36 (0.0%)	13/21 (61.9%)
caught by grounding	8/36 (22.2%)	6/21 (28.6%)
uncaught grounded wrong	28/36 (77.8%)	2/21 (9.5%)

The rows are ordered by how much they matter to a deployment. A loud failure, an empty or invalid record, costs a retry. A failure the grounding check catches costs a review. A failure that is grounded and still wrong is the one that ships.

Under the guarded prompt that last row is 2 documents, against 28 under the naive one. The guards cut the count and leave the class exactly where it was: a value that is present in the document and wrong is not something a grounding check can see, whatever the prompt says.

Detectability and recovery, both conditions

	naive	guarded
failing documents	36	21
passing documents	29	44
separable by runtime score	false	false
underpowered	false	false
caught by the grounding check	8/36 (22.2%)	19/21 (90.5%)
false positives	13	35
severity	critical	critical

Neither condition is separable by the runtime score, and neither is underpowered, so these are real negative results rather than missing ones. The recovery check is deployable, and under the naive condition it catches 8/36 (22.2%) of the failures with 13 false positives on 29 passing documents.

Neither condition produced a threshold to report: both record their best threshold as null. What they do record is that any fit would have been made in sample, with a holdout of exactly 0 documents. A threshold chosen on the documents it is scored against would not be deployable, and the run carries that as a field rather than leaving a reader to notice.

Reproducing this

Produced by projects/p4_construction_vertical/cdocs/measurements_export.py, mode derived, from:

file	sha256
projects/p4_construction_vertical/results/results.json	f54affc30a597fca1c39003dd7de1771495ae760f04c9827c477647c75911f8a
projects/p4_construction_vertical/data/manifest.json	4f34e242c287032ac93b1b098b8c5e85b143b29495895ba8d0c41717b69aea2d
projects/p4_construction_vertical/results/corpus_independence.json	c7b7fc9554ba22d91cc13fd893e2c8efe4e45ad4c0eeea4a3e7f2f67e3831487

Derived, not copied: this file is a summary computed from the inputs above. Re-derive and compare with `uv run python -m cdocs.measurements_export --check`.

What we would tell a construction firm

1. The expensive loss is not model accuracy. It is documents and pages that never make it into text (intake) and fields that were never in the text at all (coverage). Measure those before buying an extraction model.
2. For UK procurement, take the structured OCDS data; it is published beside the prose, licensed for reuse, and already contains the fields. A text pipeline on the notices alone re-derives a worse copy of it.
3. If a pipeline must read prose, the tradeoff to manage is fabrication against over-abstention, and a prompt guard moves both at once. Grounding checks catch fabrication, not wrong-but-present values.

---

Contains public sector information licensed under the Open Government Licence v3.0: tender and award notice data from Contracts Finder and Find a Tender (Cabinet Office). US government documents (UFGS, CFR, FHWA) are public domain. Provenance for every document, including URL, publisher, licence and retrieval date, is recorded in the manifest inside the code archive.