We measured what a stock PII redactor leaves behind

Redaction tools get scored the way NER models get scored: token overlap, partial credit, one blended F1. That grading has a trap in it. If the redactor turns “John Smith” into “John [REDACTED]”, token-level scoring books it as mostly correct. The person is still identified. For a privacy tool, an entity is either fully gone or it is leaked, so we measured stock Microsoft Presidio with all-or-nothing entity recall, kept partial detection as its own error class, and recorded exactly how much credit partial matching was quietly adding.

Two tiers, never pooled: the Text Anonymization Benchmark test split (127 real ECHR court judgments, human-annotated at character level, MIT licence) and 2,000 English rows sampled from ai4privacy’s open-pii-masking-500k, which is synthetic and is labeled as such everywhere it appears here. Those denominators are not quite as independent as they look, and we measured that rather than assuming it. 2 of the 127 judgments are near-duplicates of each other, overlapping 0.93 against a 0.8 threshold once the boilerplate every judgment shares is removed, so the tier carries 126 judgments’ worth of independent evidence. The synthetic tier, where templated repetition was the thing to look for, has none: 305,819 candidate pairs examined and nothing above the threshold, the closest being 0.67. Neither corpus was edited to improve those numbers. TAB is a published split, and figures computed on a trimmed copy of it would no longer be comparable to anyone else’s. Grading is deterministic character arithmetic against annotated spans. No LLM judge anywhere. Every figure on this page is rendered from the results file in the archive below, not typed in by hand.

What “stock” means here, exactly

“Stock Presidio” is not one thing across versions, so the run records which one:

component	version
presidio-analyzer	2.2.364
spaCy	3.8.16
spaCy model	en_core_web_lg
Python	3.12.11
platform	macOS-26.3-arm64-arm-64bit

No custom recognizers, no context words, no deny lists, no allow lists, and the analyser’s score threshold left at 0, which is the setting that returns everything the recognizers produce. Every figure below is what that configuration does, and the threshold section later shows what moving that one number costs.

One smoke test before the corpora, because it says more about the defaults than a rate can. The string My name is John Smith, SSN 078-05-1120, email j@x.com is about as easy as PII gets. Stock Presidio returned 4 detections on it, and US_SSN was not among them: the run records has_us_ssn as false. The social security number was typed as PHONE_NUMBER at 0.4.

That is not a failure to detect. It is a failure to name, and it matters because a redaction policy written per entity type (“always redact US_SSN, review PHONE_NUMBER”) would not fire on it.

The two corpora, and what each is worth

	TAB	ai4privacy
kind	real ECHR court judgments, human-annotated	SYNTHETIC
source	github.com/NorskRegnesentral/text-anonymization-benchmark	huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy
split	test	validation (English rows only)
licence	MIT	CC-BY-4.0
licence verified locally	true	false
documents	127	2,000
annotated mentions	30,541	5,161
mentions dropped for misalignment	0	0

Two things in that table are load-bearing.

The licence evidence differs. TAB’s licence is a file in the archive, data/tab/LICENSE.txt, checked locally. ai4privacy’s is declared on the dataset card and recorded at fetch time, with no file to check, so the run says false rather than claiming a verification it did not perform.

Neither tier dropped a mention for misalignment. Character-offset annotations and a tokeniser disagree constantly, and the usual response is to drop the spans that will not line up. Dropping mentions silently improves recall, because the ones that will not align are disproportionately the awkward ones. The count is 0 on both tiers and it is published for that reason.

The synthetic tier is drawn evenly across 4 regions, 500 CA, 500 GB, 500 IN and 500 US, under seed 7. That matters later: several of its labels mean different things per region, and a US-shaped recognizer meets non-US formats here.

Controls before findings

15 documents authored to contain no identifying information went through first. A redactor that flags clean text is manufacturing review work. Stock Presidio flagged 2 of 15, with 3 detections, all DATE_TIME, on the words "annual", "each season" and "annually". Those words genuinely are temporal expressions, so this is less a detector bug than a scope mismatch: the DATE_TIME recognizer is wider than PII, and every hit still lands on a reviewer’s desk. 5 positive controls with one obvious entity each (email address, person, phone number, credit card and ip address) all detected correctly; the run aborts if they do not.

The synthetic tier carries a second group that nobody authored: 60 of its rows carry no annotated PII. Presidio flagged 3 of them, with 4 detections between them. This is a weaker instrument than the 15 and worth separating from them. The 15 were written to be clean and are checked to contain no identifier shape at all; the 60 are rows the generator left unannotated, and a missing annotation would make a detection correct rather than spurious.

Looking at what was actually flagged settles it, and the answer is mixed. The 3 plain false positives are URL on "1parerregistrationform.do", URL on "..ma" and LOCATION on "Volunteering". The remaining 1 is US_DRIVER_LICENSE at 0.3 on "64839276HS", which reads like an identifier the generator simply did not annotate. So on this evidence the second group is mostly detector noise with a corpus gap in it, 3 of the 4 detections against 1, which is why we do not pool it with the authored controls or claim the two agree.

The headline: entities, not mentions

An entity is a person, a case number, an organisation. In a court judgment the applicant is named a dozen times, and redacting eleven of those still identifies them, so the number that matters is how many entities were redacted completely, every mention, every character. On the court texts that is 11,076 of 17,448: 63.5%. Another 2,506 were partly redacted and 3,866 had at least one mention that drew no detection at all. On the synthetic rows it is 3,195 of 5,161, 61.9%, and there the entity rate equals the mention rate by construction, since that dataset labels spans and nothing joins two mentions of one person.

Every other recall figure below is per mention, which is the looser question. Both are reported because the gap between them is a real property of a corpus, not a rounding choice.

The partial-redaction trap, quantified on real text

One thing about the denominators before the numbers. TAB is annotated by up to 10 people per judgment, and its protocol counts each annotator’s mentions separately, so every per-mention figure below is annotator-weighted. So are the entity counts above it: 17,448 is entities per annotator, not distinct entities. The 1,612 DIRECT mentions are 407 distinct spans, drawn from judgments each seen by a median of 3 annotators. That is TAB’s own unit and the one its published figures use. Counting distinct spans in the corpus instead gives 6,931 mentions rather than 20,809, so a reader recomputing from the files should expect the smaller number.

On the court judgments, direct identifiers (the spans that name a person outright) were fully redacted in 274 of 1,612 annotated mentions by our all-or-nothing rule: 17.0%. Grant credit for any overlap, the way token-level scoring does, and the same run reads as 62.5%: 274 fully redacted plus 733 partially covered, against the same 1,612.

That 45.5 points gap is tempting to call safety that does not exist. Most of it is not. Scoring the same detections with the TAB authors’ own evaluation code, which forgives a surviving punctuation mark or listed function word, direct-identifier recall is 60.3% per mention, 971 of 1,610. The denominator moved by 2, and it is worth saying why rather than leaving a reader to suspect a typo: the two rules count different mention sets before any masking question arises. The authors take every mention of an entity that needs masking, including 332 the annotators individually marked NO_MASK, so their overall total is 21,141 against our 20,809. For DIRECT they bucket by the entity’s identifier type where we bucket by the mention’s, which moves 18 mentions one way and 16 the other. These judgments write the title without a full stop, “Mr” then the name, and the recorded divergence says what survives: of the 2,333 mentions the two rules disagree about, the uncovered remainder is exactly “Mr” in 2,009 of them, and 2,040 of the divergent mentions are PERSON. Almost all of those are mentions we covered in part: 2,325 of the 2,333, against 8 we did not cover at all, which TAB masks because the whole mention is text its ignore lists forgive and nothing is left to leak once it is. A surviving “Mr” identifies nobody. We name no applicant anywhere: the corpus is real court judgments, so every value quoted in this entry comes from the synthetic tier, from the controls we wrote, from an invented example, or, as with the surviving “Mr” above, is a title or function word that the results file records because TAB’s own rules forgive it. Nothing quoted here is a name or an identifying value taken from a judgment. We keep the stricter rule, because for a privacy tool “every character or it leaked” is the right default, but the citable figure is theirs and it is the one to compare against published work.

What survives the correction is the part that was never about titles. Case and application numbers (CODE) were almost never caught: 90 of 1,612, or 5.6%, and both rules agree on it exactly, not merely to the rounding: 90 of 1,612 under theirs as well, because a case number has no title attached to forgive. TAB’s own headline, entity-level recall, where an entity counts as redacted only if every mention of it is, comes to 74.5% overall (13,006 of 17,448) and 48.6% on direct identifiers (562 of 1,157). And every one of the 127 documents still leaks at least one annotated identifier under the authors’ rule as well as ours, so the finding that nothing here is safe to ship does not rest on our stricter reading.

Scoring the same detections as span sets, with nervaluate’s SemEval-2013 schemes and every label collapsed to one tag, puts a figure on the choice of rule. All three schemes score the same 5,121 redaction intervals against the same 6,201 annotated spans, so only the numerator moves. Strict boundary matching counts 3,154 exact matches: 61.6% precision and 50.9% recall. SemEval’s partial scheme adds the 1,104 boundary-mismatched intervals at half credit and reads 72.4% and 59.8%. Crediting any overlap at all, which is what a token-level score effectively does, counts all 4,258 and reads 83.2% and 68.7%. Those 1,104 spans between the first reading and the last are predictions that touch an annotated span without covering it: the redaction that leaves something behind.

The synthetic tier says the same thing in a different accent

On ai4privacy rows, 1,247 of 2,000 documents (62.4%) leaked at least one gold entity. Emails did well (246/250 fully redacted). Phone numbers did not: 116/326 (35.6%) fully redacted, with another 99 partially covered, worth another 30.4 points of illusory recall under lenient scoring. At the bottom sit the 3 weakest of the dataset’s own labels: SEX (1/63), GENDER (1/56) and TITLE (52/222). The list is taken by rank from the results file rather than assembled by hand, so a rerun cannot leave it naming the wrong three. They are named here as the dataset names them: TITLE holds 26 distinct values, and while the 8 commonest are honorifics and cover 203 of its 222 spans, the other 18 values account for the remaining 19 spans and are mostly offices and job titles, Attorney, Captain, Director, Senator and the like. Calling the label honorifics would quietly narrow what was measured. All of which is why everything here is reported per entity and there is no single headline F1: a missed SSN and an over-redacted date are not the same kind of wrong, and averaging them hides the one that matters.

The known Presidio quirk reproduced exactly: given the sentence “My name is John Smith, SSN 078-05-1120, email j@x.com”, the SSN is typed PHONE_NUMBER at confidence 0.4, with a spurious URL detection inside the email address beside it. Both mechanisms are pinned by live tests before they are reported.

Threshold recall is standing on noise

Stock Presidio ships weak-pattern recognizers that fire on bare digit runs at confidence 0.01 to 0.3; a nine-digit number simultaneously draws US_DRIVER_LICENSE, US_BANK_NUMBER, US_PASSPORT, US_SSN and US_ITIN detections. The stock anonymizer redacts everything regardless of score, so those detections really do redact ID numbers at default settings. But the moment a deployment adds a confidence threshold, that recall collapses: passport numbers go from 73/73 fully redacted at threshold 0 to 33/73 at 0.4. The SOCIALNUM label, which in this corpus means social insurance and national identity numbers rather than only US social security numbers, 8 of the 47 spans being US against 17 Canadian, 12 Indian and 10 British, goes from 47/47 to 20/47 at 0.6. Name recall behaves differently, but not better: spaCy NER emits a fixed 0.85, so GIVENNAME holds at 1,174/1,646 from threshold 0 through 0.85 and then goes to 0/1,646 at 0.9. The sweep runs to 0.95, so the fixed score is not above every threshold on the dial. It is the same cliff, further along the dial than a deployment would usually set it.

Across the synthetic tier, 19 of the 20 labels with any recall at threshold 0 have lost more than half of it by 0.95. EMAIL is the exception, and not a small one: 246/250 at threshold 0 and 245/250 at 0.95, one mention across the whole dial. So a confidence threshold on this stack behaves less like a precision knob than like a recall cliff for most of what it covers, with the edges in different places and several labels stepping down more than once rather than falling at a single point. It does not hold for every label, which is why the exception is named above.

Can anything catch the leaks at runtime?

One deployable check, and one measurement that is not one.

The measurement first, because it is the one that is easy to overstate. We asked whether a document-level confidence signal, the mean detection score, separates leaking documents from the rest. On the synthetic tier it does not: leaking rows averaged 0.595 against 0.668. On the court texts the question is unanswerable, because every document leaked and there is nothing to compare against. Read the method before the verdict: the threshold is swept over the same documents it is then scored against, and the sweep is told which of them leaked. There is no holdout. So this says whether the two groups can be told apart after the fact, which is a diagnostic. It is not a check the pipeline could run, because in production nobody hands you the answers to fit the threshold on.

The check that is deployable reads only the redacted output and needs no ground truth at all: a second-pass regex for surviving SSN, email, phone, card and IP shapes. It caught 125 of 1,247 leaking synthetic documents with 0 false positives on 753 non-leaking ones, and 0 of 127 on the court texts. Of the 8,121 leaked mentions there, the largest classes are PERSON at 2,682, ORG at 1,631 and CODE at 1,522, with quantities, demographics, locations, dates and miscellany making up the rest. The structured shapes can represent none of the 8 classes: a case number, a reference code, an organisation name, an award in euros and a date all pass the second pass untouched, which is asserted rather than assumed. The severity classes below were recomputed from the stored per-document records after the shared criteria changed, without re-running the detector; the run file records that regrade and the run it came from. Under the shared severity criteria this site uses everywhere the two tiers grade differently, and the difference is the point. The synthetic tier is critical: wrong, silent, and not caught by the cheap check. The court texts are undetermined, because with no non-leaking document in the tier there is nothing to compare the confidence signal against, and critical is a claim that a signal was looked for and not found. Calling both tiers critical would assert a comparison that cannot be made on 127 of the documents.

Scored a second time, with the TAB authors’ own code

A recall figure is only as good as the rule behind it, so the same predictions were scored again with the benchmark authors’ evaluation code, unmodified.

Pilan, Lison, Ovrelid, Papadopoulou, Sanchez, Batet (2022), The Text Anonymization Benchmark (TAB), Computational Linguistics 48(4), arXiv:2202.00443. Scored with the authors' evaluation.py, unmodified.

measure	direct identifiers	quasi identifiers	all
entity recall	562/1,157 (48.6%)	12,444/16,291 (76.4%)	13,006/17,448 (74.5%)
mention recall	971/1,610 (60.3%)	14,161/19,531 (72.5%)	15,132/21,141 (71.6%)

Read the direct-identifier column against our own 274/1,612. The authors’ rule is more forgiving, because it excuses a surviving punctuation mark or a listed function word, and it counts a different mention set. Both are defensible. Neither is the other, and a page quoting one without naming the rule is quoting a number with no unit.

The authors’ own entity recall across the whole tier is 74.5%, and 127/127 documents leak at least one entity under it. Under the stricter all-or-nothing rule this entry uses, every document leaks. That gap is the argument of this page in one comparison.

Their token-level precision, under the uniform weighting the authors offer as a fallback, is 37,605/50,044 (75.1%). It is not comparable to the figure in the paper: the published one weights tokens by a language model’s surprise, which needs a dependency and a model download this project does not carry, and the run says so rather than presenting the fallback as the paper’s number.

Per gold class, under the authors’ rule:

class	masked
DATETIME	9,449/9,543 (99.0%)
PERSON	3,522/4,168 (84.5%)
ORG	401/2,190 (18.3%)
CODE	90/1,612 (5.6%)
LOC	1,177/1,477 (79.7%)
DEM	441/933 (47.3%)
QUANTITY	2/664 (0.3%)
MISC	50/554 (9.0%)

QUANTITY is the class this stack barely touches, 2/664 (0.3%). DATETIME is the one it handles, 9,449/9,543 (99.0%). The spread between those two is why a single headline recall for a redactor is close to meaningless: which classes your documents are dense in decides your number.

What the analyser called each thing

Recall says whether a span was covered. It does not say what the analyser thought it was, and a policy written per entity type depends entirely on the second question.

gold class	mentions	most common predicted type	no detection at all
DATETIME	9,536	DATE_TIME (9,494)	42/9,536
PERSON	4,138	PERSON (3,746)	368/4,138
ORG	1,946	LOCATION (298)	1,446/1,946
CODE	1,612	DATE_TIME (96)	1,501/1,612
LOC	1,459	LOCATION (1,162)	244/1,459
DEM	917	NRP (300)	403/917
QUANTITY	664	DATE_TIME (16)	640/664
MISC	537	PERSON (49)	440/537

The last column is the one to read. It is not a wrong label, it is the absence of one: nothing fired over that span at any score.

CODE, the class holding case numbers and file references, is the clearest case: 1,501/1,612 mentions drew no detection at all. Court judgments are full of identifiers shaped like nothing in a US-centric recognizer set, and the stock configuration has no pattern for them. That is not a tuning problem, it is a missing recognizer, and no threshold change reaches it.

Precision, and why ours is only a proxy

predicted type	detections	overlapping a gold span
DATE_TIME	2,944	2,799/2,944 (95.1%)
PERSON	1,104	1,091/1,104 (98.8%)
LOCATION	863	813/863 (94.2%)
NRP	198	193/198 (97.5%)
PHONE_NUMBER	6	4/6 (66.7%)
US_DRIVER_LICENSE	6	3/6 (50.0%)
URL	2	0/2 (0.0%)

Those numbers look good and they are the weakest thing on this page. TAB annotates the spans that need masking; it does not exhaustively annotate everything that is not PII. So a detection landing outside an annotated span is not necessarily wrong, and this column cannot tell a false positive from an unannotated true one. It is published as a proxy, labelled as one, and it is the reason the negative controls exist: on documents written to contain nothing, a detection is unambiguously spurious.

On the synthetic tier, where annotation is exhaustive by construction, precision means what it says:

predicted type	detections	overlapping a gold span
PERSON	1,384	1,358/1,384 (98.1%)
DATE_TIME	691	596/691 (86.3%)
LOCATION	426	399/426 (93.7%)
US_DRIVER_LICENSE	317	316/317 (99.7%)
URL	253	250/253 (98.8%)
EMAIL_ADDRESS	248	248/248 (100.0%)
PHONE_NUMBER	223	223/223 (100.0%)
US_BANK_NUMBER	204	204/204 (100.0%)
US_SSN	84	84/84 (100.0%)
US_PASSPORT	68	68/68 (100.0%)
NRP	37	13/37 (35.1%)
UK_NHS	13	13/13 (100.0%)
MEDICAL_LICENSE	5	5/5 (100.0%)
US_ITIN	5	5/5 (100.0%)
CREDIT_CARD	3	3/3 (100.0%)

Span scoring, with the labels collapsed

One more scoring rule, because the two above both bake in a choice about labels. Collapsing every type to one tag and comparing merged redaction intervals against the character union of the annotators gives:

rule	correct	incorrect	partial	missed	spurious	precision	recall
strict	3,154	1,104	0	1,943	863	3,154/5,121 (61.6%)	3,154/6,201 (50.9%)
partial credit	3,154	0	1,104	1,943	863	3,706/5,121 (72.4%)	3,706/6,201 (59.8%)

Same predictions, same gold, 6,201 possible and 5,121 actual spans in both rows. The precision and recall cells carry the numerator each rate divides. Under partial credit that numerator is not the exactly-correct count: the rule counts a partially matched span at half weight, so it is 3,154 plus half of 1,104.

Recall moves purely because those 1,104 spans move out of the incorrect column and into a partial one.

That is the partial-redaction trap a third time, on a third scoring rule. It keeps appearing because it is structural: any rule that rewards overlap will reward a redaction that left the surname showing.

The whole dial, not the summary of it

The threshold section above quotes three points. Here is the grid, for every synthetic label with recall at the bottom of the dial.

label	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	0.95
AGE	35/105	35/105	35/105	35/105	35/105	35/105	35/105	35/105	35/105	0/105	0/105
BUILDINGNUM	59/193	59/193	59/193	59/193	59/193	59/193	59/193	59/193	59/193	0/193	0/193
CITY	224/323	224/323	224/323	224/323	224/323	224/323	224/323	224/323	224/323	0/323	0/323
CREDITCARDNUMBER	44/47	17/47	17/47	17/47	17/47	15/47	15/47	15/47	15/47	3/47	3/47
DATE	220/283	220/283	220/283	220/283	220/283	220/283	220/283	181/283	181/283	16/283	16/283
DRIVERLICENSENUM	12/43	10/43	10/43	10/43	7/43	6/43	6/43	6/43	6/43	0/43	0/43
EMAIL	246/250	246/250	246/250	246/250	246/250	246/250	246/250	246/250	246/250	245/250	245/250
GENDER	1/56	1/56	1/56	1/56	1/56	1/56	1/56	1/56	1/56	0/56	0/56
GIVENNAME	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	1,174/1,646	0/1,646	0/1,646
IDCARDNUM	52/183	32/183	32/183	32/183	32/183	25/183	25/183	25/183	23/183	0/183	0/183
PASSPORTNUM	73/73	59/73	59/73	59/73	33/73	22/73	22/73	21/73	21/73	4/73	4/73
SEX	1/63	1/63	1/63	1/63	1/63	1/63	1/63	1/63	1/63	0/63	0/63
SOCIALNUM	47/47	42/47	42/47	42/47	42/47	26/47	20/47	20/47	19/47	4/47	4/47
STREET	138/218	138/218	138/218	138/218	138/218	138/218	138/218	138/218	138/218	0/218	0/218
SURNAME	470/658	470/658	470/658	470/658	470/658	470/658	470/658	470/658	470/658	0/658	0/658
TAXNUM	38/40	37/40	37/40	37/40	37/40	18/40	18/40	18/40	17/40	1/40	1/40
TELEPHONENUM	116/326	116/326	116/326	116/326	116/326	45/326	45/326	45/326	40/326	1/326	1/326
TIME	160/316	160/316	160/316	160/316	160/316	160/316	160/316	160/316	160/316	0/316	0/316
TITLE	52/222	52/222	52/222	52/222	52/222	52/222	52/222	52/222	52/222	0/222	0/222
ZIPCODE	33/69	21/69	21/69	21/69	20/69	18/69	18/69	18/69	18/69	0/69	0/69

Read across any row and the shape is a step, not a slope. Read down any column and the steps are in different places. There is no single threshold at which this stack is tuned for precision: there are 20 labels each with their own edge, and a deployment picks one number for all of them.

Severity, and what could not be decided

Both tiers are graded with the same severity code the other audits on this site use, so the classes mean the same thing across projects.

	TAB	ai4privacy
documents	127	2,000
leaking at least one entity	127/127 (100.0%)	1,247/2,000 (62.4%)
class	undetermined	critical
detectable by mean score	false	false
underpowered	true	false

The TAB column is the honest one. Every document leaks, so there are 0 passing documents to compare against, and a detectability test with nothing on one side is not a negative result. The run records it as underpowered rather than as no signal found, and the class differs between the two tiers for exactly that reason.

On the synthetic tier there were two groups to compare, and the mean top detection score on leaking documents was 0.60 against 0.67 on non-leaking ones. A threshold fitted to that separation catches

377/1,247

leaking documents at the cost of 163 false positives on 753 non-leaking ones. The run records its best threshold as null, which is the point: the fit was made in sample, and a threshold chosen on the same documents it is scored against is not a result you can deploy.

What actually leaked, named

Rates hide what a leak is. These are recorded examples from the synthetic tier, where the value can be published because nobody real is in it.

label	value	outcome	anything predicted over it
TELEPHONENUM	+453 261.021.8097	miss	nothing
GIVENNAME	Urias	miss	nothing
AGE	60	miss	nothing
GIVENNAME	Wycliffe	miss	nothing
STREET	Mulmur-Nottawasaga Townline	miss	nothing
BUILDINGNUM	536	miss	nothing
CITY	Ellershouse	miss	nothing
ZIPCODE	N0G	miss	nothing
GIVENNAME	Ljundrim	miss	nothing
SURNAME	Kalluvettamkuzhiyil Litterini	miss	nothing
CITY	Spiritwood	miss	nothing
DATE	August/57	miss	nothing

12 of the 12 drew nothing at all. That is the failure mode this configuration has: not a low-confidence detection you could rescue by lowering a threshold, but silence.

What the labels are made of

Two synthetic labels are worth opening, because their names suggest something narrower than their contents.

SOCIALNUM means social insurance and national identity numbers, not only US social security numbers: 8 of its 47 spans are US, against 17 Canadian, 12 Indian and 10 British. A recognizer set built around US formats meets mostly non-US values here.

TITLE holds 26 distinct values across 222 spans. The commonest are honorifics, and the tail is offices:

value	spans
Mr	33
Master	30
Mayoress	28
Mister	25
Mrs	25
Madame	25
Miss	20
Mstr	17
Ofc	2
Attorney	1
Not	1
Captain	1

Every label, both tiers

The headline is one number over a mixed population. These are the populations.

Court texts, by gold class. full is redacted completely, partial is redacted in part, and miss drew nothing over any mention:

gold class	mentions	full	partial	miss
DATETIME	9,536	9,249/9,536 (97.0%)	245	42
PERSON	4,138	1,456/4,138 (35.2%)	2,314	368
ORG	1,946	315/1,946 (16.2%)	185	1,446
CODE	1,612	90/1,612 (5.6%)	21	1,501
LOC	1,459	1,104/1,459 (75.7%)	111	244
DEM	917	426/917 (46.5%)	88	403
QUANTITY	664	1/664 (0.1%)	23	640
MISC	537	47/537 (8.8%)	50	440

The partial column is where the partial-redaction trap lives, per class. Where it is large the class is one a lenient scorer will flatter, and where it is near zero the class is either handled or invisible, with no middle.

Synthetic rows, by gold label:

gold label	mentions	full	partial	miss
GIVENNAME	1,646	1,174/1,646 (71.3%)	18	454
SURNAME	658	470/658 (71.4%)	30	158
TELEPHONENUM	326	116/326 (35.6%)	99	111
CITY	323	224/323 (69.3%)	19	80
TIME	316	160/316 (50.6%)	3	153
DATE	283	220/283 (77.7%)	10	53
EMAIL	250	246/250 (98.4%)	4	0
TITLE	222	52/222 (23.4%)	0	170
STREET	218	138/218 (63.3%)	20	60
BUILDINGNUM	193	59/193 (30.6%)	0	134
IDCARDNUM	183	52/183 (28.4%)	0	131
AGE	105	35/105 (33.3%)	0	70
PASSPORTNUM	73	73/73 (100.0%)	0	0
ZIPCODE	69	33/69 (47.8%)	3	33
SEX	63	1/63 (1.6%)	0	62
GENDER	56	1/56 (1.8%)	0	55
CREDITCARDNUMBER	47	44/47 (93.6%)	0	3
SOCIALNUM	47	47/47 (100.0%)	0	0
DRIVERLICENSENUM	43	12/43 (27.9%)	5	26
TAXNUM	40	38/40 (95.0%)	0	2

10 of the 20 labels record a partial redaction.

That is the difference between the two corpora in one column. Court prose names a person many ways across a judgment, so a redactor can cover some mentions and miss others. Generated rows carry one span per entity, so a span is either covered or it is not, and there is no partial state to be flattered by.

Every control, named

The rate above is 2/15 (13.3%). Here are the documents behind it.

document	detections	what fired
neg_gardening	2	DATE_TIME on "annual"; DATE_TIME on "each season"
neg_backup_policy	1	DATE_TIME on "annually"
neg_recipe	0	nothing
neg_release_notes	0	nothing
neg_style_guide	0	nothing
neg_safety_manual	0	nothing
neg_math_note	0	nothing
neg_procurement_policy	0	nothing
neg_hiking	0	nothing
neg_api_docs	0	nothing
neg_coffee	0	nothing
neg_warehouse	0	nothing
neg_music_theory	0	nothing
neg_knitting	0	nothing
neg_incident_process	0	nothing

The 2 that fired did so on "annual", "each season" and "annually", all typed DATE_TIME. Those words genuinely are temporal expressions, so this is a scope mismatch rather than a detector bug: the DATE_TIME recognizer is wider than PII. It is still 3 spans on a reviewer's desk.

And the positive controls, with everything each one drew rather than only the entity it was written for:

control	expected type	found	everything it drew
pos_email	EMAIL_ADDRESS	true	EMAIL_ADDRESS at 1; URL at 0.5; DATE_TIME at 0.85
pos_person	PERSON	true	PERSON at 0.85
pos_phone	PHONE_NUMBER	true	PHONE_NUMBER at 0.4
pos_credit_card	CREDIT_CARD	true	CREDIT_CARD at 1
pos_ip	IP_ADDRESS	true	IP_ADDRESS at 0.6; PHONE_NUMBER at 0.4

2 of the 5 positive controls drew more than the entity they were written for. That is the same scope mismatch the negatives show, seen from the other side: the analyser is not only finding the identifier, it is finding other things in the same sentence, and each of those is a span a redactor would black out.

The second pass over the output, and what it is worth

A detectability threshold has to be fitted against the leak labels, so it is a diagnostic and not something a pipeline can run. The second check is different in kind: take the redacted output, and look for a structured identifier shape that survived. No answer key, no annotation, just a pattern over the text you are about to release.

second pass over redacted output only: flag document if a structured PII shape (SSN/email/phone/card/IP pattern) survives redaction

	TAB	ai4privacy
leaking documents	127	1,247
caught by the check	0/127 (0.0%)	125/1,247 (10.0%)
false positives	0	0
non-leaking documents to test against	0	753
deployable	true	true

On the synthetic tier it catches 125/1,247 (10.0%) of the leaking documents with 0 false positives. That is a small fraction of the leaks and it is still the most useful number on this page, because it is the only one here that a deployment can act on without an answer key.

It catches what it is shaped for: identifiers with a syntax. It cannot see a leaked name, because a name has no pattern to match, and names are the largest leaking class in the court texts.

Corpus independence, measured rather than assumed

Jaccard over lowercase alphanumeric tokens, after removing tokens shared by 90% or more of that tier, which is its format rather than its content. Clusters are connected components, so three mutually similar documents count as two redundancies, not three. A property of the corpora, computed from committed files by committed code. No model call and no ground truth, so it can be recomputed at any time without re-running the detector.

	TAB	ai4privacy
documents	127	2,000
candidate pairs examined	8,001	305,819
exhaustive	true	false
redundant documents	1	0
independent documents	126	2,000
closest pair below threshold	0.74	0.67

Reproducing this

The measurements file this page reads is a copy of the run, produced by scripts/sync_measurements.mjs from:

file	sha256
projects/p2_pii_redaction/results/run_pii.json	e26d49a6e1ac94cfdffab264d49c12788f7ea84d25d1f9fc4c7f221171177f0d
projects/p2_pii_redaction/results/corpus_independence.json	af7ef01a354e216c5dc510d88e10b6bf6304b0bd7ee8071f20e0b0e773bf70de

2,127 per-document rows are held out of the site copy for page weight, under tiers.*.doc_summaries, and ship in full inside the archive below. Nothing else is dropped. Per-record rows are omitted here for page weight and ship in the source archive. The summaries in this file are computed from them. Keys listed in merged are not from the run: they are properties of the committed corpus, computed without a model or ground truth, and are merged here so the page can state them without them being mistaken for measurements of the run. Keys listed in renamed_keys were renamed on copy to avoid colliding with this block. Prose elsewhere in this file was written against the source and still refers to them by their original names.

What this run cannot claim

TAB precision is a proxy, because its annotation is not exhaustive over unmarked text. The synthetic tier measures Presidio against a generator’s distribution, not production documents. This is stock configuration only: English, one spaCy model, no custom recognizers, no context words. That is the floor a team starts from, and it is what default settings give you.

Full per-entity tables, threshold sweeps, raw counts and the per-document records are inside the downloadable archive. The recognizer scores in the threshold section are measured by the analyzer rather than derived from the corpus, and the run file records them under recognizer_behaviour beside everything else.