Microsoft Presidio out of the box: what it leaves behind

Microsoft Presidio is the default answer to “we need to strip PII before this text goes anywhere”. It is open source, it is maintained, and it is a genuinely reasonable place to start. The question nobody publishes an answer to is what the stock configuration actually catches.

We measured it. Two corpora, never pooled, and every figure below is rendered from the results file of our PII redaction audit.

Controls first

15 documents written to contain no identifying information went through before anything else. A redactor that flags clean text is manufacturing review work. Stock Presidio flagged 2 of 15, with 3 detections between them. Positive controls, one obvious entity each, were all detected; the run aborts if they are not.

That is the shape of an honest measurement: if the tool cannot pass the trivial case, nothing below would mean anything.

The headline: 63.5% of entities fully redacted

On 127 real ECHR court judgments, human-annotated at character level:

	count
entities fully redacted	11,076/17,448
partly redacted	2,506
missed entirely	3,866

An entity is a person, a case number, an organisation. In a court judgment the applicant is named a dozen times, and redacting eleven of those still identifies them. So the number that matters is how many entities were redacted completely, every mention, every character. That is 63.5%.

On 2,000 synthetic rows the same measurement gives 3,195/5,161, 61.9%. Two very different corpora, a similar answer.

The partial-redaction trap

This is the finding worth carrying away, because it changes how you read every redaction benchmark you will see.

Direct identifiers, the spans that name a person outright, were fully redacted in 274 of 1,612 annotated mentions: 17.0%.

Grant credit for any overlap, the way token-level scoring does, and the same run reads as 62.5%: 274 fully redacted plus 733 partially covered, against the same 1,612.

Those are the same detections. One number is 17.0% and the other is 62.5%, and the difference is entirely a scoring choice. If the redactor turns “John Smith” into “John [REDACTED]”, token-level scoring books it as mostly correct. The person is still identified.

For a privacy tool there is no partial credit. An entity is either gone or it is leaked.

What leaks

Across the court texts, 8,121 annotated mentions were left in the output, spread across 8 entity classes. The three largest are PERSON (2,682), ORG (1,631), CODE (1,522).

That ordering is what this corpus produced. The run did not isolate why, so treat it as a place to look on your own documents rather than as a general property of Presidio: count leaks per entity class on your corpus before deciding which recognizer to strengthen.

What this does not say

Presidio is not broken, and this is not an argument against using it. It is an argument against deploying the defaults and reporting a token-level F1 as if it were a safety guarantee.

Specifically, this run measured stock configuration only: English, one spaCy model, no custom recognizers, no context words, no deny lists. That is the floor a team starts from, not the ceiling. Every one of those knobs exists precisely because the defaults are a starting point.

The full audit, including per-entity tables, the threshold sweep and the raw per-document records, is in the PII redaction audit, with the code archive attached so the run can be repeated.

If you are deciding whether a redaction pipeline is safe to put in front of a regulator, that is what our AI automation audit is for.