Presidio PII: the confidence threshold is a recall cliff
Raising the Presidio score threshold reads like a precision knob. Measured across a full sweep it is a cliff, and names fall off it in one step.
PII detectors give you a confidence score, and the usual advice is to set a threshold on it. The mental model is a dial: turn it up, get fewer false positives, accept slightly fewer catches.
We ran the whole dial, 20 thresholds from 0 to 0.95, against 2,000 annotated documents, and that is not what it does. The figures come from our PII redaction audit.
Names do not degrade. They fall off a cliff.
spaCy’s NER emits a fixed score for person entities in this configuration. Across the probe sentences the run tested, the only observed score is 0.85. One value, not a distribution.
That single number decides everything. Recall is flat from 0 all the way to 0.85 and is zero at 0.9.
| threshold | GIVENNAME | SURNAME |
|---|---|---|
| 0 | 1,174/1,646 | 470/658 |
| 0.6 | 1,174/1,646 | 470/658 |
| 0.85 | 1,174/1,646 | 470/658 |
| 0.9 | 0/1,646 | 0/658 |
There is no gentle trade-off in that table. Presidio does not catch every name at any threshold: it catches 1,174/1,646 given names at the bottom of the dial. What the step does is take the names it does catch to zero, in one move.
Set the threshold to 0.9 because it sounded prudent and you turned off name detection completely, and the tool did not tell you.
It is not just names
Of the 20 labels with any recall at the bottom of the dial, 19 have lost more than half of it by 0.95.
EMAIL is the exception, and not a small one: 246/250 at 0 and 245/250 at 0.95, one mention across the whole dial. Email addresses have a shape a regex can be certain about. Names do not.
So the threshold behaves less like a precision knob and more like a switch that turns off different categories of identifier at different points, with the edges in places you would have to sweep the dial to discover.
The other half: identifiers nobody scored confidently
Feed the stock analyser a plain identifier and watch which recognizers fire and how sure they are. On the probe “ID card 987654321 on file.”, 5 recognizers claim the same span: US_DRIVER_LICENSE, US_BANK_NUMBER, US_PASSPORT, US_SSN, US_ITIN. Their scores run from 0.01 to 0.3.
Every one of those is below a threshold most teams would consider conservative. Set the dial anywhere above 0.3 and this stack stops redacting bare ID numbers entirely, while still confidently redacting the email address two lines above it.
What to do instead
- Sweep the dial before you pick a number. The right threshold is not a matter of taste. It is a measurement, and it is different per entity type.
- Set thresholds per recognizer, not globally. A single number cannot serve a recognizer scoring 0.01 and one scoring 85.0%.
- Report recall per label, not one blended figure. A headline number hides the fact that one whole category went to zero.
The full sweep, all 20 thresholds against all 20 labels, is in the results file attached to the PII redaction audit, along with the code that produced it.
Related: what stock Presidio leaves behind, measured on real court judgments.