Interpreting Confidence Scores

Confidence scores indicate the relative distance between the data sample being evaluated, and the closest surrounding class labels. The application will detect the user input and gather the assumed data elements from this content. Results will populate with a presumed PII label and percentage indicating the confidence of the match.

How it Works

The confidence score is assigned a OneTrust expert who creates the classifier after research and analysis. The confidence score is tested with real world-data while the initial assignment to a classifier. Regular adjustments to confidence scores are made based on feedback from multiple sources (e.g. production, data drifts, customer feedback etc.). The AI models used are adapted from industry standard models, which are trained on a large volume of data and have high accuracy and recall rates.

Examples

An exact match of 'E-Mail' to the labeled data of 'E-Mail' would result in a confidence score of 100% as the distance between these two is 0. However, matching a label like date which could be Employment Date, Date of Birth, or just a timestamp, would yield a lower confidence score.

In these scenarios, additional context can help improve the confidence score. For example, if a date of "03/10/1987" is found in a column of field labeled 'DOB' a higher confidence score can be assigned as this extra context helps reduce ambiguity.

If test data has more than one possible PII label, the application will attempt to classify the data in its used context and produce separate labels and respective scores. Confidence scores are particularly helpful if there is a data element with a possible duplicitous meaning (e.g. city/country names vs. given first names with data elements like Paris, London, Jordan).

Best Practices

Business users should consider their organization's accepted threshold for confidence score percentages for a more efficient approval process, especially given the large volumes of data points your organization may attempt to classify. A 5-10% difference between levels may seem minor; however, approving an incorrect classification could affect your data discovery results and unintended downstream issues. Manually testing and inspecting scores outside of your organization's accepted threshold will help ensure the data classification is accurate to the presumed PII label before approving the classification.

Factors Impacting Confidence Scores

  • Type of sensitive data included in the scan (e.g. credit card) has checksum logic for better validation and thereby higher confidence score (vs SSN which do not have checksum logic).

  • Specific patterns of the sensitive data found and the mix of those patterns in the database e.g. 876-223-3452 is more likely to be a US phone number and will get higher confidence than 8762233452.

  • The similarity of sensitive data with others e.g. “Georgia” can be both state or name depending on the context

  • The context where the sensitive data is found (e.g. header name with “SSN” in a structured database or presence of “social security number” adjacent to the number with pattern XXX-XX-XXXX in an unstructured database has higher confidence for SSN.