Sensitive data leakage refers to the inadvertent exposure or disclosure of confidential or private information. There are 2 scenarios where you may want to detect sensitive data to protect your company and users:

1. Detecting Sensitive Data in the Prompt - If your application is using a public or externally-hosted LLM, all data that is sent to that model may be fed back into the training data or shared across contexts with other users. In this case, you want to ensure that your user is not accidentally including sensitive company information in the prompts to the LLM.
2. Detecting Sensitive Data in the Response - Teams develop systems to add private data as context to LLMs (through fine-tuning or retrieval) to maximize the types of questions that the LLMs can answer. However, your end users may not have the right data permissions to view that private data granularly. In this case, your application should not return information considered sensitive to that end user.

Types of Sensitive Data

Broadly defined, sensitive data is confidential information stored, processed, or managed by an organization that should only be accessible by authorized users. The types of data that fall under this categorization are use-case specific.

For example, imagine we have augmented an LLM with client investment portfolio data. We want our internal users to be able to take advantage of the productivity increase by utilizing our application to ask broad questions across portfolios. However, we would want to block responses that return specific user information or client holdings they should not have access to.

Another common type of evaluation that teams may consider to be sensitive data is personal identifying information (PII). This can be in the form of common PII (such as social security numbers) or use-case-specific identifiers (e.g., API-key tokens). Please refer to our PII Leakage documentation or our Custom Rules documentation for more information on those Shield rules.

The Shield Approach

Arthur's approach leverages LLM-prompting with few-shot examples as a general purpose tool for detecting sensitive data that would be otherwise hard to capture via concrete regex patterns.

To use this rule, prepare a set of examples that you expect to PASS or FAIL according to your category of sensitive data you need to detect. Once this example set is prepared, you configure your sensitive data rule with your examples, and the rule is then used to detect your category of sensitive data from then on.

Preparing examples to configure your sensitive data rule

Positive and negative examples

It is helpful to collect both positive and negative examples to configure your rule (for example, if you wanted to detect medical data with your sensitive data rule, positive examples would be instances of medical text and negative examples would be instances of some other type of non-medical text). Finding negative examples to go with your positive examples reduces the false positive rate of your rule by giving the LLM more examples of what not to flag.

Sufficient examples

It is helpful to collect roughly 2 to 20 examples of the type of sensitive data you need to detect when you are configuring your sensitive data rule.

Typically performance improves by adding more examples, but we occasionally observed a performance dropoff adding too many examples - this dropoff tended to begin at 10 to 15 examples.

Be sure to test your rule on a validation set before productionizing your rule to determine which set of examples works best for your use case!

Adding Hints

It is helpful to add a description of the type of sensitive data you are looking for as a hint when you are creating the sensitive data rule.

Adding a hint performs quite well on our benchmarks overall - it lifts the performance of the check across benchmark datasets while not using that many tokens.

Benchmarks

The benchmark datasets we assembled contain samples from the AI4Privacy PII-Masking dataset. Each dataset is focused on one category of sensitive data from the types included in the AI4Privacy dataset: Passwords, Usernames, Company Names, and Job Titles (we are benchmarking on categories of sensitive data not covered by the shield PII Leakage rule)

When running the benchmarks, we sample N = 30 rows from each dataset.

Each row from our benchmark datasets contains a text and a label, which is a boolean. If label == True, this indicates that the text contains the type of sensitive data the dataset is focused on. For example, if text contains a username in the Username dataset we label it True, but if text contains a username and it is in the Company Names dataset (and text doesn't contain a company namy) we label it False

Our Recommendation for Your On-Premises Shield Deployment

Based on these benchmarks, we recommend using gpt-3.5-turbo-0125 with ~8 examples and a hint - this seems to be a high performing and relatively low-cost and low-latency configuration.

Consult the Azure OpenAI Model documentation to see which models work best for your deployment.

Benchmark Data

Note on Cost & Latency: Numbers represent the total calculated for an entire benchmark set of N = 30. Cost is computed as the # of prompt & completion tokens used, multiplied by their respective prices.