Masking Policies for Databricks

Overview

Organizations face an immense challenge in adhering to policy enforcements issued by governance councils and using data responsibly for their business needs. Translating those documented policies all the way to data controls on the warehouses where data is consumed is a complex and dynamic task. OneTrust's Policy Orchestration solution helps data teams institute a methodology for orchestrating policy enforcement to monitor and take action on policy violations with confidence.

This article captures how data governance teams using Databricks data warehouse can leverage OneTrust to mask data at the data source level and ensure only authorized users can view sensitive data. This is achieved by:

  • Setting up a data source connection to Databricks

  • Configuring a data policy with conditions to identify sensitive data that is unmasked

  • Customizing and applying policy actions to mask (or unmask) data for select user groups

Things to Know

  • All tags from Databricks will be saved as custom terms in OneTrust.

Prerequisites

Policy Orchestration authenticates to Databricks using a Personal Access Token, which can be associated with either a User or a Service Principal. Note that Personal Access Tokens expire after a configured number of days. To ensure continuity of services, please take measures to replace these tokens before they expire.

The following addresses the required queries and privileges.

Query

Purpose

Privileges Needed

SELECT CURRENT_TIMESTAMP

Connection / credentials test

Workspace and compute access

SHOW GROUPS

Populate UI role selection widgets

Workspace Administrator

SELECT from SYSTEM.INFORMATION_SCHEMA schema (tables: COLUMN__MASKS, COLUMN_TAGS, PARAMETERS, ROUTINES, ROW_FILTERS, TABLES, COLUMNS)

Scheduled ingestion of policies, schemas, and tags (classifications); application-time policy checks

For all applicable workspaces: USE CATALOG on SYSTEM catalog; USE SCHEMA on SYSTEM.INFORMATION_SCHEMA; SELECT on SYSTEM.INFORMATION_SCHEMA or on individual views

CREATE OR REPLACE FUNCTION

Create or update masking or row filter policy, in the schema with the tables to which the policy will be applied

CREATE FUNCTION on applicable catalog or schema; USE CATALOG on applicable catalog; USE SCHEMA on applicable schema

ALTER TABLE … ALTER COLUMN … SET / DROP MASK …

Apply / unapply masking policy

USE CATALOG on applicable catalog; USE SCHEMA on applicable schema; MANAGE on applicable table, schema, or catalog.

ALTER TABLE … SET / DROP ROW FILTER …

Apply / unapply row filter policy

USE CATALOG on applicable catalog; USE SCHEMA on applicable schema; MANAGE on applicable table, schema, or catalog.

Data Source Setup

  1. On the Data Discovery menu, select Setup > Data Sources. The Data Sources screen appears.

  2. Click the Add Data Source button. The Data Discovery Gallery screen appears.

  3. Select the Databricks tile. The Establish Connection screen appears.

  4. In the first section, choose an existing worker node.

    Note:

    For more information on creating and deploying a worker node, see Guided Worker Node Installation Overview.



  5. In the second section, provide, you can create new data source credentials or choose from existing Databricks credentials.

    Tip:

    Credentials can also be changed post setup in the Settings tab on the Data Source Details screen.



  6. If creating new credentials, provide information for the fields:



    Field
    Description
    Credential name

    Provide a unique name for this credential.

    Description

    Provide a meaningful description of the credential's intended use and purpose.

    Reference key

    Define the path to save or locate the securely stored credentials within the vault, ensuring efficient and protected access to your data source.

    Example format: "/path/<key-name>"

    Standard: Key-Value Pair

    Input the credentials as key-value pairs, where the key represents the connection field and the value represents the corresponding information. Custom keys can be added or removed.

    Advanced: JSON

    Input the credentials as a JSON file, where the key represents the connection field, and the value represents the corresponding information.

    Add credentials to vault (optional)

    This option simplifies the configuration of data source credentials by automatically adding them to the vault associated with your worker node.

  7. Click the Next button.

  8. Provide a unique name for this new data source.



  9. For the Set scan profile, you can either choose the system default profile (Databricks: Full Catalog), choose an existing profile, or create a new one.





  10. Click the Save button. The Data Source Details screen appears for the new data source.

  11. Click the Activate button.

  12. Start a scan for the new data source to begin ingesting data that has been classified by Databricks.

Databricks Column Masking

  1. You will need to create a data policy to detect violations on observed Databricks data that is sensitive but not yet masked. Additional conditions can be used to narrow the scope of the data.

    • The classification tag you use to detect sensitive data can be any custom/unique value. Our example uses Sensitive Personal Data.

    • Recommended: The policy can include the Policy condition, where the operator is Is Empty. This identifies data that is not masked.

      Note:

      If this is not specified, then OneTrust masking policies will replace an existing masking policy that was previously applied to the column. A column in Databricks can only have a single associated masking policy.



    Note:

    For more information on creating policies, see Creating & Managing Data Policies.

  2. Navigate to the Violations tab in the given Data Policy's details screen.

  3. Select one or more violations that represent columns which need to be masked.



  4. With multiple violations selected, click the Actions button to select Apply policy action. A side pane appears to create and apply a policy action.



  5. Provide a unique name for this action. This will generate the Data source policy identifier string. The Action type will default to Masking.

    Note:

    This name cannot be changed once the policy action is saved.



  6. Provide an optional description for reference of what this policy action is masking.

  7. Click the Next button to advance to the action rules.

  8. Under Data policy conditions, define the masking value. The value supplied for the mask condition will replace the sensitive data value. This defaults to REDACTED but can be updated to any text.



  9. Define inclusion/exclusion conditions on masking data for certain user groups:

    • Choose User groups except to specify which user types in Databricks that are authorized to view this data. This masks the data for all unselected users.



    • Choose Including user groups to mask data for all selected user types. All unselected users will be able to view the data.



  10. Click the Next button.

  11. Take a moment to review all of the policy action details. Use the Back button to make changes as necessary.

    Note:

    The modal below uses the User groups except masking option. If you selected the Including user groups masking option, those conditions will be shown here for review.



  12. Click the Save and apply button to activate the policy action. An application modal appears to show how many violations have been resolved.



  13. This created policy action will be available in the Policy actions menu. Use the View policy action link to navigate to the details screen.



Policy Action Management

List Screen

  1. Besides the link the Policy action applied modal, you can also access created policy actions from Policy Actions on the Data Catalog menu.

    • Policy actions that have hyperlinked names are actions created manually in OneTrust, as demonstrated in the above flow. Clicking the name will take you to the details page. Their Enforcement column value will be Manual.

    • Policy actions with no hyperlinked names are actions that are ingested directly from Databricks. You will not be able to manage these in OneTrust. Their Enforcement column value will be blank.



  2. Click the name of a policy action to enter the details screen.

Editing Policy Actions

  1. To update the policy action's conditions, click the three dots next to the name and select Edit. The Edit policy action modal appears.

    Note:

    You will not be able to edit policy actions if there are no resolved violations. At minimum there must be at least one resolved violation to edit.



  2. Update the masking value and user group selection as necessary, then click the Save and Apply button.



  3. Go to the Resolved violations tab to inspect what violations were resolved by the policy action.



  4. To remove the applied policy action, select one or more violations and click the Remove policy action button.



  5. A confirmation modal will appear to state how many violations will be restored. Click the Remove actions to remove the policy actions and revert the violations.



Deleting Policy Actions

Before you delete policy actions, ensure that you have addressed the following:

  • You can only delete a policy action if there are no resolved violations.

  • You can only delete a data policy if there are no associated policy actions.

  • You can only delete a data source if there are no associated policy actions.

  1. To delete a policy action, click the three dots next to the name and select Delete. The Delete policy action modal appears.



  2. Click the Delete policy action button to confirm. This action cannot be undone.