Row Filtering Policies for Databricks

Organizations face an immense challenge in adhering to policy enforcements issued by governance councils and using data responsibly for their business needs. Translating those documented policies all the way to data controls on the warehouses where data is consumed is a complex and dynamic task. OneTrust's Policy Orchestration solution helps data teams institute a methodology for orchestrating policy enforcement to monitor and take action on policy violations with confidence.

This article captures how data governance teams using Databricks data warehouses can leverage OneTrust to detect tables and table columns that require row-level filtering based on data usage policies. This is achieved by:

• Setting up a data source connection to Databricks.

• Active worker node created through Data Discovery

• Configuring a data policy with conditions to identify table columns with sensitive data.

• Customizing and applying policy actions to filter out rows that may contain sensitive PII and/or grant user entitlements for authorized access to these rows.

Policy Orchestration authenticates to Databricks using a Personal Access Token, which can be associated with either a User or a Service Principal. Note that Personal Access Tokens expire after a configured number of days. To ensure continuity of services, please take measures to replace these tokens before they expire.

The following addresses the required queries and privileges.

1. On the Data Discovery menu, select Setup > Data Sources. The Data Sources screen appears.

2. Click the Add Data Source button. The Data Discovery Gallery screen appears.

3. Select the Databricks tile. The Establish Connection screen appears.

4. In the first section, choose an existing worker node.

   Note:

   For more information on creating and deploying a worker node, see Guided Worker Node Installation Overview.

   
   
   

5. In the second section, provide, you can create new data source credentials or choose from existing Databricks credentials.

   Tip:

   Credentials can also be changed post setup in the Settings tab on the Data Source Details screen.

   
   
   

6. If creating new credentials, provide information for the fields:

   
   
   

   Field

   Description

   Credential name

   Provide a unique name for this credential.

   Description

   Provide a meaningful description of the credential's intended use and purpose.

   Reference key

   Define the path to save or locate the securely stored credentials within the vault, ensuring efficient and protected access to your data source.

   Example format: "/path/<key-name>"

   Standard: Key-Value Pair

   Input the credentials as key-value pairs, where the key represents the connection field and the value represents the corresponding information. Custom keys can be added or removed.

   Advanced: JSON

   Input the credentials as a JSON file, where the key represents the connection field, and the value represents the corresponding information.

   Add credentials to vault (optional)

   This option simplifies the configuration of data source credentials by automatically adding them to the vault associated with your worker node.

7. Click the Next button.

8. Provide a unique name for this new data source.

   
   
   

9. For the Set scan profile, you can either choose the system default profile (Databricks: Full Catalog), choose an existing profile, or create a new one.

   
   
   
   
   

10. Click the Save button. The Data Source Details screen appears for the new data source.

11. Click the Activate button.

12. Start a scan for the new data source to begin ingesting data that has been classified by Databricks.

Our entitlement-based example will filter rows from a Databricks table that contains sales data across regions. The policy action will be configured to filter on columns containing sales data from a particular region that sales representatives are only shown data unique to their area. The steps will mention when a reference table can be add to demonstrate the more complex filtering scenario.

1. You will need to create a data policy that can detect violations in Databricks tables and identify row values that should be considered sensitive and filtered from viewing. Additional conditions can be used to narrow the scope of the data.

   • The data policy should be configured to return table-level violations. This can be enabled when creating the data policy.

     
     
     

   • Recommended: The policy can include the Policy condition, where the operator is Is Empty. This identifies data where the table does not have a row-filtering policy applied.

     
     
     
   
   
   
   Note:

   For more information on creating policies, see Creating & Managing Data Policies.

2. Once your data policy is active and has detected violations, navigate to the Violations tab in the given Data Policy's details screen.

3. Select a violation that represents a table that needs to be filtered.

4. With your table violation selected, click the Actions button to select Apply policy action. The Create and apply policy action screen appears.

   
   
   
   Important:

   If you attempt to select multiple violations, you will receive a modal you that policy actions can only be applied to one table violation at a time.

5. In the Add details section, given this new policy action a unique name and optional description. The Data source policy identifier field will generate when you provide the name using the format [ONETRUST_NAME].

   The Action type field defaults to Row level filtering.

   
   
   

6. Click the Next button. The Define policy action screen appears.

7. In the Column filter section, select the column you want to filter from the table.

   
   
   

8. In the User entitlements section, select the Yes option to configure which reference table you want to grant entitlements for:

   Reference table

   : choose a table from the available options that stores and defines user access rights.

   Note:

   Optional: intended for entitlement-based filtering that requires a reference table to associate the desired columns and filtered rows.

   Join column

   : choose the corresponding column that matches your filter selection in the previous step.

   Entitlement column

   : choose the column that specifies which data rows a user or role is allowed to access.

   
   
   

9. In the Select user entitlements to use section, click the Create user entitlements button or select existing an existing one.

   
   
   

10. Provide a unique name for the user entitlement, then define the entitlement conditions for your users that is consistent with their job function and level of access required. Our use case will be granting the Southwest Sales Rep with access to view rows only with the SOUTHWEST value, while our Sales Manager has full access.

    
    
    

11. Click the Save button. A success message will appear when the user entitlement is created.

    
    
    

12. Click the Next button. The Review policy action screen appears.

13. Take a moment to review all of the policy action details. Use the Back button to make changes as necessary.

    
    
    

14. Click the Save and apply button to activate the policy action. An application modal appears to show how many violations have been resolved.

    
    
    

15. This created policy action will be available in the Policy actions menu. Use the View policy action link to navigate to the details screen.

Our consent-based example will filter rows from a Databricks table that contains sensitive student information. The policy action will be configured to filter on columns containing student IDs and parental consent for AI data use so that our data scientist is only shown rows meeting a specific entitlement.

1. You will need to create a data policy that can detect violations in Databricks tables and identify row values that should be considered sensitive and filtered from viewing. Additional conditions can be used to narrow the scope of the data.

   • The data policy should be configured to return table-level violations. This can be enabled when creating the data policy.

     
     
     

   • Recommended: The policy can include the Policy condition, where the operator is Is Empty. This identifies data where the table does not have a row-filtering policy applied.

     
     
     
   
   
   
   Note:

   For more information on creating policies, see Creating & Managing Data Policies.

2. Once your data policy is active and has detected violations, navigate to the Violations tab in the given Data Policy's details screen.

3. Select a violation that represents a table that needs to be filtered.

   
   
   

4. With your table violation selected, click the Actions button to select Apply policy action. The Create and apply policy action screen appears.

   
   
   
   Important:

   If you attempt to select multiple violations, you will receive a modal you that policy actions can only be applied to one table violation at a time.

5. In the Add details section, given this new policy action a unique name and optional description. The Data source policy identifier field will generate when you provide the name using the format [ONETRUST_NAME].

   The Action type field defaults to Row level filtering.

   
   
   

6. Click the Next button. The Define policy action screen appears.

7. In the Column filter section, select the column you want to filter from the table.

   
   
   

   Our use case is going to filter on the Student ID column from our databricks data source.

   
   
   

8. In the User entitlements section, select the Yes option to configure which reference table you want to grant entitlements for:

   Reference table

   : choose a table from the available options that stores and defines user access rights. Our use case is looking at the parental consent given for AI data use consent.

   Note:

   Optional: intended for consent-based/map entitlements to

   Join column

   : choose the corresponding column that matches your filter selection in the previous step. Our use case will join the Student ID column.

   Entitlement column

   : choose the column that specifies which data rows a user or role is allowed to access. Our use case will select the AI_DATA_USE_CONSENT column.

   
   
   

9. In the Select user entitlements to use section, click the Create user entitlements button or select existing an existing one.

   
   
   

10. Provide a unique name for the user entitlement, then define the entitlement conditions for your users that is consistent with their job function and level of access required. Our use case will be granting the Data Scientist with access to view rows only with the GRANTED value, while our Account Admin with full access.

    
    
    

11. Click the Save button. A success message will appear when the user entitlement is created.

    
    
    

12. Click the Next button. The Review policy action screen appears.

13. Take a moment to review all of the policy action details. Use the Back button to make changes as necessary.

    
    
    

14. Click the Save and apply button to activate the policy action. An application modal appears to show how many violations have been resolved.

    
    
    

15. This created policy action will be available in the Policy actions menu. Use the View policy action link to navigate to the details screen.