Google Bigtable Connector

Overview

Used for OneTrust Data Discovery Scanners, the Google Bigtable connector helps scan data and make more sense with metadata and other fields found by OneTrust across the distributed, column-oriented data store created by Google.

Pod Residency

nosql-scanner

Connection Method

JSON Web Token, Google Service Account Authentication

Supported Scan Actions

Scan Type

Description

Catalog

This scan type fetches only the metadata in the project like details of dataset, table, or column for scanning.

Catalog & Scan

This scan types fetches both metadata and data under the tables and returns the appropriate labels.

Credential and System Permissions

OneTrust recommends to adhere to your organization's security and infrastructure policy and include system engineer resources or other qualified subject matter experts to set up credentials on data sources. For specific instructions for how to enable these credentials, refer to your data source's implementation documentation, knowledge base and/or support resources.

This connector requires JWT Google authentication and a Google Service Account.

About Google Service Accounts

Creating a service account is similar to adding a member to your project, but the service account belongs to your applications rather than an individual end user. When you create a service account, you must provide an alphanumeric ID (SA_NAME in the samples below), such as my-service-account. The ID must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you cannot change its name.

The service account's name is a unique identifier. It will appear in the service account's email address that is provisioned during creation, such as SA_NAME@PROJECT_ID.iam.gserviceaccount.com. Each service account also has a unique numeric ID, which is generated automatically. You also provide the following information when you create a service account:

  • SA_DESCRIPTION: optional description for the service account.

  • SA_DISPLAY_NAME: friendly name for the service account.

  • PROJECT_ID: ID of your Google Cloud project.

To create a service account, at minimum the user must be granted the Service Account Admin role (roles/iam.serviceAccountAdmin) or the Editor basic role (roles/editor). You should not grant basic roles in a production environment, but you can grant them in a development or test environment.

Sharing Datasets with other Service Accounts

It is possible to share the dataset and the table with other service accounts. There is also an option to specify the role that we wish to provide to the other service accounts while it is shared. Select the Add Member option in the console share options to grant the necessary role permissions for sharing datasets.

Sample Authentication

JSON 
{
    "type": "service_account", 
    "project_id": "<your-project-id>", 
    "private_key_id": "<your-private-key-id>", 
    "private_key": "<your-private-key>", 
    "client_email": "<your-client-email">,
    "client_id": "<your-client-id>", 
    "auth_uri": "https://accounts.google.com/o/oauth2/auth", 
    "token_urk": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", 
    "client_x509_cert_url": "<your-client-cert-url>"
}

Required Permissions

  • resourcemanager.projects.list

  • resourcemanager.projects.get

  • bigtable.instances.list

  • bigtable.instances.get

  • bigtable.tables.list

  • bigtable.tables.get

  • bigtable.tables.readRows

Scan Profile Support

Note:

For additional details on scan profile configuration and management, see Setting Up Scan Profiles.

Limits

  • Number of Objects to Scan

  • Maximum Number of Projects

  • Maximum Number of Instances per Project

  • Maximum Number of Tables per Instance

  • Maximum Number of Rows per Table

Include/Exclude Paths

  • Project

  • Instance

  • Table

Collected Objects

Object

Description

Project

Top-level containers in the Google Cloud Platform that store the data. Within the project, the configuration of settings, permissions, and other metadata that describe the applications can be done. Each project has a name, ID, and number that could be used as identifiers.

Instance

A container for Bigtable clusters. An instance that has more than one cluster uses replication. You can create clusters in up to 8 regions, with as many clusters in each region as there are zones.

Table

A collection of user data indexed by row, column, and timestamp. Each table is served using the resources of its parent cluster.