Google BigQuery Connector

Overview

Used for OneTrust Data Discovery, the Google BigQuery connector is able to scan data from the Google Cloud Platform (GCP) for any users who leverage BigQuery's serverless architecture to manage their data.

Pod Residency

storage-scanner

Connection Method

JSON Web Token, Google Service Account Authentication

Supported Scan Actions

Scan Type

Description

Catalog

This scan type fetches only the metadata in the project like details of dataset, table, or column for scanning.

Catalog & Scan

This scan types fetches both metadata and data under the tables and returns the appropriate labels.

Credential and System Permissions

OneTrust recommends to adhere to your organization's security and infrastructure policy and include system engineer resources or other qualified subject matter experts to set up credentials on data sources. For specific instructions for how to enable these credentials, refer to your data source's implementation documentation, knowledge base and/or support resources.

This connector requires JWT Google authentication and a Google Service Account.

About Google Service Accounts

Creating a service account is similar to adding a member to your project, but the service account belongs to your applications rather than an individual end user. When you create a service account, you must provide an alphanumeric ID (SA_NAME in the samples below), such as my-service-account. The ID must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you cannot change its name.

The service account's name is a unique identifier. It will appear in the service account's email address that is provisioned during creation, such as SA_NAME@PROJECT_ID.iam.gserviceaccount.com. Each service account also has a unique numeric ID, which is generated automatically. You also provide the following information when you create a service account:

  • SA_DESCRIPTION: optional description for the service account.

  • SA_DISPLAY_NAME: friendly name for the service account.

  • PROJECT_ID: ID of your Google Cloud project.

To create a service account, at minimum the user must be granted the Service Account Admin role (roles/iam.serviceAccountAdmin) or the Editor basic role (roles/editor). You should not grant basic roles in a production environment, but you can grant them in a development or test environment.

Sharing Datasets with other Service Accounts

It is possible to share the dataset and the table with other service accounts. There is also an option to specify the role that we wish to provide to the other service accounts while it is shared. Select the Add Member option in the console share options to grant the necessary role permissions for sharing datasets.

Sample JSON

Note:

If "project_id": is included, the connector will allow access to the specified project. If this is not specified, the connector will discover any and all accessible projects.

JSON 
{
    "type": "service_account", 
    *"project_id": "<your-project-id>", 
    "private_key_id": "<your-private-key-id>", 
    "private_key": "<your-private-key>", 
    "client_email": "<your-client-email">,
    "client_id": "<your-client-id>", 
    "auth_uri": "https://accounts.google.com/o/oauth2/auth", 
    "token_urk": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", 
    "client_x509_cert_url": "<your-client-cert-url>"
}

Required Permissions

  • bigquery.datasets.get

  • bigquery.datasets.getIamPolicy

  • bigquery.models.export

  • bigquery.models.getData

  • bigquery.models.getMetadata

  • bigquery.models.list

  • bigquery.routines.get

  • bigquery.routines.list

  • bigquery.tables.export

  • bigquery.tables.get

  • bigquery.tables.getData

  • bigquery.tables.getIamPolicy

  • bigquery.tables.list

  • resourcemanager.projects.get

  • resourcemanager.projects.list

Enabling Cloud Resource Manager API

You will need to enable the Cloud Resource Manager API in order to discover projects in the Google Cloud Platform.

  1. Log in to your Google Cloud console.

  2. Navigate to APIs & Services > Enabled APIs & services.

  3. Select Cloud Resource Manager API.

  4. Click the Enable button to leverage this API.



Scan Profile Support

Note:

For additional details on scan profile configuration and management, see Setting Up Scan Profiles.

Limits

  • Maximum number of projects

  • Maximum number of datasets per project

  • Maximum number of tables per dataset

Include/Exclude Paths

  • Dataset name

  • Project name

  • Table name

Collected Objects

Object

Description

Projects

Top-level containers in the Google Cloud Platform that store the data. Within the project, the configuration of settings, permissions, and other metadata that describe the applications can be done. Each project has a name, ID, and number that could be used as identifiers.

Datasets

Datasets allows to organize and control access to tables. All tables must belong to a dataset. A dataset must be created before loading the data into BigQuery. You can configure permissions at the organization, project, and dataset level.

Tables

Within datasets, tables are row-column structures that hold actual data. Each table has a schema that describes the data contained in the table, including field names, types, and descriptions. BigQuery supports the following table types:

Note:

OneTrust can support strut(nested data arrays)

  • Native tables: tables backed by native BiqQuery storage

  • External tables: tables backed by external storage (Not currently supported by Data Discovery)

  • Views: virtual tables defined by a SQL query (Not currently supported by Data Discovery)

Jobs

The tasks you are performing on the data, such as running queries, loading data, and exporting data. Jobs are objects that manage asynchronous tasks such as running queries, loading data, and exporting data. There are four types of jobs:

  • Load

  • Query

  • Extract

  • Copy