Guided Worker Node Installation Overview

You can opt to follow a guided installation process when creating your on-premises worker node for OneTrust Data Discovery. This screen-by-screen walkthrough will queue several important details about your desired data source, worker node deployment options, the available discovery scan and catalog options, and many more advanced settings for your best use cases. The end product will be an available worker node that can be used immediately without much further configuration as the script will run and ask you to follow the on-screen instructions.

For information on how to manually install the worker node, see Manual Worker Node Installation Overview.

Prerequisites

You will need at least the following to support worker node creation with OneTrust Data Discovery. The platform-specific installation articles in Data Discovery Knowledge Base will provide additional detail around what support is needed for the given deployment.

  • Environment with nodes to support at least 8 vCPU(s), 32 GB RAM

  • A subscription which can run the required cluster

  • An owner/contributor user of the given subscription which will be used to access, create, and manage the cluster

  • A local (laptop) or remote (Jump box/bastion) machine which will be used to access the cluster master. Minimum: 4 CPU, 8GB RAM, 250 GB HDD, Ubuntu 16+

    Note:

    Not applicable for Onebox Microkubernetes Service deployments.

  • Virtual Network and subnet in which the given cluster is getting deployed should have access to data sources

  • Virtual Network and subnet in which the given cluster is getting deployed should have access to Internet to reach Docker Hub

  • Docker >=19.x ce + pre-installed (OneTrust can also install this)

  • Latest version of kubectl to ensure full compatibility of services

Create a Worker Node

  1. On the Data Discovery menu, select Setup > Worker Nodes. The Worker Nodes screen appears.

  2. Click the Add Worker Node button. The Create Worker Node modal appears.

  3. Enter a unique name for this new worker node. The name you enter here will be populated as the namespace in the follow screen. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.

  4. Select the Guided Install option.

    • The Managing Organization field will default to the parent org level.

    • The Worker Node Owner field will default to the current user.

    • The Label field can be used to provide a description of this node uses or purpose(s).

    Optional: With the guided installation option, you can also choose to create the new node by copying an existing guided worker node template configuration.

    Note:

    For more information on worker node owners and email alerts, see Managing Email Notifications for Worker Nodes.



  5. Click the Next button. The Platform Provider and Credential Store screen appears. Proceed to Configuring the worker node.

    If using the worker node template option, the Enter PIN modal appears. See step #7 for more.

  6. In order to use an existing worker node configuration, please enter the PIN created as part of the templated worker node creation. For more information on how to create and obtain a PIN, see Worker Node PIN.

    Note:

    OneTrust 202403.1.0 supports storing worker node PINs in the given namespace as a Kubernetes secret to maintain PIN access and easily retrieve the PIN if forgotten or lost.



  7. Click the Next button. The Platform Provider and Credential Store screen appears.

Configuring the worker node

By selecting the Guided Install option, you can then proceed with configuring the worker node by following the wizard steps. In this example, we're creating a worker node to use with the Azure platform. The majority of these fields will be generally available to any platform provider.

See Data Discovery Deployment Options for more information on available systems.

  1. In the Platform Provider and Credential Store screen, select your platform provider. The interface will auto-populate fields applicable to the chosen system. Here, we have selected AWS.

  2. In the Platform Provider block, select your desired provider from the field and then complete the fields for given configuration.

    See the provider information tables below for more information.



  3. In the Credential Store block, select your desired credential store from the field and then complete the fields with your applicable vault details.



    Note:

    For more information on support and configuration of credential vaults, see Credential Vault Options for Data Sources.

    Field
    Description
    Credential Store

    Select from the available Credential Store choices. The selection will populate unique fields for each store option.

    Deploy with Native Authentication (specific to AWS Secrets Manager)

    Enable this setting to delegate permissions using IAM role(s) to make certain API calls for fetching credentials.

    Store Options

    AWS Secrets Manager

    Azure Key Vault
    CyberArk - Conjur
    Google Secrets Manager
    HashiCorp Vault - Open Source (Internal)
    HashiCorp Vault - Open Source (External)
    HashiCorp Vault - Enterprise
    Note:

    See Data Discovery Deployment Options for additional information on current deployment options for OneTrust Data Discovery.

  4. Click the Next button. The Advanced Options screen appears.

AWS Fields

Field
Description
Automate Infrastructure Deployment

Select Yes to easily perform worker node installations for cluster setup. This requires the Worker Node Size and Storage.

Select No to populate the fields below and continue with this configuration.

AWS Region

Enter the appropriate AWS region for your environment.

EKS Cluster Name

Enter the cluster name of your Elastic Kubernetes Service.

Storage

Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.

If External NFS, you must also provide the following:

  • NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.

  • NFS Path: Enter the NFS path name for the file system to be mounted.

If Elastic File System (EFS), you must also provide the following:

  • Elastic File System ID: Enter the ID of the Amazon Elastic File System.

  • Elastic File System Region: Enter the region associated with the Amazon Elastic File System.

Namespace name

This field will populate with your entered worker node name by default. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.

Deploy with Instance Profile

Enable this setting to pass the IAM role for authentication to the cluster. For more information on Instance Profile deployments, see AWS Instance Profile Deployment Guide.

AWS Fargate Fields

Field
Description
Automate Infrastructure Deployment

Select Yes to easily perform worker node installations for cluster setup. This requires the AWS Account ID and AWS Region.

Select No to populate the fields below and continue with this configuration.

AWS Region

Enter the appropriate AWS region for your environment.

EKS Cluster Name

Enter the cluster name of your Elastic Kubernetes Service.

Elastic File System ID

Enter your Elastic File System ID.

Elastic File System Region

Enter your Elastic File System Region.

Namespace name

Enter the same namespace created in the AWS Fargate profile.

Deploy with Instance Profile

Enable this setting to pass the IAM role for authentication to the cluster. For more information on Instance Profile deployments, see AWS Instance Profile Deployment Guide.

Azure Fields

Field
Description
Automate Infrastructure Deployment

Select Yes to easily perform worker node installations for cluster setup. This requires the Worker Node Size and Storage.

For more information on this specific process, see Azure Automated Deployment Feature Guide.

Select No to populate the fields below and continue with this configuration.

Azure Subscription

Enter your Azure subscription information.

Azure Resource Group

Enter the name of your Azure Resource Group.

Azure Cluster Name

Enter the name of your Azure cluster.

Storage

Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.

OneTrust also offers Standard and External NFS as available options.

If External NFS, you must also provide the following:

  • NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.

  • NFS Path: Enter the NFS path name for the file system to be mounted.

Namespace Name

This field will populate with your entered worker node name by default. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.

Google Fields

Field
Description
GCP Region

Enter the appropriate Google region hosting the resources for your environment.

GKE Cluster Name

Enter the name of your Google Kubernetes Engine cluster.

GCP Zone

Enter the appropriate zone hosting the resources for your environment.

GCP Project ID

Enter the unique ID of the Google Cloud Project.

Storage Class

Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.

OneTrust also offers Standard, Cloud NFS, and External NFS as available options.

If Standard or Cloud NFS, you must also provide the following:

  • GCP Filestore Instance: Enter the name of the filestore instance for the GCP managed NFS service. If not supplied, a new name will be generated automatically.

If External NFS, you must also provide the following:

  • NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.

  • NFS Path: Enter the NFS path name for the file system to be mounted.

Namespace Name

Enter the namespace for the deployment.

Onebox Fields

Field
Description
Namespace Name

Enter the namespace for the deployment.

On-Premises Kubernetes Fields

Field
Description
Storage Class Name

Enter the preferred storage class for volume implementations that supports dynamic PVC provisioning and Read Write Many (RWX) privileges.

Namespace Name

Enter the namespace for the deployment.

OpenShift Fields

Field
Description
OpenShift API Server

Enter the API server URL of your OpenShift cluster.

OpenShift Username

Enter the username of your OpenShift account.

OpenShift Password

Enter the password of your OpenShift account.

Storage Class

Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.

OneTrust currently offers Azure, External NFS, and Custom as available options.

If External NFS, you must also provide the following:

  • NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.

  • NFS Path: Enter the NFS path name for the file system to be mounted.

If Custom, you must also provide the following:

  • Storage Class Name: Enter the storage class name which supports read-write-many.

Storage Class Platform

Populated for Network File System (NFS) Pod, select the platform where OpenShift is deployed for read-write-once.

  • OpenShift on AWS

  • OpenShift on Azure

Namespace Name

Enter the namespace for the deployment.

Oracle Fields

Field
Description
Oracle User

Enter OCID for the user.

Fingerprint

Enter the fingerprint to the corresponding private key.

Key File

Enter the key content in PEM format (not SSH-RSA)

Oracle Tenancy

Enter the OCID for the tenancy.

Oracle Region

Enter the region for the infrastructure.

Oracle Compartment

Enter the OCID for the compartment.

OKE Cluster

Enter the OCID for the cluster.

OKE Cluster Endpoint IP

Enter the IP address to the cluster.

Storage

Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.

OneTrust also offers Standard, Cloud NFS, and External NFS as available options.

If External NFS, you must also provide the following:

  • NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.

  • NFS Path: Enter the NFS path name for the file system to be mounted.

Namespace Name

Enter the namespace for the deployment.

Advanced Options

1. General

In the General section, configure your preferred general settings for this worker node.



Field
Description
Index Data for DSAR Request

When enabled, the index data will be used for serving DSAR requests. The Elasticsearch storage will be increased to 100GB.

Client Docker Registry

Enable to use a client docker registry. Otherwise the default OneTrust Docker Registry will be used to populate the Username and Password fields. To use a client docker registry, input the following fields:

  • Docker Registry

  • Username

  • Password

Note:

Please contact OneTrust for more information on getting the OneTrust Docker Registry details.

Outbound Proxy

Enable to send outbound traffic through a proxy. This is optional support if your infrastructure is configured to have network traffic from the installer virtual machine (from which the deployment script is executed to deploy) routed through a proxy server.

For more information, see Using Proxy Variables During Worker Node Installation.

Enable Telemetry Collection

Enable this optional feature to allow OneTrust to monitor and collect information about the health of the worker node and troubleshoot based on identified errors.

Note:

Please contact OneTrust Support to learn more and get started with this functionality.

Enable Micro K8s Autograde

For Onebox only

Allows automatic upgrade of Micro K8s to the latest version.

Enable K3S

For Onebox only

Installs and configures K3S as the underlying cluster.

Airgapped Worker Node Install

For Onebox only

Appears when the K3S setting is enabled. This allows worker nodes to be downloaded as TAR instead of relying on Docker Hub URLs. These URLs must be on your allowlist so that worker node upgrade jobs work as expected after installation.

2. Classification

  1. Enable the Document Classification setting to create document summaries during classification in order to detect files which may have retention or security policy applicable to it.

    Note:

    For more information, see Understanding Document Classification.



3. Storage

In the Storage section, configure the amount of storage required for the scan job processes used by the worker node. The recommended amounts will be provided by default.



Field
Description
Elasticsearch

Set the desired storage size for Elasticsearch.

Used for DSAR results data, specifically index results.

Kafka

Set the desired storage size for Kafka.

Used to maintain catalog and classification results before sending to OneTrust Cloud.

Postgres

Set the desired storage for Postgress.

Used to track scanner tasks and the status of scan jobs.

Enable Local Cache

Enable this setting to cache catalog information locally on the worker node.

You will need to list the desired storage size, minimum 256 GB.

4. Scaling

In the Scaling section, configure your preferred scaling settings for this worker node.



Field
Description
Enable Auto-Scaling

Enabled by default, this setting allows the worker node to automatically scale down the services when not in use. Whenever a new scan is started, the pods will be scaled up to the previous state.

Idle Time

Enter the time idle duration the system will wait to scale down the services.

Scale Services

This setting allows the worker node to increase the default number of pod replicas for the storage scanner without having the user manually override later in the installer file.

Enter the number of pod replicas per scanner to maximize scanning capacity. Defaults are provided.

  • Storage Scanner

  • Office 365 Scanner

  • App Scanner

  • RDBMS Scanner

  • NoSQL Scanner

  • Apache Tika

  • Apache Tika with Tesseract

  • Google Scanner

  • Document Classification

Kafka

Enable to scale Kafka to 3 brokers.

Important:

Once Kafka scaling is enabled and the worker node is deployed, this setting cannot be disabled.

Dynamic Kafka TTL

Enable to dynamically increase Kafka retention for long-running jobs up to 60 days. This may cause Kafka services to consume additional disk space. Allocating additional space is advised.

Note:

Please be noted that a paused scan job can be resumed only within the 14 day limit and Dynamic Kafka TTL is not going to affect the pause-resume scan job activity.

5. Security

In the Security section, configure your preferences for this worker node's security measures.



Field
Description
Pod Security

Enable secure pod-to-pod communications through mutual TLS authentication (mTLS).

If you have enabled Kafka scaling, a modal will appear to alert that pod security scaling is mutually exclusive with scaling Kafka and enabling this setting will disable Kafka scaling.

Important:

Once the worker node is deployed, this setting cannot be disabled.

Client Certificates

Enter the client certificate in PEM format.

PEM format is Base64 encoded ASCII which includes "----BEGIN CERTIFICATE-----" and "----END CERTIFICATE-----" lines.

6. Custom Options

In the Custom Options section, enable two additional customization options for select worker node properties. Click the Plus icon to add additional property value pairings.



Field
Description
Feature Properties

Select an available feature variable then enter a corresponding value of true or false.

  • Enable API Gateway

  • Enable Data Discovery Bridge

  • Enable Classification Service

  • Enable Kafka Connector

  • Enable Dynamic Topics

  • Kafka Storage Scanner Topic Partition

  • Kafka Scan Job Manager Topic Partition

  • Kafka O365 Scanner Topic Partition

  • Kafka NoSQL Scanner Topic Partition

  • Kafka Google Scanner Topic Partition

  • Kafka RDBMS Scanner Topic Partition

  • Scan Profile V2 Enabled System

Service Properties

Select an available environment variable and service, then enter a corresponding value.

Environment:

  • JAVA_OPTS

Service:

Note:

You can select multiple services at once to align against one value instead of creating property-value pairs for each distinct service.

  • app-scanner

  • google-scanner

  • custom-rdbms-scanner

  • databricks-gateway

  • data-discovery-bridge

  • datadiscovery-onprem-agent

  • dd-index

  • nosql-scanner

  • o365-scanner

  • rdbms-scanner

  • scan-job-manager

  • storage-scanner

  • onprem-gateway

Worker Node PIN

This PIN is used for encryption as part of extra security for your worker node installation.

CAUTION:

Document this PIN in a safe and external location. If you lose or forget, there is no way to retrieve it and the installation will need to start anew.

Note:

OneTrust 202403.1.0 supports storing worker node PINs in the given namespace as a Kubernetes secret to maintain PIN access and easily retrieve the PIN if forgotten or lost. See step 4 below.

OneTrust does not have access to customer worker nodes, including this Kubernetes secret.

  1. In the Set Up a Pin screen, create a pin by following the requirements and recommended prompts.



  2. Click the Save button. If successful, the worker node will be created and the Worker Details screen appears.

  3. To retrieve a worker node pin, log in to the kubernetes cluster:

    CODE
    az account set --subscription <subscription_name>
    az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>
  4. Run the following command to extract the PIN:

    CODE
    kubectl get secret wn-pin-secret -n <namespace> -o jsonpath='{.data.PIN}' | base64 --decode

Next Steps

Now that you've created a worker node, it's ready to use for your worker node installation.

  1. From the Settings tab, click the Launch Install button.



  2. The Launch Install modal appears with a command to copy so you can get started with your worker node installation.



    Note:

    Log in to the local or remote machine that will be used to access the cluster master. This is the location where the installer scrip will need to be extracted.

  3. Run the curl command in the command prompt.

  4. Run the deployment script and follow the on-screen instructions.

    CODE
    /installer.sh

Updating Worker Node Versions

For the manual overview, you will need to use the installer script command found above in Settings screen. For more information on the available update methods, please see Updating Worker Nodes.

Adding Proxy Support

A proxy server can be configured to ensure communications with OneTrust are functional and secured. The input requires the Proxy Server Address to be updated for the following variable in the .envs file:

CODE
#Optional proxy server, used when connecting to OneTrust
export HTTP_PROXY=""

For additional information on configuring proxy variables during installation, see Using Proxy Variables During Worker Node Installation.

Updating Variables In Configuration File

In the event that any variable is not currently configurable via the OneTrust UI, you can still manage the given variable directly through the .guided-envs file. This also helps avoid potential configuration mismatches during deployment.

  1. Navigate to the directory: datadis.../k8s.../inventory

  2. Open the .guided-envs file.

  3. Locate the required variable that needs to be updated.

  4. Use the relevant keywords to search for the variable in the .guided-envs file. For example, if you're updating the region configuration, search for "region" to quickly locate the related variable.

  5. Modify the variable as necessary. For example:

    • Replace: export AWS_VAULT_REGION="<incorrect_value>"

    • With: export AWS_VAULT_REGION="<correct_value>"

  6. To redeploy the configuration, move up one directory using command: cd...

  7. Execute the deployment script with: ./deploy.sh install