Guided Worker Node Installation Overview
You can opt to follow a guided installation process when creating your on-premises worker node for OneTrust Data Discovery. This screen-by-screen walkthrough will queue several important details about your desired data source, worker node deployment options, the available discovery scan and catalog options, and many more advanced settings for your best use cases. The end product will be an available worker node that can be used immediately without much further configuration as the script will run and ask you to follow the on-screen instructions.
For information on how to manually install the worker node, see Manual Worker Node Installation Overview.
Prerequisites
You will need at least the following to support worker node creation with OneTrust Data Discovery. The platform-specific installation articles in Data Discovery Knowledge Base will provide additional detail around what support is needed for the given deployment.
-
Environment with nodes to support at least 8 vCPU(s), 32 GB RAM
-
A subscription which can run the required cluster
-
An owner/contributor user of the given subscription which will be used to access, create, and manage the cluster
-
A local (laptop) or remote (Jump box/bastion) machine which will be used to access the cluster master. Minimum: 4 CPU, 8GB RAM, 250 GB HDD, Ubuntu 16+
Note:Not applicable for Onebox Microkubernetes Service deployments.
-
Virtual Network and subnet in which the given cluster is getting deployed should have access to data sources
-
Virtual Network and subnet in which the given cluster is getting deployed should have access to Internet to reach Docker Hub
-
Docker >=19.x ce + pre-installed (OneTrust can also install this)
-
Latest version of kubectl to ensure full compatibility of services
Create a Worker Node
-
On the Data Discovery menu, select . The Worker Nodes screen appears.
-
Click the Add Worker Node button. The Create Worker Node modal appears.
-
Enter a unique name for this new worker node. The name you enter here will be populated as the namespace in the follow screen. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.
-
Select the Guided Install option.
-
The Managing Organization field will default to the parent org level.
-
The Worker Node Owner field will default to the current user.
-
The Label field can be used to provide a description of this node uses or purpose(s).
Optional: With the guided installation option, you can also choose to create the new node by copying an existing guided worker node template configuration.
Note:For more information on worker node owners and email alerts, see Managing Email Notifications for Worker Nodes.
-
-
Click the Next button. The Platform Provider and Credential Store screen appears. Proceed to Configuring the worker node.
If using the worker node template option, the Enter PIN modal appears. See step #7 for more.
-
In order to use an existing worker node configuration, please enter the PIN created as part of the templated worker node creation. For more information on how to create and obtain a PIN, see Worker Node PIN.
Note:OneTrust 202403.1.0 supports storing worker node PINs in the given namespace as a Kubernetes secret to maintain PIN access and easily retrieve the PIN if forgotten or lost.
-
Click the Next button. The Platform Provider and Credential Store screen appears.
Configuring the worker node
By selecting the Guided Install option, you can then proceed with configuring the worker node by following the wizard steps. In this example, we're creating a worker node to use with the Azure platform. The majority of these fields will be generally available to any platform provider.
See Data Discovery Deployment Options for more information on available systems.
-
In the Platform Provider and Credential Store screen, select your platform provider. The interface will auto-populate fields applicable to the chosen system. Here, we have selected AWS.
-
In the Platform Provider block, select your desired provider from the field and then complete the fields for given configuration.
See the provider information tables below for more information.
-
In the Credential Store block, select your desired credential store from the field and then complete the fields with your applicable vault details.
Note:For more information on support and configuration of credential vaults, see Credential Vault Options for Data Sources.
- Field
- Description
- Credential Store
-
Select from the available Credential Store choices. The selection will populate unique fields for each store option.
- Deploy with Native Authentication (specific to AWS Secrets Manager)
-
Enable this setting to delegate permissions using IAM role(s) to make certain API calls for fetching credentials.
- Store Options
-
AWS Secrets Manager
- Azure Key Vault
- CyberArk - Conjur
- Google Secrets Manager
- HashiCorp Vault - Open Source (Internal)
- HashiCorp Vault - Open Source (External)
- HashiCorp Vault - Enterprise
Note:See Data Discovery Deployment Options for additional information on current deployment options for OneTrust Data Discovery.
-
Click the Next button. The Advanced Options screen appears.
AWS Fields
- Field
- Description
- Automate Infrastructure Deployment
-
Select Yes to easily perform worker node installations for cluster setup. This requires the Worker Node Size and Storage.
Select No to populate the fields below and continue with this configuration.
- AWS Region
-
Enter the appropriate AWS region for your environment.
- EKS Cluster Name
-
Enter the cluster name of your Elastic Kubernetes Service.
- Storage
-
Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.
If External NFS, you must also provide the following:
-
NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.
-
NFS Path: Enter the NFS path name for the file system to be mounted.
If Elastic File System (EFS), you must also provide the following:
-
Elastic File System ID: Enter the ID of the Amazon Elastic File System.
-
Elastic File System Region: Enter the region associated with the Amazon Elastic File System.
-
- Namespace name
-
This field will populate with your entered worker node name by default. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.
- Deploy with Instance Profile
-
Enable this setting to pass the IAM role for authentication to the cluster. For more information on Instance Profile deployments, see AWS Instance Profile Deployment Guide.
AWS Fargate Fields
- Field
- Description
- Automate Infrastructure Deployment
-
Select Yes to easily perform worker node installations for cluster setup. This requires the AWS Account ID and AWS Region.
Select No to populate the fields below and continue with this configuration.
- AWS Region
-
Enter the appropriate AWS region for your environment.
- EKS Cluster Name
-
Enter the cluster name of your Elastic Kubernetes Service.
- Elastic File System ID
-
Enter your Elastic File System ID.
- Elastic File System Region
-
Enter your Elastic File System Region.
- Namespace name
-
Enter the same namespace created in the AWS Fargate profile.
- Deploy with Instance Profile
-
Enable this setting to pass the IAM role for authentication to the cluster. For more information on Instance Profile deployments, see AWS Instance Profile Deployment Guide.
Azure Fields
- Field
- Description
- Automate Infrastructure Deployment
-
Select Yes to easily perform worker node installations for cluster setup. This requires the Worker Node Size and Storage.
For more information on this specific process, see Azure Automated Deployment Feature Guide.
Select No to populate the fields below and continue with this configuration.
- Azure Subscription
-
Enter your Azure subscription information.
- Azure Resource Group
-
Enter the name of your Azure Resource Group.
- Azure Cluster Name
-
Enter the name of your Azure cluster.
- Storage
-
Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.
OneTrust also offers Standard and External NFS as available options.
If External NFS, you must also provide the following:
-
NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.
-
NFS Path: Enter the NFS path name for the file system to be mounted.
-
- Namespace Name
-
This field will populate with your entered worker node name by default. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.
Google Fields
- Field
- Description
- GCP Region
-
Enter the appropriate Google region hosting the resources for your environment.
- GKE Cluster Name
-
Enter the name of your Google Kubernetes Engine cluster.
- GCP Zone
-
Enter the appropriate zone hosting the resources for your environment.
- GCP Project ID
-
Enter the unique ID of the Google Cloud Project.
- Storage Class
-
Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.
OneTrust also offers Standard, Cloud NFS, and External NFS as available options.
If Standard or Cloud NFS, you must also provide the following:
-
GCP Filestore Instance: Enter the name of the filestore instance for the GCP managed NFS service. If not supplied, a new name will be generated automatically.
If External NFS, you must also provide the following:
-
NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.
-
NFS Path: Enter the NFS path name for the file system to be mounted.
-
- Namespace Name
-
Enter the namespace for the deployment.
Onebox Fields
- Field
- Description
- Namespace Name
-
Enter the namespace for the deployment.
On-Premises Kubernetes Fields
- Field
- Description
- Storage Class Name
-
Enter the preferred storage class for volume implementations that supports dynamic PVC provisioning and Read Write Many (RWX) privileges.
- Namespace Name
-
Enter the namespace for the deployment.
OpenShift Fields
- Field
- Description
- OpenShift API Server
-
Enter the API server URL of your OpenShift cluster.
- OpenShift Username
-
Enter the username of your OpenShift account.
- OpenShift Password
-
Enter the password of your OpenShift account.
- Storage Class
-
Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.
OneTrust currently offers Azure, External NFS, and Custom as available options.
If External NFS, you must also provide the following:
-
NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.
-
NFS Path: Enter the NFS path name for the file system to be mounted.
If Custom, you must also provide the following:
-
Storage Class Name: Enter the storage class name which supports read-write-many.
-
- Storage Class Platform
-
Populated for Network File System (NFS) Pod, select the platform where OpenShift is deployed for read-write-once.
-
OpenShift on AWS
-
OpenShift on Azure
-
- Namespace Name
-
Enter the namespace for the deployment.
Oracle Fields
- Field
- Description
- Oracle User
-
Enter OCID for the user.
- Fingerprint
-
Enter the fingerprint to the corresponding private key.
- Key File
-
Enter the key content in PEM format (not SSH-RSA)
- Oracle Tenancy
-
Enter the OCID for the tenancy.
- Oracle Region
-
Enter the region for the infrastructure.
- Oracle Compartment
-
Enter the OCID for the compartment.
- OKE Cluster
-
Enter the OCID for the cluster.
- OKE Cluster Endpoint IP
-
Enter the IP address to the cluster.
- Storage
-
Select the preferred storage class for volume implementations. Network File System (NFS) Pod will be selected by default.
OneTrust also offers Standard, Cloud NFS, and External NFS as available options.
If External NFS, you must also provide the following:
-
NFS Server IP: Enter the IP address or domain name of an NFS server reachable from the cluster that will host the shared libraries.
-
NFS Path: Enter the NFS path name for the file system to be mounted.
-
- Namespace Name
-
Enter the namespace for the deployment.
Advanced Options
1. General
In the General section, configure your preferred general settings for this worker node.
- Field
- Description
- Index Data for DSAR Request
-
When enabled, the index data will be used for serving DSAR requests. The Elasticsearch storage will be increased to 100GB.
- Client Docker Registry
-
Enable to use a client docker registry. Otherwise the default OneTrust Docker Registry will be used to populate the Username and Password fields. To use a client docker registry, input the following fields:
-
Docker Registry
-
Username
-
Password
Note:Please contact OneTrust for more information on getting the OneTrust Docker Registry details.
-
- Outbound Proxy
-
Enable to send outbound traffic through a proxy. This is optional support if your infrastructure is configured to have network traffic from the installer virtual machine (from which the deployment script is executed to deploy) routed through a proxy server.
For more information, see Using Proxy Variables During Worker Node Installation.
- Enable Telemetry Collection
-
Enable this optional feature to allow OneTrust to monitor and collect information about the health of the worker node and troubleshoot based on identified errors.
Note:Please contact OneTrust Support to learn more and get started with this functionality.
- Enable Micro K8s Autograde
-
For Onebox only
Allows automatic upgrade of Micro K8s to the latest version.
- Enable K3S
-
For Onebox only
Installs and configures K3S as the underlying cluster.
- Airgapped Worker Node Install
-
For Onebox only
Appears when the K3S setting is enabled. This allows worker nodes to be downloaded as TAR instead of relying on Docker Hub URLs. These URLs must be on your allowlist so that worker node upgrade jobs work as expected after installation.
2. Classification
-
Enable the Document Classification setting to create document summaries during classification in order to detect files which may have retention or security policy applicable to it.
Note:For more information, see Understanding Document Classification.
3. Storage
In the Storage section, configure the amount of storage required for the scan job processes used by the worker node. The recommended amounts will be provided by default.
- Field
- Description
- Elasticsearch
-
Set the desired storage size for Elasticsearch.
Used for DSAR results data, specifically index results.
- Kafka
-
Set the desired storage size for Kafka.
Used to maintain catalog and classification results before sending to OneTrust Cloud.
- Postgres
-
Set the desired storage for Postgress.
Used to track scanner tasks and the status of scan jobs.
- Enable Local Cache
-
Enable this setting to cache catalog information locally on the worker node.
You will need to list the desired storage size, minimum 256 GB.
4. Scaling
In the Scaling section, configure your preferred scaling settings for this worker node.
- Field
- Description
- Enable Auto-Scaling
-
Enabled by default, this setting allows the worker node to automatically scale down the services when not in use. Whenever a new scan is started, the pods will be scaled up to the previous state.
- Idle Time
-
Enter the time idle duration the system will wait to scale down the services.
- Scale Services
-
This setting allows the worker node to increase the default number of pod replicas for the storage scanner without having the user manually override later in the installer file.
Enter the number of pod replicas per scanner to maximize scanning capacity. Defaults are provided.
-
Storage Scanner
-
Office 365 Scanner
-
App Scanner
-
RDBMS Scanner
-
NoSQL Scanner
-
Apache Tika
-
Apache Tika with Tesseract
-
Google Scanner
-
Document Classification
-
- Kafka
-
Enable to scale Kafka to 3 brokers.
Important:Once Kafka scaling is enabled and the worker node is deployed, this setting cannot be disabled.
- Dynamic Kafka TTL
-
Enable to dynamically increase Kafka retention for long-running jobs up to 60 days. This may cause Kafka services to consume additional disk space. Allocating additional space is advised.
Note:Please be noted that a paused scan job can be resumed only within the 14 day limit and Dynamic Kafka TTL is not going to affect the pause-resume scan job activity.
5. Security
In the Security section, configure your preferences for this worker node's security measures.
- Field
- Description
- Pod Security
-
Enable secure pod-to-pod communications through mutual TLS authentication (mTLS).
If you have enabled Kafka scaling, a modal will appear to alert that pod security scaling is mutually exclusive with scaling Kafka and enabling this setting will disable Kafka scaling.
Important:Once the worker node is deployed, this setting cannot be disabled.
- Client Certificates
-
Enter the client certificate in PEM format.
PEM format is Base64 encoded ASCII which includes "----BEGIN CERTIFICATE-----" and "----END CERTIFICATE-----" lines.
6. Custom Options
In the Custom Options section, enable two additional customization options for select worker node properties. Click the Plus icon to add additional property value pairings.
- Field
- Description
- Feature Properties
-
Select an available feature variable then enter a corresponding value of true or false.
-
Enable API Gateway
-
Enable Data Discovery Bridge
-
Enable Classification Service
-
Enable Kafka Connector
-
Enable Dynamic Topics
-
Kafka Storage Scanner Topic Partition
-
Kafka Scan Job Manager Topic Partition
-
Kafka O365 Scanner Topic Partition
-
Kafka NoSQL Scanner Topic Partition
-
Kafka Google Scanner Topic Partition
-
Kafka RDBMS Scanner Topic Partition
-
Scan Profile V2 Enabled System
-
- Service Properties
-
Select an available environment variable and service, then enter a corresponding value.
Environment:
-
JAVA_OPTS
Service:
Note:You can select multiple services at once to align against one value instead of creating property-value pairs for each distinct service.
-
app-scanner
-
google-scanner
-
custom-rdbms-scanner
-
databricks-gateway
-
data-discovery-bridge
-
datadiscovery-onprem-agent
-
dd-index
-
nosql-scanner
-
o365-scanner
-
rdbms-scanner
-
scan-job-manager
-
storage-scanner
-
onprem-gateway
-
Worker Node PIN
This PIN is used for encryption as part of extra security for your worker node installation.
Document this PIN in a safe and external location. If you lose or forget, there is no way to retrieve it and the installation will need to start anew.
OneTrust 202403.1.0 supports storing worker node PINs in the given namespace as a Kubernetes secret to maintain PIN access and easily retrieve the PIN if forgotten or lost. See step 4 below.
OneTrust does not have access to customer worker nodes, including this Kubernetes secret.
-
In the Set Up a Pin screen, create a pin by following the requirements and recommended prompts.
-
Click the Save button. If successful, the worker node will be created and the Worker Details screen appears.
-
To retrieve a worker node pin, log in to the kubernetes cluster:
CODEaz account set --subscription <subscription_name> az aks get-credentials --resource-group <resource_group_name> --name <cluster_name> -
Run the following command to extract the PIN:
CODEkubectl get secret wn-pin-secret -n <namespace> -o jsonpath='{.data.PIN}' | base64 --decode
Next Steps
Now that you've created a worker node, it's ready to use for your worker node installation.
-
From the Settings tab, click the Launch Install button.
-
The Launch Install modal appears with a command to copy so you can get started with your worker node installation.
Note:Log in to the local or remote machine that will be used to access the cluster master. This is the location where the installer scrip will need to be extracted.
-
Run the curl command in the command prompt.
-
Run the deployment script and follow the on-screen instructions.
CODE/installer.sh
Updating Worker Node Versions
For the manual overview, you will need to use the installer script command found above in Settings screen. For more information on the available update methods, please see Updating Worker Nodes.
Adding Proxy Support
A proxy server can be configured to ensure communications with OneTrust are functional and secured. The input requires the Proxy Server Address to be updated for the following variable in the .envs file:
#Optional proxy server, used when connecting to OneTrust
export HTTP_PROXY=""
For additional information on configuring proxy variables during installation, see Using Proxy Variables During Worker Node Installation.
Updating Variables In Configuration File
In the event that any variable is not currently configurable via the OneTrust UI, you can still manage the given variable directly through the .guided-envs file. This also helps avoid potential configuration mismatches during deployment.
-
Navigate to the directory: datadis.../k8s.../inventory
-
Open the .guided-envs file.
-
Locate the required variable that needs to be updated.
-
Use the relevant keywords to search for the variable in the .guided-envs file. For example, if you're updating the region configuration, search for "region" to quickly locate the related variable.
-
Modify the variable as necessary. For example:
-
Replace:
export AWS_VAULT_REGION="<incorrect_value>" -
With:
export AWS_VAULT_REGION="<correct_value>"
-
-
To redeploy the configuration, move up one directory using command:
cd... -
Execute the deployment script with:
./deploy.sh install