Scaling Data Discovery Scanners for Increased Throughput

OneTrust worker nodes have several components that help extract data from a variety of data sources and then detect and classify personal information. To efficiently process this data, worker nodes can leverage different techniques to increase the overall throughout of the system.

OneTrust recommends configuring two environment variables on the scan-job-manager services. These commands will restart the scan-job-manager service.

  • MAX_PARALLEL_JOBS_BY_JOB_CLASS: controls the maximum number of active jobs allowed at one time. This does not include the DLQ topics that are created for each corresponding task topic.

  • MAX_DYNAMIC_TOPICS_BY_JOB_CLASS: controls the maximum number of topics, which should at minimum always be set to twice that of the maximum number of active jobs. This accounts for the latency when topics are deleted as part of job cancellation or completion.

OneTrust Application

To accomplish scaling in the application, navigate to the Configurations tab in the Worker Node Details screen. Locate the Custom Options section and provide the following as a Service Property value:

CODE 
-D<property-name>=<value>

Example: -Dlogging.level.root=DEBUG

Example Commands

This command example is increasing the throughput of scanning a maximum of 10 parallel scans for a given scanner.

CODE 
kubectl config set-context --current --namespace=<namespacename>

kubectl set env deployment/scan-job-manager MAX_PARALLEL_JOBS_BY_JOB_CLASS=10
kubectl set env deployment/scan-job-manager MAX_DYNAMIC_TOPICS_BY_JOB_CLASS=20

You can validate this action by running:

CODE 
kubectl get pods