Lightweight Worker Node Troubleshooting and Administration

Required outbound URLs

The lightweight worker node VM must reach the following endpoints over port 443 from within your VPC or VNet:

Tenant URL examples include app.onetrust.com, app-br.onetrust.com, app-uk.onetrust.com, uat.onetrust.com, trial.onetrust.com, and custom OneTrust domains.

Container registry access (optional)

The lightweight worker node uses airgapped images by default. External registry access is not strictly required.

If available, safelisting docker.io/otcr can speed up installation and upgrades.

What happens when URLs are not safelisted

VM sizing and operating system

Lightweight worker node VMs are preconfigured with defaults in the cloud deployment templates. The minimum requirements are:

• 16 vCPU

• 64 GB RAM

• 512 GB disk

For provider-specific VM-size and operating-system options, see Lightweight Worker Node Deployment for AI Automations.

SSH access

SSH access is required for troubleshooting, manual debug collection, and advanced administration.

Example SSH command:

ssh -i ~/.ssh/your-key.pem onetrust@<vm-public-ip>

ssh -i ~/.ssh/your-key.pem onetrust@<vm-public-ip>

Proxy configuration

If outbound internet access from the lightweight worker node VM uses a corporate HTTP or HTTPS proxy, configure the proxy URL during cloud deployment.

Each cloud template has a proxy field in the deployment form. The proxy is applied to HTTPS traffic to the tenant URL and WSS traffic to ot01.nats.onetrust.com.

Format:

http://<proxy-ip>:<port>

Example:

http://172.18.0.12:3128

The installer sets HTTP_PROXY, HTTPS_PROXY, and NO_PROXY on the VM and within K3s.

http://<proxy-ip>:<port>

http://172.18.0.12:3128

AI Guard

AI Guard is optional and provides real-time content filtering for AI workloads. It requires inbound port 4443 on the lightweight worker node VM.

• Azure: Enabling AI Guard in the ARM form adds an inbound NSG rule for port 4443.

• AWS: Enabling AI Guard in the CloudFormation form adds an inbound security-group rule for port 4443 and sets the worker-node profile to aigov.

• GCP: The GCP Terraform template does not currently add a firewall rule for port 4443. If AI Guard is used on GCP, create the firewall rule manually.

If AI Guard is not needed, leave it disabled. It is off by default, and no inbound ports are open.

For more information, see AI Guard on the OneTrust Developer Portal. You can also visit AI Guard SDK on GitHub.

OneTrust Token and Worker Node PIN

When creating a lightweight worker node in the OneTrust platform, two values are involved:

OneTrust Token

- an encoded configuration blob that contains lightweight worker node deployment configuration. It is automatically populated in the cloud deployment form whenActivateis selected from theAI Automationsscreen. Do not edit it.

Worker Node PIN

- a fixed value written automatically during deployment. It does not need to be entered manually.

Awaiting activation after failed cloud deployment

If Activate is selected and cloud deployment fails because of insufficient permissions, quota, or network configuration:

• The worker node remains in Awaiting activation.

• The VM, K3s cluster, and pods may not exist yet.

• Select Activate to return to the cloud console and retry.

• Select Delete to remove the OneTrust worker node resource.

Launch install

The Launch install button on the Worker details screen performs the same action as Activate on the AI Automations and redirects to the originating cloud platform:

• Azure-created lightweight worker node: Launch in Azure.

• AWS-created lightweight worker node: Launch in AWS.

• GCP-created lightweight worker node: Launch in GCP.

Installation phases

1. Cloud infrastructure.

   The ARM, CloudFormation, or Terraform template creates the VPC/VNet, subnet, firewall rules, NAT gateway, where applicable, VM, disk, and startup script.

2. K3s cluster setup.

   The startup script downloads and installs K3s, configures networking, waits for the node to become ready, and verifies the Kubernetes system pods.

3. Container image import.

   Container images are downloaded as a compressed archive and imported into the K3s containerd image store.

4. Helm deployment.

   Lightweight worker node creates the onetrust namespace, writes required secrets, deploys the lightweight worker node Helm chart, runs health checks, and starts GoAgent.

Common failure points

Worker node is unhealthy

Check the following in order:

1. Can the VM reach the OneTrust tenant URL?

2. Can the VM reach ot01.nats.onetrust.com:443 over WSS?

3. Is the Kubernetes cluster running?

4. Are the core pods running?

Key pods include:

• datadiscovery-onprem-agent

• dd-workernode-ops

• scan-job-manager

If only these three pods are in CrashLoopBackOff, check NATS connectivity. Other pods may remain running while these services fail.

Pods in CrashLoopBackOff

Installation completed but services are not starting

If the cloud deployment succeeded but the worker node is unhealthy:

sudo systemctl status k3s
    kubectl get pods -n onetrust
    kubectl get jobs -n onetrust

Check the update-scanner-to-version job. If it failed, the likely cause is a K3s-level network issue.

K3s recovery

If K3s is not running or the node is not ready:

sudo systemctl restart k3s
    sleep 60
    sudo systemctl status k3s
    kubectl get nodes

If K3s fails to start, review disk space and the K3s journal:

df -h
    sudo journalctl -u k3s -xe --no-pager | tail -50

K3s is installed as a systemd service and will auto-restart on VM reboot. If the service is consistently crashing, check for disk space issues (df -h) or corrupted data in /var/lib/rancher/k3s/.

Export details

The Export details option is available from Worker details > Summary.

It triggers automated debug collection through NATS and dd-workernode-ops.

Prerequisites:

• K3s is functional.

• dd-workernode-ops is running.

• NATS connectivity is working.

If Export details fails, use manual debug collection.

Services tab

The Services tab shows pods in the onetrust namespace and provides streaming logs.

The Services tab requires:

• K3s to be running.

• dd-workernode-ops to be running.

• Loki to be running.

• The log-collection components to be healthy.

k9s

Lightweight worker nodes include k9s:

k9s -n onetrust

Use k9s to view pod status, restarts, resource usage, logs, and pod details.

Manual debug collection

SSH to the lightweight worker node VM and navigate to the deployment directory:

cd ~/onetrust-dd-installer-*/data-discovery/litenode-deployment/
    ./debug.sh run

Options include:

./debug.sh run upload
    ./debug.sh run skip-kafka
    ./debug.sh run include-pg-dump
    ./debug.sh run jobId=<uuid>

Diagnostic scripts

Service restart and configuration scripts

Destructive scripts

Custom certificates

If a proxy or firewall performs TLS inspection, the lightweight worker node needs the custom CA certificate to establish trusted connections.

1. Open Configurations > Security and enable the Client certificates setting.

2. Paste the full PEM certificate chain, including the BEGIN CERTIFICATE and END CERTIFICATE lines.

3. Click the Save & Update button.

GoAgent files and locations

Verify GoAgent

sudo systemctl status goagent
    cat /var/log/goagent/goagent.log
    ps aux | grep go-agent

Troubleshoot failed upgrades

Manual upgrade fallback

If GoAgent cannot perform the upgrade automatically, trigger the upgrade manually from the lightweight worker node VM.

1. Download the latest installer from the tenant URL:

   CODECopy

   cd ~ curl -o installer.sh "https://<tenant>.onetrust.com/api/discovery-scan-config/v1/installer-config/<WORKER_NODE_ID>/download"

   cd ~
   curl -o installer.sh "https://<tenant>.onetrust.com/api/discovery-scan-config/v1/installer-config/<WORKER_NODE_ID>/download"

2. Make the installer executable and extract the package:

   CODECopy

   chmod +x installer.sh ./installer.sh extract-zip

   chmod +x installer.sh
   ./installer.sh extract-zip

3. Navigate to the extracted lightweight worker node deployment directory:

   CODECopy

   cd ~/onetrust-dd-installer-litenode-onetrust-<new-version>/data-discovery/litenode-deployment/

   cd ~/onetrust-dd-installer-litenode-onetrust-<new-version>/data-discovery/litenode-deployment/

4. Run the installation script:

   CODECopy

   ./deploy.sh install

   ./deploy.sh install

Key paths

Common VM commands

# Check k3s status
sudo systemctl status k3s

# View all pods
kubectl get pods -n onetrust

# View pod logs
kubectl logs deployment/datadiscovery-onprem-agent -n onetrust --tail=100

# Check LiteNode version
cat ~/onetrust-dd-installer-*/data-discovery/litenode-deployment/worker_node_version.sh

# Check disk usage
df -h

# Check GoAgent
sudo systemctl status goagent

# Launch k9s
k9s -n onetrust

# Run debug collection
cd ~/onetrust-dd-installer-*/data-discovery/litenode-deployment/
./debug.sh run# Check k3s status
sudo systemctl status k3s

# View all pods
kubectl get pods -n onetrust

# View pod logs
kubectl logs deployment/datadiscovery-onprem-agent -n onetrust --tail=100

# Check LiteNode version
cat ~/onetrust-dd-installer-*/data-discovery/litenode-deployment/worker_node_version.sh

# Check disk usage
df -h

# Check GoAgent
sudo systemctl status goagent

# Launch k9s
k9s -n onetrust

# Run debug collection
cd ~/onetrust-dd-installer-*/data-discovery/litenode-deployment/
./debug.sh run

Disk usage and management

The 512 GB disk hosts K3s data, container images, and persistent volumes for Kafka, Postgres, and Loki.

The following components can consume disk space over time:

• Loki logs, especially when scanner debug logging is enabled.

• Kafka data from long-running scans with many tasks.

• Postgres data containing scan job metadata and task records.

• Container images retained in the K3s image store after upgrades.

To check disk usage on the VM, run:

df -h

To check persistent-volume usage, run:

kubectl exec -n onetrust kafka-0 -- df -h /bitnami/kafka
kubectl exec -n onetrust postgres-0 -- df -h /var/lib/postgresql
kubectl exec -n onetrust loki-0 -- df -h /data

If disk usage is high, remove obsolete installer directories and prune unused K3s images:

rm -rf ~/onetrust-dd-installer-litenode-onetrust-<old-version>/
sudo k3s crictl rmi --prune

If Loki is responsible for high disk usage, restart the Loki StatefulSet using restart_sts with STATEFULSET_NAME=loki.

Backup and Recovery Considerations

A lightweight worker node can be recreated if its VM is lost. However, local K3s state, Kafka topics, Postgres scan history, Loki logs, and Internal Vault credentials may not be recoverable. If HashiCorp Vault (Internal) is used, back up the datasource credentials before replacing the worker node or performing recovery actions. Recovery actions may result in data loss.