Lightweight Worker Node AWS Deployment Troubleshooting

CloudFormation execution role

The IAM role that CloudFormation assumes must include the permissions in the policy below. The AWS console user must additionally have permissions to execute CloudFormation stacks and inspect provisioned resources.

Required IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CFNStackManagement",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:UpdateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStacks",
                "cloudformation:DescribeStackEvents",
                "cloudformation:ListStackResources",
                "cloudformation:GetTemplate",
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VPCNetworking",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateVpc", "ec2:DeleteVpc", "ec2:DescribeVpcs",
                "ec2:CreateInternetGateway", "ec2:DeleteInternetGateway",
                "ec2:DescribeInternetGateways", "ec2:AttachInternetGateway",
                "ec2:DetachInternetGateway",
                "ec2:CreateSubnet", "ec2:DeleteSubnet", "ec2:DescribeSubnets",
                "ec2:AllocateAddress", "ec2:ReleaseAddress", "ec2:DescribeAddresses",
                "ec2:CreateNatGateway", "ec2:DeleteNatGateway", "ec2:DescribeNatGateways",
                "ec2:CreateRouteTable", "ec2:DeleteRouteTable", "ec2:DescribeRouteTables",
                "ec2:CreateRoute", "ec2:DeleteRoute",
                "ec2:AssociateRouteTable", "ec2:DisassociateRouteTable"
            ],
            "Resource": "*"
        },
        {
            "Sid": "SecurityGroupsAndKeyPairs",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup", "ec2:DeleteSecurityGroup",
                "ec2:DescribeSecurityGroups",
                "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress", "ec2:RevokeSecurityGroupEgress",
                "ec2:CreateKeyPair", "ec2:DeleteKeyPair", "ec2:DescribeKeyPairs"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EC2InstancesAndTagging",
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances", "ec2:TerminateInstances",
                "ec2:DescribeInstances", "ec2:DescribeImages",
                "ec2:DescribeAvailabilityZones",
                "ec2:CreateTags", "ec2:DeleteTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "SSMParameterRead",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter", "ssm:GetParameters",
                "ssm:GetParametersByPath", "ssm:DescribeParameters",
                "ssm:PutParameter", "ssm:DeleteParameter"
            ],
            "Resource": "*"
        },
        {
            "Sid": "KMSDecryptSSM",
            "Effect": "Allow",
            "Action": ["kms:Decrypt"],
            "Resource": "*"
        }
    ]
}

Why these permissions matter

Service quotas to check

Deployment scenarios

Stack-level failures

CREATE_FAILED - Insufficient IAM permissions

Console error:

Resource handler returned message: "User: arn:aws:sts::xxx:assumed-role/xxx/xxx
is not authorized to perform: ec2:CreateVpc"

Where to check: AWS Console > CloudFormation > Stack > Events tab.

Root cause: The CloudFormation execution role or console user lacks required IAM permissions.

Fix: AWS Console > IAM > Roles > find the execution role. Attach the required policy (see Prerequisites and IAM permissions). Retry deployment from the OneTrust platform (click Activate again).

CREATE_FAILED - VPC quota exceeded

Console error:

"The maximum number of VPCs has been reached."
Code: VpcLimitExceeded

Where to check: AWS Console > VPC > Your VPCs (count existing VPCs in the region).

Root cause: Default VPC limit is 5 per region. Too many VPCs already exist.

Fix: Delete unused VPCs in the region, OR Request quota increase: AWS Console > Service Quotas > VPC > "VPCs per region" > Request increase. Alternatively, use an existing VPC by providing ExistingVPC parameter.

CREATE_FAILED - Elastic IP limit exceeded

Console error:

"The maximum number of addresses has been reached."
Code: AddressLimitExceeded

Where to check: AWS Console > VPC > Elastic IPs (count EIPs in the region).

Root cause: Default EIP limit is 5 per region. NAT gateway requires an EIP.

Fix: Release unused Elastic IPs, OR Request quota increase: AWS Console > Service Quotas > EC2 > "Elastic IPs for each region". Alternatively, use an existing VPC + subnet (no NAT created, no EIP needed).

CREATE_FAILED - NAT gateway quota exceeded

Console error:

"Resources are not available for the NatGateway in the requested Availability Zone"

Where to check: AWS Console > VPC > NAT Gateways.

Root cause: NAT gateway limit reached or requested AZ has no NAT gateway capacity.

Fix: Delete unused NAT gateways, OR Request quota increase, OR Use existing VPC + subnet (skips NAT creation entirely).

EC2 instance failures

EC2 instance - key pair not found

Console error:

"The key pair 'xxx' does not exist."
Code: InvalidKeyPair.NotFound

Where to check: AWS Console > EC2 > Key Pairs (verify the key exists in the correct region).

Root cause: ExistingKeyName parameter references a key pair that doesn't exist in the deployment region.

Fix: Verify the key pair name in AWS Console > EC2 > Key Pairs. Ensure the key is in the same region as the deployment. If the key doesn't exist, leave ExistingKeyName blank to auto-create one.

EC2 instance - instance type not available in AZ

Console error:

"Your requested instance type (m5.4xlarge) is not supported in your requested
Availability Zone (us-east-1e)"

Where to check: AWS Console > EC2 > Instance Type > check availability per AZ.

Root cause: Not all instance types are available in all AZs. The template uses !Select [ 0, !GetAZs "" ] which picks the first available AZ.

Fix: Try a different region where m5.4xlarge is available, OR Use an existing VPC + subnet in an AZ that supports the instance type.

EC2 instance - AMI not found (RHEL9)

Console error:

"ImageId" refers to an AMI that does not exist in this region

Where to check: AWS Console > EC2 > AMIs > search for the AMI ID from RHEL9RegionMap in the template.

Root cause: The RHEL9RegionMap in the CloudFormation template has hardcoded RHEL9 AMI IDs. These AMIs are periodically deprecated by AWS and may no longer exist.

Fix: Use UBUNTU2404 or UBUNTU2204 instead (these use SSM parameter lookups, not hardcoded AMIs). If RHEL9 is required, contact OneTrust support for an updated template with current AMI IDs.

EC2 instance - SSM parameter resolution failed (Ubuntu)

Console error:

"{{resolve:ssm:/aws/service/canonical/ubuntu/pro-server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id}}
is not a valid AMI ID"

Where to check: AWS Console > Systems Manager > Parameter Store > search for the parameter path.

Root cause: IAM role lacks ssm:GetParameter permission KMS key used for the SSM parameter is not accessible (kms:Decrypt missing). The SSM parameter path has changed (Canonical updated their publishing).

Fix: Verify ssm:GetParameter and kms:Decrypt are in the IAM policy. Verify the SSM parameter exists: aws ssm get-parameter --name "/aws/service/canonical/ubuntu/pro-server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id" --region <region>

If parameter doesn't exist, try UBUNTU2204 or contact OneTrust support.

Instance profile / IAM failures

IAM role creation failed - already exists

Console error:

"Role with name ot-xxxx-ec2-role already exists."
Code: EntityAlreadyExists

Where to check: AWS Console > IAM > Roles > search for the stack name prefix.

Root cause: A previous deployment (or failed stack) left an orphaned IAM role with the same name. CloudFormation generates role names from the stack name.

Fix: Delete the orphaned IAM role: AWS Console > IAM > Roles > find ot-<stack-name>-ec2-role > Delete. Delete the orphaned instance profile: AWS Console > IAM > Roles > Instance Profiles > find and delete. Retry deployment.

Alternative: Provide an ExistingInstanceProfile parameter to skip auto-creation.

Instance profile not associated

Console error:

"IAM instance profile xxx is not associated with an IAM role"

Where to check: AWS Console > IAM > Roles > Instance Profiles.

Root cause: The ExistingInstanceProfile parameter references a profile that has no role attached.

Fix: AWS Console > IAM > Roles > Instance Profiles > find the profile. Ensure a role is attached to the instance profile. The role must have at minimum: AmazonBedrockReadOnly, AmazonSageMakerReadOnly, SecretsManagerReadWrite.

UserData / Bootstrap failures

Stack succeeds, but the lightweight worker node shows the Awaiting Activation status

Symptom: CloudFormation shows CREATE_COMPLETE, but the OneTrust platform shows the Awaiting activation status.

Where to check: AWS Console > EC2 > Instances > find the instance > check Status checks. SSH to the VM: ssh -i <key.pem> ubuntu@<vm-ip>. Check bootstrap log: cat /var/log/bootstrap.log. Check installer log: cat /home/ubuntu/installer.log.

Root causes: VM cannot reach <tenant>.onetrust.com - firewall/proxy blocking outbound HTTPS. installer.shdownload failed - network timeout. deploy.sh install failed - K3s installation or Helm deployment error. Proxy not configured correctly in UserData.

Fix: Test connectivity from VM: curl -I https://<tenant>.onetrust.com. If proxy is required, ensure DeploymentProxy parameter was set. Check installer.log for specific error. If K3s/Helm failed, see Lightweight Worker Node Troubleshooting and Administration.

CustomScript timeout - installer download stuck

Symptom: EC2 instance state is Running, but installer.log shows curl timeout or partial download.

Where to check: SSH to VM > cat /home/ubuntu/installer.log.

Root cause: The installer downloads 5–10 GB of container images. On slow networks or through proxies, this can timeout.

Fix: Check network throughput from VM: curl -o /dev/null -w "%{speed_download}" https://<tenant>.onetrust.com/api/discovery-scan-config/v1/installer-config/<id>/download. If proxy is throttling, work with your network team.

Manually re-run the installer:

cd /home/ubuntu
curl -L -o installer.sh -H "Authorization: Bearer <TOKEN>" \
  "https://<tenant>.onetrust.com/api/discovery-scan-config/v1/installer-config/<WN_ID>/download"
chmod +x installer.sh
echo "1111" | ./installer.sh extract-zip
cd onetrust-dd-installer-*/data-discovery/litenode-deployment
./deploy.sh install skip-prompt

AI Guard-specific failures

Security group ingress creation failed

Console error:

"InvalidParameterValue: CIDR block xxx is invalid"

Root cause: AIGuardSourceCidr parameter has an invalid CIDR format.

Fix: Ensure AIGuardSourceCidr is a valid CIDR (e.g., 10.0.0.0/24) or leave blank to default to VPC CIDR.

CloudFormation console

Go to the AWS Console > CloudFormation > Stacks and select ot-<stack-name>. Confirm that the status is CREATE_COMPLETE, then open the Outputs tab and check for:

• EC2InstanceId - the EC2 instance ID

• VPCId - VPC used

• SubnetId - subnet used

• KeyPairUsed - key pair name

EC2 console

Go to AWS Console > EC2 > Instances and select the instance. Confirm that the instance state is Running, both Instance status and System status checks are green, and the tag Name: ot-<stack-name>-ec2 is present.

SSH verification

# Use the key pair from the stack (if auto-created, download from CFN console)
ssh -i <key.pem> ubuntu@<vm-public-or-private-ip>

# Check K3s is running
sudo systemctl status k3s

# Check pods
kubectl get pods -n onetrust
# Expected: 18 pods Running (17 without AI Guard)

# Check installer completed
cat ~/installer.log | tail -20

OneTrust platform

To check the lightweight worker node status, go to Data Discovery > Worker nodes. The status should transition from Awaiting activation to Healthy within 20–30 minutes. If it is still in Awaiting activation after 30 min, SSH and check installer.log.

Delete a failed stack

Go to AWS Console > CloudFormation > Stacks, select the stack, and choose Delete. This terminates the EC2 instance and deletes all resources created by the stack.

Orphaned resource cleanup

If a stack fails and rollback doesn't clean up properly:

OneTrust platform cleanup

To delete a lightweight worker node, see Managing Worker Node Details: To delete worker nodes.

The delete action removes the OneTrust resource but does not delete AWS cloud resources. You must manually clean up AWS resources.