AWS Fargate Deployment Guide

This document outlines the steps required to install OneTrust Data Discovery solution using AWS EKS to run Kubernetes pods via Amazon Fargate. Currently, this installation is only supported on Mac-OS and Linux systems. Users may have to install a Linux instance on their Windows machine if they lack access to Mac-OS.

Prerequisites

  • AWS EFS instance which can access by AWS EKS worker nodes.

  • AWS environment, with the following:

    • AWS subscription which can run the required cluster.

    • An owner/contributor user of the AWS subscription which will be used to create and manage the cluster.

    • Virtual Network and subnet in which the AWS cluster is getting deployed should have access to data sources.

    • Virtual Network and subnet in which the AWS cluster is getting deployed should have access to Internet to reach Docker Hub.

  • A local (laptop) or remote (Jump box/bastion) machine which will be used to access the cluster master.

    • Minimum: 4 CPU, 8GB RAM, 250 GB HDD, Ubuntu 16+

    • Should have access to Internet to pull the OneTrust Docker Images from docker hub.

    • Docker >=19.x ce + pre-installed (OneTrust can also install this)

  • Latest version of kubectl, with a minimum version of 1.22

Creating Fargate Profile in AWS

  1. Log in to your AWS account.

  2. Search and select EKS.

  3. Select your cluster.

  4. Select Configuration > Compute.

  5. Locate the Fargate Profile and select Add Fargate Profile.

  6. Assign a unique name for the new Fargate Profile.

  7. Select FargatePodExecutionRole.

    Tip:

    An AWS Role must be created for the cluster during Fargate Profile creation. This role does not exist by default and must be created by the customer. For more information, see the following resources:

  8. Select Private Subnet > Next.

  9. Provide the Namespace name.

    Note:

    Ensure that the Fargate Profile has two namespaces assigned: the worker node namespace and the kube-system namespace. You will note this information in the Configure pod selection screen when adding the profile.

    The namespace created in Fargate Profile shall be used as the namespace for the installation.

  10. Create the Fargate Profile.

Fargate Cluster Configuration Checks

Please review this cluster configuration check before advancing with your worker node deployment.

Namespace Check

  1. Ensure the kube-system namespace is available:

    CODE 
    kubectl get namespaces

  2. Ensure the corresponding pod is running in the namespace kube-system.

    CODE 
    kubectl get pods -n kube-system

Applied Annotation Check

  1. If the coredns-xxxx pod is in a Pending state, describe the deployment of the coredns using the below command:

    CODE 
    kubectl describe deploy/coredns -n kube-system

  2. Check if the following annotation is applied:

    CODE 
    eks.amazonaws.com/compute-type: ec2

  3. If the ec2 annotation is applied, run the following command:

    JSON 
    kubectl patch deployment coredns \
    -n kube-system \
    --type json \
    -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
    Note:

    If the cluster was created via the eksctl command, this should be already done.

EFS Check

  1. Ensure the VPC has enabled the DNS hostnames and DNS resolutions. If not, you must manually enable them.

  2. Ensure the EFS will be created on the same VPC as the same region as the cluster.

  3. Ensure the Regional option under Availability and Durability is selected while creating the EFS.

  4. Note the IPv4 CIDR for the VPC tied to the Fargate cluster.

  5. Navigate to the Security Group assigned to the EFS, and add an inbound rule to ensure the VPC IPv4 CIDR is safelisted, with the type as NFS.

Guided Installation Steps

  1. On the Data Discovery menu, select Worker Nodes. The Worker Nodes screen appears.

  2. Click the Add Worker Node button. The Create On-Premises Worker modal appears.

  3. Enter a name and description for the new AWS Fargate worker node.

    Note:

    The name you enter here will be populated as the namespace in the follow screen. It will default to lowercase with no spacing in order to meet the namespace formatting requirements.

  4. Select the Guided Install option.

  5. Click the Next button. The Platform Provider and Credential Store screen appears.

  6. In the Platform Provider and Credential Store screen, select Fargate as your platform provider. The interface will auto-populate fields applicable for the system.

    • Platform Provider

    • AWS Region

    • EKS Cluster Name

    • Elastic File System ID

    • Elastic File System Region

    • Namespace Name

      Note:

      For the Namespace Name field, input the namespace used during the Fargate Profile creation.

  7. In the Credential Store block, complete the fields with your applicable vault details.

  8. Click the Next button. The Docker Registry Details screen appears.

  9. Create a new directory folder where you want to perform the install.

  10. Once the worker node is created, navigate to Settings tab and click the Launch Install button.

  11. Copy the curl command paste it into the folder you previously created.

  12. Run the command:

    CODE 
    ./installer.sh extract-zip

  13. Run the following command so the user has the necessary installation privileges.

    CODE 
    chmod +x installer.sh

  14. Navigate within the Data Discovery installation folder. Below is example path:

    CODE 
    /onetrust-dd-installer-k8s-aws-eks-202307.1.0.ddsecurity1/data-discovery/k8s-deployment

  15. Navigate into the inventory folder and locate the .guided-envs file.

  16. Ensure both variables exist within the .guided-envs file:

    CODE 
    export EXTLIB_SHARED_STORAGE_NAME='efs-sc'
export STORAGE_CLASS_NAME='efs-sc'

  17. Run the command:

    CODE 
    ./deploy.sh install efs-new

  18. Ensure the pods are up and running.

  19. Port forward and create the credential if you are using an internal vault.

Manual Installation Steps

  1. On the Data Discovery menu, select Worker Nodes. The Worker Nodes screen appears.

  2. Click the Add Worker Node button. The Create On-Premises Worker modal appears.

  3. Select the Manual Install option.

  4. Click the Save button. The worker node is now created and available in the Worker Nodes screen. Use the search bar to quickly locate the new worker node.

  5. Use the Summary, Logs, and Settings tab to edit, review, or further configure the worker node.

  6. The Worker Node ID can be copied in the Settings tab. Save this for later refernce during deployment.

  7. Create a client credential. For reference, please see No Content found for /db/organizations/onetrust/repositories/portal-production/content/documents/4-DocPortal/solutions/data_use_governance/data_discovery/145030-Data_Discovery/client_credential_setup.dita below.

  8. Download the latest .zip installation package and move it to the respective location on the VM.

  9. Rezip and copy the inventory_template to the template.

    CODE 
    cp -r inventory_template inventory

  10. Change the directory to /inventory

  11. Edit the .env file and input the necessary fields and details below:

    • Provide the namespace name which was configured in Fargate Profile

    • Input the docker configuration details

    • Update storage class configuration to efs-sc

    • Input the scanner configuration, including the Worker Node ID and Client Credentials.

    • Update shared storage configuration to efs-sc

    • Input the EFS configuration

    • Set IS_FARGATE_PROFILE to true

  12. Edit the .envs-aws file and input the necessary fields and details below:

    CODE 
    export CLOUD_AWS_REGION=" "
export CLOUD_AWS_EKS_CLUSTER=" "

  13. If using Instance Profile, provide the following:

    CODE 
    export IS_AWS_INSTANCE_PROFILE="true"

  14. Run the command:

    CODE 
    cd ../

  15. Run the command:

    CODE 
    ./deploy.sh install efs-new