Amazon S3 Connector

Overview

Used for OneTrust Data Discovery, the Amazon S3 data source connects to S3 storage options and can be used to scan file system information directly from those file systems.

Pod Residency

storage-scanner

Connection Method

API

Supported Scan Actions

Scan Type

Description

Catalog

This will fetch the list of buckets and objects present in those buckets. It collects the Metadata also for these objects and send back to OneTrust app.

Catalog & Scan

This will fetch the list of buckets and objects present in those buckets. We collect the metadata and then proceed with content extraction. It will return the personally identifiable information, data elements, and terms with the applied classification labels.

Credentials and System Permissions

OneTrust recommends to adhere to your organization's security and infrastructure policy and include system engineer resources or other qualified subject matter experts to set up credentials on data sources. For specific instructions for how to enable these credentials, refer to your data source's implementation documentation, knowledge base and/or support resources.

OneTrust Data Source Credentials

The following credential options are available for Amazon S3 connections. You will need to choose either the basic or advanced option in the Build Credential JSON section. Access to your Amazon S3 console is required to obtain the values for the JSON fields.

For more information, see Creating Data Source Credentials.

Option

Build Credential JSON Fields

Sample JSON

Access Key and Secret Option

  • AWS Access Key ID

  • AWS Secret Access Key

  • Connection URL
JSON 
{
    "aws_access_key_id": "your_aws_access_key_id",
    "aws_secret_access_key": "your_aws_secret_access_key",
    "connectionUrl": "https://aws.amazon.com/"
}

Assume Role via Container

  • Connection URL

  • AWS Auth Scheme

  • AWS Assume Role

  • AWS Role Session Name

  • AWS Disable Force Global Bucket Access.
JSON 
{
    "connectionUrl": "https://aws.amazon.com/",
    "aws_auth_scheme": "AwsIAMRoles",
    "aws_assume_role": "role",
    "aws_role_session_name": "session",
    "aws_disable_force_global_bucket_access": "true"
}

Assume Role with STS Access Key and Secret Option

  • AWS Auth Scheme

  • AWS Assume Role

  • AWS Role Session Name

  • AWS STS Expiry In Seconds

  • AWS Region

  • AWS STS Region

  • AWS STS Endpoint

  • AWS Disable Force Global Bucket Access
JSON 
{
    "aws_access_key_id": "your_aws_access_key_id",
    "aws_secret_access_key": "your_aws_secret_access_key",
    "connectionUrl": "https://aws.amazon.com/",
    "aws_auth_scheme": "AwsIAMRoles",
    "aws_assume_role": "role",
    "aws_role_session_name": "session",
    "aws_sts_expiry_secs": "123",
    "region": "us-east-1",
    "stsRegion": "us-east-1",
    "stsEndpoint": "test",
    "aws_disable_force_global_bucket_access": "true"
}

Permissions

  • s3:ListAllMyBuckets

  • s3:GetBucketLocation

  • s3:ListBucket

  • s3:GetObject

  • s3:GetBucketAcl (Required to obtain shared access metadata)

  • s3:GetObjectAcl (Required to obtain shared access metadata)

  • s3:GetEncryptionConfiguration (Required to build bucket metadata)

Tip:

By default OneTrust will find buckets across all regions in the account being scanned. If you need to limit buckets to a specific region, you should specify the region in the vault credential JSON. Example:

CODE 
"region": "us-east-2"

Sample IAM Role

JSON 
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"VisualEditor1",
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectAcl",
            "s3:GetObject",
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:GetBucketAcl",
            "s3:GetBucketLocation",
            "s3:GetBucketPolicy"
         ],
         "Resource":"*"
      }
   ]
}

Scan Profile Support

Note:

For additional details on scan profile configuration and management, see Setting Up Scan Profiles.

Limits

  • Max number of buckets: you can specify the max count of the buckets which you want to scan in whole S3 account.

  • Max number of files per bucket: you can specify max count of files you want to scan per bucket.

  • Max number of Files per scan: you can specify the max number of files to be scanned per scan job. If Total number of files calculated in Point 2 is more than the value provided in this field. This field will take the priority.

Scope

Scope

Description

Min/Max File Size

You can provide the minimum and maximum file size with appropriate unit to be scanned in this field. By default, If these fields are left empty, there will not be any limit on file size for Catalog.

Exclude File Types

You can provide the File extensions in this field which you are not at all interested to scan. We expect to provide these extensions in comma separated format.

Exclude Paths

Bucket: You can mention the particular bucket name to not to be scanned here. If you provide wildcard i.e * will enable the user to not to scan any bucket present in S3 account. Keeping empty has no effect.

Prefix: You can mention the particular prefixes as well which you don’t want to scan. S3 SDK provide us flat structure which means it adds the Folder Name as Prefix to the fileName.

If you give fileName as well in prefix section it supports it. Following are the formats to be provided in prefix section:

  • Prefix/filename.txt: will not scan the file filename.txt present in folder name prefix

  • PrefixName: will not scan all the files present in folders having name or prefix starting with prefixName (e.g. prefixName1/somefile.txt and prefixName2/prefixName3/somefile.txt)

Include Paths

Bucket: You can mention the particular bucket name to be scanned here. If left empty or provided wildcard i.e * will enable the user to scan all of the buckets present in S3 account.

Prefix: You can mention the particular prefixes as well which you want to scan. Following are the formats to be provided in prefix section.

  • Prefix/filename.txt: will scan the file filename.txt present in folder name prefix.

  • PrefixName: will scan all the files present in folders having name or prefix starting with prefixName (e.g. prefixName1/somefile.txt and prefixName2/prefixName3/somefile.txt).

Collected Objects and Fields

Bucket

The container where files are stored in S3. They are assigned to a specific S3 region. An S3 bucket is globally unique, giving a specific namespace not shared by any other account.

Object

The actual content stored within the S3 bucket. S3 doesn’t have a true file hierarchy system, but virtual folders can be created using prefixes as part of objects (example: folder/file). Each individual object can be up to 5TB and includes:

Objects

Fields

Files

  • Name

  • ID

  • Content Type

  • ETag

  • Modified Date

  • Owner (User)

  • Size on Disk

  • Path

  • Shared Access (Bucket, File)

Note: Access Control Information

You can control access to the objects you store in Amazon S3. Amazon S3 supports both the resource-based access control, such as an access control list (ACL) and bucket policies, and user-based access control. For more information, see Identity and access management in Amazon S3.

For files and folders, we apply the following to the shared access attribute in this priority order:

  • Open Access - If someone can access the file or folder with no restriction with a hyperlink.

  • Shared External - The file or folder is shared externally to another AWS account.

  • Private - The file or folder is not shared by the file owner.

Common Structured Files Support

S3 has special support for CSV, Parquets and JSON Files, which treat these files as similar to DB Table and enable us to select file content using SQL queries.

OneTrust currently makes use of this S3 capability to get the content of CSV/TSV and Parquet Files. We map the classifier label at the column level for these files instead file level.

Parquet Files

OneTrust gets the column names and maps the data extracted per cell to those columns and classify it.

CSV/TSV Files

OneTrust always treat these files with no headers, however S3 enables us to auto generate the column name using a sequence.

For example, if a CSV has three columns, then S3 SDK will return us the columns as _1, _2 and _3. OneTrust will map the data of these files to these columns and classify it. We map the classifier label at the column level for these files instead file level.

Special Support for Compressed CSV/TSVs

S3 has support for compressed files as well, BZIP2 and GZIP type of compression is supported. OneTrust makes use of these feature for only CSV/TSVs, if a file is having below extensions then set the compression type as per extension in our SelectObjectContent request and S3 return us the uncompressed data stream.

  • csv.gz

  • csv.bz2

  • tsv.gz

  • tsv.bz2

Scanning Glacier Deep Archive Files

This section outlines the functionality of the scanner for cataloging files which are stored in Amazon S3 Glacier Deep Archive storage class using the S3 Connector. It also outlines the limitations of scanning files from some of the storage classes.

Overview

Amazon S3 offers various storage classes that determine how files are stored and accessed. Each storage class has its own retrieval characteristics. Files stored in certain classes, such as Amazon S3 Glacier and S3 Glacier Deep Archive, are not immediately accessible and require a restore operation before they can be retrieved.

For example, files stored under the S3 Glacier Deep Archive storage class are archived and require restoration to be accessed. On the other hand, files in the S3 Standard storage class are readily accessible without such a restoration step.

Note:

For more information about the different storage classes, see AWS S3 Storage Classes.

Scanning Behavior

The scanner in S3 Connector behaves differently based on the storage class of the files it is scanning.

  1. Catalog-Only Scan

    The scanner will catalog all files in all storage classes, including those that require restoration (such as Glacier Deep Archive). However, it will not attempt to retrieve or scan the content of the files in these classes.

  2. Catalog and Scan

    When performing a catalog and scan, the scanner will ignore files stored in the following storage classes that require restoration or have restricted access:

    DEEP_ARCHIVE

    (Amazon S3 Glacier Deep Archive)

    GLACIER

    (Amazon S3 Glacier)

    REDUCED_REDUNDACY

    GLACIER_IR

    (Amazon S3 Glacier Instant Retrieval)

Note:

If there are other storage classes where scanning issues occur, you can configure the environment to ignore these classes by setting an environment variable.

To ignore specific storage classes during scanning, run the following command:

CODE 
kubectl set env deployment/storage-scanner S3_IGNORE_EXTRACTION_STORAGE_CLASS="DEEP_ARCHIVE,GLACIER,REDUCED_REDUNDANCY,GLACIER_IR" -n <namespace>

Additional Information

If you need to change the storage class of a file to one that is immediately accessible, such as S3 Standard, follow these steps:

  1. Locate the file in your S3 bucket.

  2. From the Actions drop-down menu, select Change storage class.

  3. Choose the Standard storage class (or another accessible storage class).

  4. Save the change.

  5. After the storage class has been updated, you can trigger a scan and verify the results.

Relevant AWS Documentation

For more details on managing storage classes and restoring files in S3, please refer to the following AWS documentation: