O365 (Microsoft) SharePoint Connector

Overview

The Data Discovery SharePoint connector is designed to discover and scan various file types that reside in SharePoint sites, subsites, Document Libraries (Drives). As part of the Catalog and Scan process it can report on the volume of files being discovered and scanned.

Pod Residency: o365-scanner
Connection Method: API

Supported Scan Actions

Shared access is only captured and displayed for the 'Catalog & Scan' option. The 'Catalog' scan type is excluded. If you want to capture shared access during scanning, please use the 'Catalog & Scan' type.


Scan Type	Description
Catalog	This scan type will retrieve metadata from the SharePoint sites and list of drives, folders, and files within SharePoint.
Catalog & Scan	Data will be fetched from the SharePoint sites and list of drives, folders, and files within SharePoint, plus the file content from all files included in a SharePoint instance.

Credential and System Permissions

OneTrust recommends to adhere to your organization's security and infrastructure policy and include system engineer resources or other qualified subject matter experts to set up credentials on data sources. For specific instructions for how to enable these credentials, refer to your data source's implementation documentation, knowledge base and/or support resources.

Microsoft Graph supports Client Credentials for authentication based on the scenario. For more information, refer to Microsoft Graph authentication provider documentation.

You will need several values from this app registration for your vault credential that OneTrust Data Discovery will reference. From your app registration, your vault credential will need to securely record the Application ID (Client ID), and Directory ID (Tenant ID). Also, from the Endpoints section of your app registration, your vault credential will need the "OAuth 2.0 token endpoint (v2)" URL.

Finally, as part of your app registration you have created, under "Certificates and Secrets" you need to create a new client secret. This is the last item you need for your vault configuration. Record this securely as soon as you create it per your security policy and this should not be shared with OneTrust or anyone else not authorized. Ensure you capture the actual secret "value." Azure hides this value after it is created so you must get it the first time you create it.

CAUTION:

The Secret ID will NOT work, do not use this value in your credentials.

All Sites

In your app registration, under API Permissions, set the following permissions all found under the Microsoft Graph API section except where noted otherwise.


Permission	Type
Files.Read.All	Application
Sites.Read.All (Found under SharePoint APIs category)	Application
Sites.Read.All (Graph API) Note: Can be used as alternative to SharePoint's Sites.Read.All permission.	Application
User.Read.All (Found under SharePoint APIs category)	Application
Directory.Read.All	Application
Group.Read.All	Application
User.Read Note: When your credentials are created, Azure automatically creates this permission. Please leave it and do not delete:	Delegated

Select Sites

If you prefer to grant access only to specific SharePoint sites instead of all sites, use the following permissions:

Important:

There are additional access requirements needed for specific sites. See Microsoft documentation here.


Permission	Type
Files.Read.All	Application
Sites.Selected	Application
User.Read.All (Found under SharePoint APIs category)	Application
Directory.Read.All	Application
Group.Read.All	Application
User.Read Note: When your credentials are created, Azure automatically creates this permission. Please leave it and do not delete.	Delegated

Sample JSON for Select Sites

JSON

{   
    "clientId": "<clientId>",
    "clientSecret": "<clientSecret>",
    "host": "<Host>",
    "isSiteSelected": "true",
    "scopes": "https://graph.microsoft.com/.default",
    "tenantId": "<Tenant-ID>" 
}

{   
    "clientId": "<clientId>",
    "clientSecret": "<clientSecret>",
    "host": "<Host>",
    "isSiteSelected": "true",
    "scopes": "https://graph.microsoft.com/.default",
    "tenantId": "<Tenant-ID>" 
}

Scan Profile Support

Note:

For additional details on scan profile configuration and management, see Setting Up Scan Profiles.

Limits

File Types
Include/Exclude Paths for support site, folder, file

Include Path Config


Object	Description
Site	Enter the root site name. If scanning for specific sites, specify the target site(s) in the "include" section of the scan profile.
Subsite	Root site name followed by Subsite path.
Drive	Enter the drive name.
Folder	Supports wildcards to scan the rest of folder path.
File	Enter the file name as is, optionally include the file extension. Also supports wildcards to scan all files.

Collected Objects & Fields


Objects	Fields
Files	ID Modified Date Created Date Owner (User) Size on Disk Content Type Path URL (Library, Sites) ReferenceId (Library)
Discovered via Data Governance metadata attributes	Direct Link (Drive, File, Folder) Shared Access (Files) Web URL (Site)

Concerning Volume/Size Metrics

The Volume Scanned field on the Job Insights represents the total amount of data extracted by this job, which is the cumulative sum of the content sizes of all the files extracted during the job. The content size of each file is obtained from the file's metadata. This will be displayed accurately once all results are published.

Note:
The limits specified by scan profile as designed would affect the amount of Volume Scanned.

Exception: In case of archive/compressed files such as zip, tgz, gz, tar, 7z, etc., the size of the archived/compressed file is include and not the uncompressed size.
The Volume Scanned at the Data Source Level does not include the sum of each job's scanned volume. Instead, it reflects the content size of distinct files scanned within the data source. For instance, if the same file is scanned by multiple jobs within the data source, the data source level volume size should be considered as the content size of that file as a single entity. In other words, it represents the aggregated content size of distinct files scanned by the data source.
The Total Size at the Job Level is the total amount of data reported by the SharePoint for the site. This includes the storage used for all the drives under the site that were discovered during the job.

Differences Between Volume Scanned and Storage Used in SharePoint Admin

Depending on the scan profiles used for all the jobs, the Total Size/Volume Reported may differ from the Storage Used reported by SharePoint Admin. SharePoint supports versioning and backups, contributing to the storage used at drive, site or tenant level. SharePoint storage used includes the storage consumed by all versions of a file.

Volume Scanned in Job Insights include the latest version of each file extracted in SharePoint.
Volume Scanned at Data Source Level includes the max size of the file since it was first cataloged.

For instance, if a scan job identifies a file size as 10MB and later the file is edited, reducing its size to 8MB, the subsequent scan job will record the new size of 8 MB in the Volume Scanned on Job Insights. Assuming no other files were altered or added, the Volume Scanned at the Data Source Level would remain unchanged after the second scan job, since the file's updated size is smaller than the original. However, due to versioning, SharePoint usage would increase by 8 MB because it retains both versions of the file.

For additional information and guidance regarding these topics, see O365 SharePoint's Usage Reports and Versioning guides.

Data Use Governance