Understanding Duplicate File Detection

Note:

This functionality is current under Public Preview. Please contact OneTrust Support or your dedicated representative to submit a request.

For every file that is scanned and logged in the Data Catalog, the system has the ability to determine if a file could be a potential duplicate of another file. This allows users to identify if large number of documents are redundant so they can avoid unnecessary storage and mitigate risk and monitoring costs.

Things to Know

  • Only files of the same file type will be considered a duplicate. So if one version of a powerpoint presentation is saved as a PDF, it will not be considered duplicate.

  • Identical files of the same type but if one has additional content, that will not be considered duplicate since the the content increases the object size.

  • Duplicate detection is only available for data of new scans.

  • Original file detection is not currently supported.

  • The following file types are excluded for duplicate detection:

    • Content Types: audio, video, font, and model files.

    • File Types + Size Combinations: PDF, Word, and PPT files that are 5 MB or greater.

    • S3 Files: CSV and TSV files.

To enable classification profiles for duplicate detection

You will need to configure the classification profile you intended to scan with so that it is enabled for document duplicate detection. This can be done for brand new profiles or turned on for existing ones. You will, however, need to rescan a data source with the enabled classification profile for the system to pick up duplicates.

  1. On the Data Discovery menu, select select Setup > Classification Profiles. The Classification Profiles screen appears.

  2. If you are creating a new classification profile, click the Create New button.
  3. In the creation screen, enable the setting Document Duplicates. Apply your classifiers, then save the profile.


  4. For existing classification profiles, click on the name of a profile in the list screen. The Classification Profile Details screen appears.

  5. In the Details tab, enable the setting Document Duplicates.


  6. Click the Save button to retain this change.

To review/manage duplicate files

  1. The Files list screen includes a column, Has Duplicates, to indicate if the file record has identified potential duplicates.

    Use the list screen filter to surface up file records with this indication.



  2. The Duplicates tab in each File Record screen will display the number, occupied size, and additional metadata, which includes the file name, which data source it was scanned from, the shared access type and its last modified date.


  3. Clicking a duplicate file name will take you to that respective file's record screen.
  4. The Shared Access column captures the access type. Click the question mark to display the Shared Access Definitions modal.


  5. Once you have determined which files are necessary and which can be removed from the catalog, you can manage these files in your data source so they are no longer scanned and presented during discovery review.