Scan Management Feature Guide

Scan Management offers Data Discovery users an effective method for monitoring their scan activities with less manual, daily interaction via automatic data source scanning, instant scan profile generation, sequential running of scan jobs, and scan status notifications. This allows users to quickly identify what data has been scanned and respond to notifications when scans are completed or require a direct inspection.

Supported Data Sources

All data sources listed in our connector page are supported for Scan Management.

To create a new plan

  1. On the Data Discovery menu, select Setup > Scan Management. The Scan Management screen appears.

  2. Click the Add Scan Management button. The Set Scan Plan screen appears.

  3. In the Scan Plan Information section, provide a unique name for the plan in the New Scan Name field.


  4. In the Select Data Source section, choose an existing data source for the scan plan.
    Note:

    For more information on creating data sources, see Connecting & Managing Data Sources.



  5. In the Set Discovery Type section, choose which scan type should be applied to the data source.

    • Limited: performs a full scan (meaning no file exclusions) but sets a default limit on the amount of data per object. This is best to narrow the scan scope and reserve resources.

      • Structured data sources, retrieves the first 1000 rows of each table.

      • Unstructured data sources, retrieves the first 100 files for a given folder.

    • Full Scan: allows you to retrieve a scan of the entire data source.

    • Custom: allows you to choose from the available scan profiles configured in your tenant.



  6. In the Select Classification Profile section, choose an existing Classification Profile for the scan plan.
    Note:

    For more information on creating and managing classification profiles, see Creating & Managing Classification Profiles.



  7. Click the Save & Begin Scan Process button. The new scan plan will be added to the Scan Management list screen.

Phase 1: Connection Test & Run Scan

The first step is testing the connection to the identified data source. This is a prerequisite for the rest of the scan management process.

Available Statuses:

  • Waiting

  • Pending

  • Processing

  • Complete

  • Failed

A successful connection will return a Complete status along with a timestamp of the test.



In the event of a failure, the user will be prompted to validate their credentials and have a chance to retry the connection.



If the data source is inactive or the worker node is unhealthy, the test connection ability will be disabled.



Note:

For more information on testing data sources, see Connecting Data Sources.

Phase 2: Discover Objects

Once the application can successfully test and validate the connection, the chosen scan type will be begin. Once it is complete, the number of objects will be listed. Click the View Job Overview link to inspect job progress.



In the event of a failed scan, the system will display an error and offer users a chance to retry the catalog scan. Users are encouraged review their data source or credentials in addition to retry scan.



Phase 3: Scan Profile Creation

Once the scan ends, the chunking process will divide the data source into multiple scan profiles. The application will then display the number of scan profiles as well as a link to inspect the scan profiles.

The name of each scan profile will follow this pattern: [data source name] + [Num] + [Base36(milliseconds from epoch)]

  • Each element is separated by hyphen. Example: ttps://www.multiutil.com/base10-number-converter/

  • Num refers to the number, starting at 00. Example: if it is the 8th scan profile of 10, it would be “07”

  • Example: "mysql-03-1ggczo45"

Note:

The application will create a scan profile for each identified object, with a max of 10. Example: 6 objects found, 6 scan profiles created.

With this first edition of Scan Management, the size of objects does not impact the number of scan profiles generated.





Phase 4: Scan Execution

Once the scan profiles have been created, the corresponding scan jobs will be sequenced and the first job will begin. The scan jobs will be in either an In Progress, Completed, or Queued status.



The jobs that are Completed or In Progress will provide a link to the Job Overview tab in the Scan Job Details screen.



Any jobs with an error will provide a link to the View Error Log, which is pre-filtered for error messages, for easier navigation from the scan management plan. Use the Retry Failed button to reprocess a failed scan job without having to navigate to the specific job log.





To inspect scan management

  1. On the Data Discovery menu, select Setup > Scan Management. The Scan Management screen appears.

  2. The initial screen will present a list of all created Scan Management plans with the following column details:

    • Scan Management Object Name

    • Discovery Scan Type

    • Data Source Name

    • Data Source Type

    • Created By

    • Created Date



  3. Click on the name of a Scan Management Object Name. The Scan Management Details screen appears.

  4. The screen will present a summary of the scan management progress and list the scan plan's relevant details.


    • Scan Plan Status: displays the current phase status or error.

    • Data Source

    • System

    • Created By

    • Created Date

    • Volume Scanned: Captures the total unique volume that has ever been scanned from the given data source over its lifetime.

To delete a scan management plan

  1. On the Data Discovery menu, select Setup > Scan Management. The Scan Management screen appears.

  2. Hover over the row of a Scan Management Object Name.
  3. Click the Context Menu icon, then select Delete. The Delete modal appears.

  4. Click the Confirm button to delete the scan management plan.
    Attention:

    This is an irreversible action and all incomplete jobs created by the given plan that are Queued or In Progress will be canceled.