Scanning a Website

The first step of your Cookie Consent implementation is scanning your domain. The main aim of scanning a domain is to identify first- and third-party cookies, tags, trackers, pixels, beacons, forms and storage.

When a new website is added to the OneTrust application to be scanned, a record is created for the domain and the scan. The scan will remain in the Pending status while records are being created and until the scanner starts.

The scanner is a virtual machine that runs on Google Chrome. It will only scan pages that are within the domain that you have entered. For example, if you scanned onetrust.com and there is link within onetrust.com to zentoso.com, zentoso.com pages would not be scanned. Subdomains of the domain scanned will be included in the scan if scanned from the root domain. For example, if you scan onetrust.com, all subdomains of onetrust.com will be included in the scan, but if you scan www.onetrust.com only, this subdomain and pages/paths of this subdomain will be scanned.

Note:

OneTrust leverages a Google Chrome-based scanner. This scanner, which replaced the Firefox-based scanner, provides a more holistic view of tracking technologies that are dropped on your site and features enhanced stability and performance.

You can leverage the Google Chrome scanner for the following benefits:

  • Improved functionality for identifying third-party tracking technologies.

  • Reduced scan times.

  • Improved performance when scanning behind login authentication.

  • Enhanced bandwidth for large scan volumes.

  • Built-in unique user agent.

When a domain scan is added in the OneTrust tenant, the OneTrust cookies OptanonConsent and OptanonAlertBoxClosed will be added in the Categorizations tab. For more information on these cookies, see OneTrust Cookies.

CAUTION:

If your scanned domain does not align with the domain where the script is integrated, consent will not be correctly captured and the banner will reappear every time you load the page.

The scanner treats any URL starting with www. as a subdomain.

If you scan www.onetrust.com, the OneTrust cookies will only write to this subdomain with base script functionality. 

If you scan onetrust.com, the OneTrust cookies will write to www.onetrust.com, cookies.onetrust.com, consent.onetrust.com, and so on. Similarly, if you scan a path like onetrust.com/company with the Limit scan to this path within site setting enabled, the OneTrust cookies would be saved to that path.

For more information, see OneTrust Cookies.

Any cookies that are dropped on a user action and not page load (for example, a form submission other than login or add-to-cart action) will not be recorded in the scan results. If you have a login page behind which there are additional cookies and would like OneTrust to identify those cookies, see Scanning Behind Authentication. If the login page is redirecting to a different domain, then the login URL as well as redirect domain both will be required to be entered in the login configuration.

While the scanner will attempt to reach all pages it can, scanning every page on your site is not necessary. Scanning most of the pages will record the cookies and allow you to perform the necessary blocking activities efficiently.

If there is a particular page that you would like to include/exclude in the scan, or scan with priority or add it to the target pages in the scan configuration. You can also enter your sitemap directly in the scanner configuration. If you put a page limit in the scan configuration, the scanner will only scan up to the limit of pages, but is not guaranteed to scan that exact number of pages.

When the scanning activity has completed, the status changes to Processing Data.

During the data processing activity, the scan data will be migrated to your tenant and categorized based on the OneTrust Cookie database Cookiepedia.

Once the data processing activity is complete, the scan will move into a Completed status.

OneTrust typically scans a website within 6 to 24 hours. However, the scan time is dependent on the number of scans added to your OneTrust environment at one time and the number of pages a website has. For example, if your site has 15,000 pages, it might take longer than the suggested time for the scan to complete. To help improve scanning time, you can limit the number of pages scanned for a domain.

Scanner Statuses

Pending

Your scan has been submitted and the proper tasks are being created for the scan to begin.

Queued

Your scan is waiting to begin once other scans have completed.

Scanning

The scanner is now scanning the domain that you have entered to be scanned. The scanner acts like a user clicking from page to page. During this process the cookies dropped on each page load will be recorded.

Note:

While it is normal for scans to take multiple days, if your scan is taking too long to complete you can click Get Help on the Context menu to retrieve additional scan metadata. This will help support resolve the issue.

Completed

The scan has completed when in this state, you can click into the domain and view the scan results. For more information on viewing scan results, see Viewing Scan Results.

Partially completed

Authentication of the scanner failed. Check your login credentials and configuration. For more information on how to configure login, see Scanning Behind Authentication.

Scan error

This status indicates the scan has completed, but there was an issue migrating the data.

Canceled

A user canceled the scan.

To add a website to the scanner

  1. select On the On the Cookie Consent menu, select Websites.

    The Websites screen appears.

  2. Click the Add website button on the top right of the screen.

    The Add website screen appears.

  3. Complete the fields. For more information, see Add Website Screen Reference.
  4. Click the Continue to banner setup button.
    Note:

    You can click the Scan only button to scan your website without configuring an experience kit. See Add Website Screen Reference.

  5. Select an experience kit (geolocation rule group) to which you want to assign the website.

    For information on configuring custom experience kits, see To configure a custom experience kit.

  6. Click the Scan button.

    The Review configurations screen appears.


    Review configurations screen
  7. Review your banner and preference center configurations.
  8. Click the Confirm button.
  9. When the website scan is finished, the status will display as Completed.
Note: To safelistlist the OneTrust user agent:

Mozilla/5.0 (X11;\ Linux\ x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.199 Safari/537.36;OneTrust;

OneTrust recommends using scanner IPs to safelist the user agent. You have to safelist the OneTrust user agent if bot detection is present on the site that could block OneTrust from accessing the site.

Add Website Screen Reference





Field

Description
Details

URL

Enter the URL of the site you want to scan. The depth of the scan will depend on how the URL is entered. The domain entered will be top level domain scanned with the subdomains being scanned depending on how the URL is entered as described below.

Below, fid examples of how each site would be scanned depending on the URL entered when scanning:

URL Entered
Scanning Result
http://www.example.com

Only scan the root domain.

Note:

Note: this scan would only scan pages that include www.example.com like www.example.com/blog but it would not scan blog.example.com as it does not include the root www.example.com

example.com

Scan the root domain and all subdomains.

Note:

This is the OneTrust recommendation for scanning domains Note: this scan will include pages, paths and sub-domains that include example.com. For instance, it will include abc.example.com, xyz.example.com; however, it will not include abc.example.de.

http://www.example.com/sub

Scan only the subdomain (www.example.com).

example.com/sub

Scan the subdomain (example.com) and all lower domains.

Note:

The scanner text field has a character limit of 1024 characters.

Owner

Select a user to designate as the website owner. The user launching the scan displays as the owner by default.

Users designated as owners are notified via email of changes to their website scans and associated tracking technologies.

Manage consent separately from root domain

Note:

This setting only displays when scanning a subdomain.

Enable this setting to capture and manage consent for this subdomain independently from its root domain.
Advanced Settings

Sitemap

If you want to scan a site in a particular pattern or scan pages that may not be easily accessible through the website interface, enter the URIs for hosted XML sitemaps, each on its own line.

Smart Scope

Enable this setting to automatically limit the scanning scope to 1,000 pages and prioritize sitemap pages.

Custom number of pages in scan scope

Enter the number of pages to which you want to limit your scan.

OneTrust recommends limiting scans to 2,000–3,000 pages.

Note:

The scanner can scan up to 15,000 pages. OneTrust does not recommend scanning all 15,000 pages. However, if this is required, OneTrust recommends splitting pages with sitemap URLs and scanning them separately.

Skip pages outside sitemap

Enable this setting to only scan pages specifically included in the sitemap.

Organization

Select the organization responsible for the website.

Note:

Users who have access to this organization will be able to view the scan.

Scanner server location

Select the location from which you want the scan to originate.

By default, OneTrust scans the website from North Europe. If your website is not accessible from Europe, please safelist our IP address. For more information on OneTrust IP addresses, see About OneTrust Hosting Options, Backup, and Locations.

Capture time delayed cookies

Enable this setting to instruct the scanner to wait for the configured amount of time (up to 60 seconds) before scanning webpages in order to detect time-delayed cookies.

Query parameters

If you want to limit the scan to pages with certain query parameters (which appear at the end of a url in the format ?parameter=value), enter the parameters separated by commas.

Target specific sections

Enable this setting to include, exclude, or prioritize certain pages in the scan.

Configure the fields and enter the URLs for the pages, paths, or subdomains.



Field
Description
Condition

Select a condition to include only, exclude, or prioritize specific pages in your scan.

  • Include: scan only the pages, paths, or subdomains you specify.

  • Exclude: scan your domain except for the pages, paths, or subdomains you specify.

  • Prioritize: scan the pages, paths, or subdomains you specify first before scanning the rest of your domain.

Section

Select a section to define the scope of your inclusion, exclusion, or targeting.

  • Pages: limit the scan to specific pages on your domain.

  • Paths: limit the scan to all the pages on specific paths within your domain.

  • Subdomain: limit the scan to specific subdomains within your domain.

    Note:

    Subdomain is not available for selection if the Manage consent separately from root domain setting is enabled.

URL list

Enter the exact URLs for the pages, paths, or subdomains, with each one on its own line.

Note:

This field has a limit of 100 domains or 6000 characters, whichever comes first.

Add target section

Select to add another site section to target in the scan.

To configure a custom experience kit

When scanning a domain, if you click Continue to banner setup, you will be prompted to select or create an experience kit after adding a website.

Creating a new experience kit will add a new geolocation rule group. You can manage your geolocation rule groups on the Geolocation rules tab.

For more information, see Configuring Geolocation Rules.

  1. Select Create custom experience kit on the Assign experience kit screen. The Select audience screen appears.


  2. Enter a name and select the framework(s) for your experience kit.

    Configure additional languages by clicking the Manage languages button.



  3. Click the Next button. The Choose a cookie banner layout screen appears.


  4. Configure your banner layout and branding.
  5. Click the Next button. The Choose a preference center layout screen appears.


  6. Configure your preference center layout and branding.
  7. Click the Next button. The Review configurations screen appears.


  8. Review your configurations and make changes as needed.
  9. Click the Confirm button.

To re-scan an existing website

  1. On the Cookie Consent menu, select Websites. The Websites screen appears.

  2. Hover over the row for the website which you want to re-scan until the Context Menu icon appears. 
  3. Click the Context Menu icon. The Context menu appears.

  4. Select Re-scan. The Re-audit modal appears.


  5. Complete the fields. For more information, see Add Website Screen Reference.
  6. Click the Scan Only button.

To reassign a website to a different organization

  1. On the Cookie Consent menu, select Websites.  The Websites screen appears.

  2. Hover over the row for a website you want to reassign until the Context Menu icon appears. 
  3. Click the Context Menu icon. The Context menu appears.

  4. Select Reassign. The Reassign Organization modal appears.


  5. In the Organization field, select the organization to which you want to assign the website.
  6. Click the Reassign button.

To schedule a website scan

Note:

Scheduled scans are limited to 20 per day unless overridden at the tenant level. Contact OneTrust Support to override the default scan limit.

If there are leftover scans scheduled, they will be picked up the following day.

Example:

Day 1: 30 scans scheduled. 20 scans run.

Day 2: 10 scans scheduled. 10 day-2 scans are run, plus 10 leftover day-1 scans.

  1. On the Cookie Consent menu, select Websites.  The Websites screen appears.

  2. Hover over the row for the website for which you want to schedule a scan until the Context Menu icon appears. 
  3. Click the Context Menu icon. The Context menu appears.

  4. Select Schedule. The Schedule Scan modal appears.


  5. Configure the fields.
    Field
    Description
    Scan frequency (in months)

    Select the number of months to determine scan frequency.

    Next Scan

    Select a date for the next scan.

    Geo-location

    Select a scanner location.

    Notify When Done

    Enter an email address to receive a notification when the scan is complete.

    Note:

    You can only add one email address.

  6. Click the Save button.
Note:

You can schedule scans for your domains based on new content being added and maintaining your Cookie Consent implementation. Read more here.

To stop a website scan

  1. On the Cookie Consent menu, select Websites.  The Websites screen appears.

  2. Hover over the row for an In Progress website scan you want to stop until the Context Menu icon appears.
  3. Click the Context Menu icon. The Context menu appears.

  4. Select Stop. A confirmation modal appears.
  5. Click the Confirm button.
Note:

If a scan is Scanning, the scan will stop, and the results will be available as the latest scan result.

If a scan is Pending and there are no previous scans completed on the site, the scan is removed.

If a scan is Pending and there is a history of one or more successful scans, then only the new scan is removed from the pending list, and previous reports will be available as before.

To access the scan job queue

  1. On the Cookie Consent menu, select Websites.

    Websites screen within the Cookie Consent product showing a list of websites.

    The Websites screen appears.

  2. Click View job queue button.

    Job queue screen in the Cookie Consent product showing a list with one scan job.

    The Job queue appears.

  3. Select a website name from the queue. The job details appear.

    In the Scan configuration section, you can review details about the website being scanned, including the sitemap, number of pages being scanned, and scanner server location. These details reflect the configuration at the time of the scan being launched.

    In the Targeted pages that encountered an issue section, you can review any webpages where the scanner encountered errors, including the page URL, the error description, and the date the error occurred.


    Job details screen in the Cookie Consent product showing scan configuration information and a section for targeted pages that encountered issues.
  4. Select the Scan checks tab. The scan checks appear.

    Scan checks tab in the Cookie Consent product showing grouped validation categories for website scans.

    Scan checks are validation measures the scanner takes to ensure the website is scanned successfully. The checks are grouped by the following categories:

    • Reachability

    • Accessability

    • Crawlability

    • Protection

    • Health & Performance

    You can expand or collapse each category of scan checks to view or hide its details.

To view a website's activity history

In Cookie Consent, you can access an audit log of activity for each of your domains to review any changes made over time.

  1. On the Cookie Consent menu, select Websites. The Websites list screen appears.

  2. Select a website from the list. The Website Details screen appears.

  3. Go to the Activity tab. The Activity screen appears.

  4. Review the website's activity history.

To export website data

  1. On the Cookie Consent menu, select Websites. The Websites list screen appears.

  2. Click the Column Selector icon. The Website Column Selector modal appears.


  3. Use the arrow keys to configure column visibility for the Websites list screen. This configuration will be reflected in the export.
    Note:

    Column sorting will also be reflected in the export file. Click the header of a column to sort the list screen in ascending or descending order of that column's data.

  4. Click the Save button.
  5. Click the Export button in the header. A confirmation modal appears.
  6. Click the Confirm button.
  7. Click the Notifications icon . A website data export appears in the Notifications popover.


  8. Click Download. An excel report downloads to your device.
  9. View the data export.

To delete a website from the scanner

  1. On the Cookie Consent menu, select Websites.  The Websites screen appears.

  2. Hover over the row for a website you want to delete until the Context Menu icon appears.
  3. Click the Context Menu icon. The Context menu appears.

  4. Select Delete. A confirmation modal appears.
  5. Select the data you want to delete.
  6. Click the Confirm button.

Cookie Scanner IP Addresses

Follow the links to find the current scanner IP addresses.

OneTrust

CookiePro

Frequently Asked Questions

What should I do when the scanner returns zero pages and zero cookies?

  • Check if the website is available in the region that the scanner is located. For more information, see About OneTrust Hosting Options, Backup, and Locations. Safelisting the scanner’s IP address will help if the website is not available in the scanner’s region.

  • If the website is behind a firewall, that will require safelisting the scanner’s IP address.

  • Check if the website has a re-direct. If this re-direct is to a domain other than the one entered for scanning, the scanner will not yield expected results. For instance, if you are trying to scan example.com but it is automatically re-directed to xyz.com, then the scanner will return zero pages and zero cookies and you would have to scan xyz.com.

If all the above fail or are not applicable, OneTrust recommends adding a few pages in Target Pages to Scan. This can direct the scanner to visit all the pages mentioned in the scan configuration.

How do I scan pages that are behind authentication?

See see Scanning Behind Authentication for information on configuring a scan to pick up pages behind login.

Why does the scan return different results with every re-scan?

Multiple scans of the same website can produce varying outcomes as the scanner may navigate different paths throughout the site. For instance, in a case where a page becomes unavailable during a scan due to load timeout, and it includes links to other pages that lead to additional links branching out to other pages, there is a possibility that this cluster of pages may not get scanned.

How long does it take for the scanner to complete a website scan?

Scan time varies from a couple hours to several days. Scan time depends on several factors including total number of pages as well as number of scans launched at the same time in the tenant.

How do I scan my website if it has more than 15,000 pages?

You can scan your domain in parts by initiating multiple scans of the same website. Each time, add a few sub-domains in the Target Pages to Scan section of the scan configuration. You can also add a few sitemaps each time in the Sitemap URL section.

Is OneTrust able to perform user actions?

No, our scanner does not have the ability to perform any user actions like checking boxes, accepting existing banners, submitting forms, etc.

What is the difference between the Pages and Total Pages columns on the Websites screen?

Pages is the number of pages scanned for that particular scan.

Total Pages is the sum of pages for all scans ever run for that domain.

Can the scanner successfully scan Single Page Application (SPA) sites?

No. The scanner does not support SPA sites. You can either manually add the pages you want to scan or manually add the cookies relevant to your site.

Does the scanner require any robots.txt directives?

The scanner does not require any specific Allow rule within robots.txt. You only need to safelist the OneTrust user agent within your security or bot‑detection systems (e.g., Cloudflare Bot Management, WAF, or custom firewalls) to prevent the scanner from being blocked.