Consent Bulk Import Webhook Integration

You can use a built-in webhook trigger to import large volumes of consent records. The following are instructions for configuring the integration workflow needed to launch the Consent Bulk Import workflow integration.

Prerequisites

To set up the integration workflow

  • A valid user with the CONSENT and INTEGRATIONS scopes.

  • A valid access token for the user which will be used to connect with the Schedule endpoint. For more information, see Client Credentials.

  • A valid SFTP/External FS where the Bulk import file will be shared. For more information, see SFTP Server .

  • A valid Schedule Service Credential using a valid access token or API key. For more information see Schedule Service .

  • The following services should be deployed in the corresponding environment:

    • Schedule

    • Scheduler

    • Document

    • Document-Gateway

    • Consent

API details

A new public API has been added in Schedule Service to trigger the import job. See below the details:

Path

Method

Response

Authentication

/v3/jobs

POST

201 CREATED

Bearer token (the user must have CONSENT scope)
Payload Field
Description
Name

Name for the import job.

Description

Description for the import job.

Type

Import job type has to be "stream_consent_receipts.

Token

Blank for now.

EntityId

Reference ID of the Bulk import file stored in Document Storage.

PYTHON 
{
    "Name":"consent stream log",
    "Description":"this is a consent stream import request",
    "Type":"stream_consent_receipts",
    "Token": "",
    "EntityId": "cb203ff9-fc57-4e83-9763-9ab3c1a591b4"
}

Client Credentials

If the user already exists, you may use an existing token or regenerate a new token based on Client Secret or use an existing API Key. If the user does not exist, please follow the procedure below and note the bearer token or API Key for future steps.

  1. In the OneTrust platform, click the gear icon in the upper right-hand corner to access Global Settings.

  2. Navigate to Access management > Client credentials.

  3. Select the Client credentials tab and click the Add button.

  4. Select the CONSENT and the INTEGRATIONS scopes.

  5. Click the Create button.

Note:

For more information, see Managing OAuth 2.0 Client Credentials.

Integration Credentials

SFTP Server

  1. Navigate to Integrations > Credentials The Credentials screen appears.

  2. Click the Add credential button. The Create credentials screen appears.

  3. In the Select existing system field, select SFTP from the list. Click the Next button.

  4. Enter the SFTP connection details and click Save.



Schedule Service

  1. Navigate to Integrations > Credentials The Credentials screen appears.

  2. Click the Add credential button. The Create credentials screen appears.

  3. In the Select existing system field, select Generic from the list. Click the Next button.

  4. Enter the Schedule Service details and click the Save button.



    • In the Hostname field, add the host details where the service is running specific to the environment.

    • Open the Test settings (recommended) collapsible menu and paste this URL into the Test URL field: https://OT_ENV_URL/api/schedule/v1/jobs.

    • Add the generated bearer token in the Token field if you have selected the Bearer token Authentication method. If other, proceed with the corresponding configuration.

  5. Click the link in the Credential name column for the credential that you have created. The Credential details screen appears.



  6. Click the Test Credentials button to check if the connection was successful.



Note:

For more information, see Configuring Integration Credentials.

To create the integration workflow

Manual Workflow Creation

  1. Navigate to Integrations > Integrations. The Integrations screen appears.

  2. Click the Add integration button. The Add Integration modal appears.

  3. Complete the fields as needed.


    Note:

    Ensure that you select the Generic option in the System field.

  4. Click the Add button. The Integrations builder screen appears.

  5. In the Add OneTrust Trigger block, go to the Built-in tab.
  6. Select Webhook. The Webhook action block appears.

  7. Select Json as your Payload type.
  8. Paste a blank JSON sample in the Sample field.


  9. Click the Done button.
  10. Click the Add action button to add an SFTP action to the workflow. The Add action section appears.

  11. On the System tab, select Download File(s) from FTP.


  12. Select the the existing SFTP credential that you created in the Integration Credentials > SFTP Server section before. Then click the Next button.


  13. Enter the Path field to the directory where the bulk import files are located. Click the Finish button.


  14. Click the Add action button to add the Schedule Job Creation action to the workflow.
  15. On the Logic tab, select Apply to each.
  16. Select step.ftp_0.fileInfoList in the JSONPath to output from previous steps field. This references the file(s) downloaded from the directory.
  17. Click the Finish button.
  18. Click the Add action button again.
  19. On the System tab, select Https POST request.
  20. Select the the existing Schedule Service credential that you created in the Integration Credentials > Schedule Service section before. Then click the Next button.


  21. On the Request tab, update the Request URL field to https://lkgb.1trust.app/api/schedule/v3/jobs.
  22. Select the POST method.
  23. Add the Content-Type key and the application/json value in the Header(s) field.
  24. Paste the following JSON sample in the Body field.
    PYTHON 
    { 
  "Name":"consent import from Integration", 
  "Description":"", 
  "Type":"stream_consent_receipts", 
  "Token": "", 
  "EntityId": "${(step.applyEach_0.reference)!}"
}


  25. Click the Finish button and click Save.
  26. Click the Activate button to activate the workflow.
Note:

For more information, see Managing Custom Integration Workflows.

Importing an Existing Workflow

Follow this procedure to create the Consent Bulk Import integration workflow by importing a .json file.

  1. Copy the code below and paste it into a text editor. Save the document as a .json file called OTWorflow.json to your computer.
    PYTHON 
    {
  "name": "Consent bulk import",
  "description": "wed",
  "enabled": false,
  "workflowType": "COMPOSITE",
  "integrationName": "Consent bulk import",
  "referenceIntegrationName": "Generic",
  "dlqMessageCount": 0,
  "throttlingLimit": 0,
  "defaultDlqAction": "NONE",
  "failedAlertFrequencyHrs": 0,
  "redactLoggingEnabled": false,
  "sensitivity": "DEFAULT",
  "formatVersion": "1.0",
  "trigger": {
    "name": "Consent bulk import",
    "workflowName": "Consent bulk import",
    "triggerType": "WEBHOOK",
    "triggerContext": {
      "WebHookTrigger": {
        "type": "WEBHOOK",
        "eventCode": "9010",
        "referenceUrl": "",
        "schema": null,
        "sample": null,
        "decryptionRequired": false,
        "decryptionCredentialId": null,
        "webhook": true,
        "subFlow": false,
        "config": false
      }
    },
    "description": "wed",
    "eventCode": "9010",
    "schema": "{\"format\":\"JSON\",\"type\":\"record\",\"fields\":[]}"
  },
  "actions": [
    {
      "processType": "FTP",
      "processContext": {
        "FileConnector": {
          "type": "FTP",
          "requestURI": "___REPLACE_WITH_SFTP_URL___",
          "fileName": null,
          "decryptionType": "NONE",
          "decryptionCredentialId": null,
          "signatureVerificationRequired": false,
          "pgpSignatureVerificationStrategy": "IGNORE",
          "internal": false,
          "continueOnError": "NONE",
          "isInternal": false,
          "viewType": null,
          "download": true,
          "deleteCommand": false
        }
      },
      "nodeLabel": "ftp_0",
      "nodeDescription": "Download File(s) from FTP",
      "requestSchema": null,
      "responseSchema": {
        "name": "performDownloadFile",
        "description": null,
        "schema": "{\"format\" : \"JSON\",\"type\" : \"record\",\"fields\" : [ {\"type\" : \"array\",\"name\" : \"fileInfoList\",\"items\" : {\"format\" : \"JSON\",\"type\" : \"record\",\"fields\" : [ {\"type\" : \"string\",\"name\" : \"fileName\",\"default\" : \"sample_file_name\"}, {\"type\" : \"string\",\"name\" : \"absoluteFilePath\",\"default\" : \"\"}, {\"type\" : \"double\",\"name\" : \"fileLength\",\"default\" : 10.0}, {\"type\" : \"string\",\"name\" : \"lastModifedDate\",\"default\" : \"\"}, {\"type\" : \"string\",\"name\" : \"reference\",\"default\" : \"a5dbb737-df43-41e1-a56b-5e20b9d0448c\"}, {\"type\" : \"string\",\"name\" : \"extension\",\"default\" : \"csv\"}, {\"format\" : \"JSON\",\"type\" : \"record\",\"name\" : \"dataFields\",\"fields\" : [ ]}, {\"type\" : \"string\",\"name\" : \"file\",\"default\" : \"a5dbb737-df43-41e1-a56b-5e20b9d0448c:10:csv\"} ]}} ]}"
      }
    },
    {
      "processType": "APPLYEACH",
      "processContext": {
        "ApplyEach": {
          "type": "APPLYEACH",
          "jsonPath": "ftp_0:$..fileInfoList.*",
          "listKey": "step.ftp_0.fileInfoList",
          "commonNodeLabel": "messagelog",
          "continueOnError": null
        }
      },
      "nodeLabel": "applyEach_0",
      "nodeDescription": null,
      "requestSchema": null,
      "responseSchema": {
        "name": "performDownloadFile",
        "description": null,
        "schema": "{\"format\" : \"JSON\",\"type\" : \"record\",\"fields\" : [ {\"type\" : \"array\",\"name\" : \"fileInfoList\",\"items\" : {\"format\" : \"JSON\",\"type\" : \"record\",\"fields\" : [ {\"type\" : \"string\",\"name\" : \"fileName\",\"default\" : \"sample_file_name\"}, {\"type\" : \"string\",\"name\" : \"absoluteFilePath\",\"default\" : \"\"}, {\"type\" : \"double\",\"name\" : \"fileLength\",\"default\" : 10.0}, {\"type\" : \"string\",\"name\" : \"lastModifedDate\",\"default\" : \"\"}, {\"type\" : \"string\",\"name\" : \"reference\",\"default\" : \"a5dbb737-df43-41e1-a56b-5e20b9d0448c\"}, {\"type\" : \"string\",\"name\" : \"extension\",\"default\" : \"csv\"}, {\"format\" : \"JSON\",\"type\" : \"record\",\"name\" : \"dataFields\",\"fields\" : [ ]}, {\"type\" : \"string\",\"name\" : \"file\",\"default\" : \"a5dbb737-df43-41e1-a56b-5e20b9d0448c:10:csv\"} ]}} ]}"
      }
    },
    {
      "processType": "MESSAGELOG",
      "processContext": {
        "MessageLog": {
          "type": "MESSAGELOG",
          "continueOnError": null
        }
      },
      "nodeLabel": "messagelog",
      "nodeDescription": "Terminating Node : messagelog",
      "requestSchema": null,
      "responseSchema": {
        "name": null,
        "description": null,
        "schema": null
      }
    },
    {
      "processType": "TRANSFORM",
      "processContext": {
        "DataTransformer": {
          "type": "TRANSFORM",
          "template": "{ \n  \"Name\":\"consent import from Integration\", \n  \"Description\":\"\", \n  \"Type\":\"stream_consent_receipts\", \n  \"Token\": \"\", \n   \"EntityId\": \"${(step.applyEach_0.reference)!}\"\n}",
          "independentNode": false,
          "continueOnError": null,
          "isIndependentNode": false
        }
      },
      "nodeLabel": "transform_0",
      "nodeDescription": "Https POST request",
      "requestSchema": null,
      "responseSchema": {
        "name": null,
        "description": null,
        "schema": null
      }
    },
    {
      "processType": "HTTP",
      "processContext": {
        "HttpConnector": {
          "type": "HTTP",
          "headers": {
            "Content-Type": "application/json"
          },
          "requestURI": "__REPLACE_WITH_PROVIDED_URL__",
          "httpMethod": "POST",
          "template": null,
          "jweEncryptionRequired": false,
          "jweDecryptionRequired": false,
          "encryptionCredentialId": null,
          "decryptionCredentialId": null,
          "continueOnError": "NONE",
          "pathParams": {},
          "queryParams": {},
          "isInternal": false,
          "isDSPortal": false,
          "viewType": null
        }
      },
      "nodeLabel": "http_0",
      "nodeDescription": "Https POST request",
      "requestSchema": null,
      "responseSchema": {
        "name": null,
        "description": null,
        "schema": null
      }
    }
  ],
  "directions": [
    {
      "startNode": "ftp_0",
      "endNode": "applyEach_0"
    },
    {
      "startNode": "applyEach_0",
      "endNode": "messagelog"
    },
    {
      "startNode": "applyEach_0",
      "endNode": "transform_0"
    },
    {
      "startNode": "transform_0",
      "endNode": "http_0"
    }
  ],
  "categoryBitValue": 258,
  "seedSystem": false,
  "halted": false,
  "dlqEnabled": true
}
  2. Navigate to Integrations > Integrations. The Integrations screen appears.

  3. Click the Import button. The Import modal appears.

  4. Complete the fields and upload the .json file.


  5. Click the Add button. This will fetch all workflow actions from the .json file.
  6. Click the link in the Name column for the integration that you have created. The Integration builder screen appears.


  7. On the Builder tab, update the details for each action created as part of the import.
  8. Click the Edit button for the Download File(s) from FTP Generic action.
  9. Select the appropriate account credentials for SFTP and the path from where the file has to be downloaded.


  10. Click the Edit button for the Https POST request Generic action and update the Schedule Service credential details.
  11. Click the Activate button to activate the workflow.
Note:

For more information, see Managing Custom Integration Workflows.

To verify the workflow

Note:

Ensure that the generated client credentials and the Schedule Service credential details are valid for the user and have been tested.

You can call the workflow from an external system to test the webhook. These instructions use Postman as a proxy system to demonstrate this verification.

  1. Click the Copy button close to the HTTP POST URL field to copy the webhook URL that was generated in the Webhook action block during the webhook trigger creation.


  2. Paste the webhook URL in Postman or any other testing platform to trigger the workflow.


  3. Ensure a valid token is used as Bearer Token in the API call.


  4. If the call is successful, A 202 HTTP response status code will be shown.


  5. Navigate to the Integration builder screen again. Click on the Logs tab and check if it has completed.


    The Success status confirms that the import has triggered correctly.



    In the event of a failure, you can select Reprocess message or View Log(s) to inspect the issue for debugging purposes.

    Note:

    For more information, see Using Integration Logs.

  6. Click the link in the Event ID column for the event that you want to check for more details. You can also view import statuses in Global Settings. Click the gear icon in the upper right-hand corner to access Global Settings and navigate to Data import > Bulk import. The Bulk import screen appears with the list of imports.

Note:

For more information, see Integrating with Webhooks.