High Volume Data Processing

User data first enters the system via a gateway, is processed at compute nodes, then caches and messages brokers before the data is stored in your tenant's database.

As a users data transverses the system, it is processed by a number of services: Gateway, Application Pods, Message Broker and Caches, all of which are shared across environments. The sharing of resources can lead to a noisy-neighbor effect as described in the What is multitenancy? article under Additional Resources, where one tenant can consume a large share of resources, impacting the performance of all other tenants.

The message broker used by OneTrust provides a mechanism to decouple the processing of application data asynchronously. As message brokers are one of first shared resources, however, this often becomes the first place where a bottleneck is experienced when a tenant starts publishing data at an extraordinarily high rate.

In order to process data efficiently and quickly for all tenants, a throttling capability has been added to the platform. The mechanism used for throttling calculates the request rate and if it is above the preconfigured threshold, the messages are held in a tenant-specific Waiting Room. These messages are then delivered to the customer for processing after a delay, which achieves the primary goal of load-leveling under surge. This is similar to the concept of a Waiting Room when waiting for a service. The high-rate environment's data is sent to the server's specific Waiting Room where a delay is calculated based on the amount of data seen. When the delay time expires, the data is released to the message broker from the Waiting Room.

• The algorithm keeps count of the number of messages per time period, per tenant, and per message flow in the message broker.

  Waiting Room

  If this is above the configured threshold value,  the message is diverted to the.

• Each message has a fixed delay compared to previous message publish time to the main message flow from the waiting room.

• Once the Delay Time for a message in the Waiting Room is met, the message is diverted back to the message flow in the Message Broker and follows the regular processing logic.

• Once the incoming message rate for the high-rate tenant drops back below the configured threshold value, the Delay Time is reduced. Eventually, when the Delay Time drops back to zero, the messages are no longer sent to the Waiting Room and are no longer subject to delays.

• What is multitenancy? (Cloudflare)

• Scalability and performance targets for Queue Storage (Azure)

• Storage queues and Service Bus queues - compared and contrasted (Azure)