Auto Scaling of Broadcom TDM Masking on Kubernetes

Overview

Broadcom Test Data Manager (TDM) offers advanced Al data profiling algorithms for automatic discovery of potentially sensitive information that is stored enterprise wide. TDM enables organizations to protect and manage large, complex data sets efficiently—even as they scale—while maintaining data integrity and preserving complex relationships. The result is high-quality test data that mirrors production without exposing sensitive content. As customer data volumes grow exponentially, TDM users increasingly depend on scalable, on-demand compute resources to meet performance and compliance demands.

Orchestration platforms such as Kubernetes help customers reduce costs by enabling on-demand, scalable compute resources. Customers can manually scale out and scale in their Kubernetes compute resources as needed.

Scaling a Kubernetes deployment enables dynamic adjustment of application instances to meet varying demand, ensuring optimal performance and resource utilization.

Historically, scaling of TDM Kubernetes masking containers was handled manually, resulting in increased operational overhead and slower response times.

By implementing auto-scaling, we reduce infrastructure and labor costs while boosting agility—enabling faster responses to market shifts and customer demands, and ultimately strengthening our competitive edge.

Auto-scaling is the process of automatically adjusting compute resources to meet a system's performance requirements. As workloads grow, systems require additional resources to sustain performance and handle increasing demand. Conversely, when demand decreases, an automatic scaling service can reduce costs by scaling in unutilized compute resources.

Auto-scaling eliminates the need for manual human intervention in the scaling process, reducing the probability of incorrect resource allocation and saving cost by automatically allocating the exact compute power that is required.

To deliver a truly scalable and cost-efficient masking solution, TDM supports automated scaling of masking pods. This includes the ability to seamlessly scale out by monitoring demand and launching additional pods as needed, and to scale in by identifying and shutting down idle pods.

Such automation enhances operational efficiency, optimizes resource usage, and ensures the platform can adapt quickly to changing workloads.

In this post, we will show you how you can use Kubernetes Event Driven Autoscaling (KEDA) to easily auto-scale the TDM masking service based on your own masking workloads.

KEDA is a Kubernetes-based, Event-Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on custom application metrics that are external to Kubernetes.

Prerequisites

You will need the following to complete the steps in this post:

• Kubernetes cluster
• helm
• kubectl

The following diagram shows the complete setup that we will walk through in this post:

TDM masking auto-scaling setup

Step 1: Before you begin TDM Kubernetes deployment

Before you deploy TDM in Kubernetes, ensure that these prerequisites are met.

Step 2: Install KEDA on the TDM Kubernetes cluster

KEDA should be installed and available on the Kubernetes cluster. TDM uses KEDA for the automatic scaling of the masking service pods. For more information on KEDA deployment, see KEDA Documentation.

You can use the following Helm commands to install KEDA:

helm repo add kedacore https://kedacore.github.io/charts

helm repo update

helm install keda kedacore/keda --namespace keda --create-namespace

Step 3: Configure values.yaml file for auto-scaling

You should configure the values of TDM Helm Chart according to your Kubernetes system setup in the values.yaml file.

After configuring, use the updated values.yaml file for the TDM deployment on your Kubernetes cluster using Helm Chart.

Please verify that autoscaling.scaledObject.enabled helm value is true.

For this simple demonstration, please set the masking.service.poolSize helm value to “1”.

Step 4: Deploy TDM on Kubernetes using Helm Chart

After you complete the pre-deployment tasks, you can deploy TDM on the Kubernetes cluster using the Helm Chart following these steps.

Auto-scaling in action

The external KEDA ScaledObject is now periodically polling the TDM Portal web service (over gRPC protocol) for the value of the "numberOfActiveMaskingTasks" metric. Based on the value and the target size (masking.service.poolSize helm value) for this metric, KEDA is scaling the TDM masking deployment.

The default value of the masking.service.poolSize helm value is "4". That is, each masking pod should not run more than "4" masking tasks.

For this simple demonstration, we will set the masking.service.poolSize helm value to “1”.

For example, when the "numberOfActiveMaskingTasks" metric is "5", KEDA should automatically scale the masking deployment to run (5 / 1 =) "5" masking pods.

The default polling interval (autoscaling.scaledObject.pollingInterval helm value) is set to 30 seconds in helm values.yaml.

The Kubernetes logs of the keda-operator pod (in the keda namespace) should show that the “masking” ScaledObject in the “tdm” namespace has detected the “masking” deployment in the “tdm” namespace:

The TDM Portal web service is splitting each masking job to one or more masking tasks. Each masking task is executed by a dedicated FastDataMasker process (running concurrently inside the masking pod). The default value of the tdmweb.TDMMaskingService.tableTaskRowThreshold property is 1000000. That is, when running a masking job on multiple database tables, every database table that has more than 1,000,000 rows will be assigned to its own dedicated masking task (within this masking job).

Let’s put some load on the TDM masking service by running a masking job of 5 large (>1,000,000 rows) SQL database tables with masking.service.poolSize set to 1.

You may use the following SQL commands for creating 5 large (>1,000,000 rows) SQL database tables (or use your own large database tables).

Immediately after the status of this masking job is RUNNING, TDM will respond to KEDA with the current value of the "numberOfActiveMaskingTasks" metric. The reported value should reflect the current number of active masking tasks. If the masking job is masking 5 large database tables, then the reported metric value is ‘5’.

If the masking.service.poolSize helm value is set to “1”, then the reported target-size is ‘1’.

You may check the TDMMaskingService.tdmweb.log log file (in the $TDM_HOME/logs/ folder inside any of the pods) for the following log message:

So since the reported "numberOfActiveMaskingTasks" is "5", KEDA should now automatically scale the masking deployment to run (5 / 1 =) "5" masking pods.

You may now run the following commands and watch KEDA auto-scaling in action:

You may see KEDA auto-scaling in action by watching the number of TDM masking pods automatically scaling out:

Once each of the masking tasks is completed, the reported value for the “numberOfActiveMaskingTasks” metric will gradually decrease from ‘5’ back to ‘0’. KEDA will now gradually automatically scale in the number of TDM masking pods.

This example is scaling the number of masking pods (replicas) based on the value of an application TDM metric in real time.

The TDM Portal pod is running a gRPC server that is responding to the periodic getMetrics requests that are generated by the external KEDA ScaledObject.

The external KEDA ScaledObject is dividing the returned value of the “numberOfActiveMaskingTasks” metric by its target size (masking.service.poolSize helm value) and sets the number of masking pods (replicas) accordingly.

For example, metricValue: "5" and targetSize: "1" will set the masking replicas to 5 ( "5" masking tasks require 5 concurrently running masking pods). Each masking pod should not run more than "1" (targetSize) masking tasks.

Clean up

Use the following commands to delete resources created during this post:

helm uninstall tdm -n tdm

Helm uninstall tdmpvc -n tdm

helm uninstall tdmzk -n tdm

helm uninstall keda -n keda

kubectl delete storageclass tdm

kubectl delete namespace tdm

kubectl delete namespace keda

References

• The documentation of Broadcom TDM Helm solution is here: Deploy TDM using Helm Chart in Kubernetes
• The documentation of the KEDA external scalers is here: External scalers
• For more information, refer to KEDA.