Revolutionizing TDM Synthetic Data Generation: Introducing Data Assistant Plus

Synthetic data generation

Synthetic data is fabricated but realistic data created using rules, algorithms, or models to mimic the structure and characteristics of real-world data, without containing any actual sensitive or personally identifiable information (PII). Synthetic data generation in Broadcom Test Data Manager (TDM) works by creating structured, rule-based test data that mimics the shape and logic of real production data, without containing any actual user or business information. The process begins with defining the data model or schema of the application database being tested, such as customer records, transactions, or policy details. This schema includes field types, constraints, relationships, and dependencies. One of the key steps in TDM is PII scanning, which enables users to identify and tag columns containing sensitive data. This ensures that PII is properly flagged and relevant custom rules are applied to it.  Based on this, TDM tools apply data generation rules, business logic, and validation constraints to produce meaningful, coherent, and usable test data.

For example, if a customer’s age must be greater than 18 to open an account, the generator enforces this rule while producing the test data. Additionally, we can help maintain data integrity across databases, such as account numbers in SQL Server and Oracle. TDM's synthetic data feature can simulate real-world distributions (like 80% of users being active and 20% inactive), generate edge cases (for example, max-length names or future-dated transactions), and include negative test data for robustness.

The generated synthetic data can be injected into test environments on demand, integrated with CI/CD pipelines, and used across functional, performance, and security testing. Because the data is not derived from production systems, it eliminates the risk of data breaches and simplifies compliance with privacy regulations like GDPR or HIPAA. Overall, synthetic data generation allows teams to create clean, controlled, and secure test datasets that accelerate development, improve quality, and ensure that testing is both comprehensive and legally safe.

Introducing Data Assistant Plus in Broadcom TDM

Test data management has long been a critical, yet often challenging, aspect of the software development lifecycle. Generating realistic, consistent, and compliant test data can be a significant bottleneck, particularly for complex applications. We are excited to announce a new capability in Broadcom’s TDM that aims to transform this process: Data Assistant Plus, which is powered by a heuristic-based model.

TDM Data Assistant Plus is designed to address key pain points in traditional synthetic data generation and data management. Let's delve into the challenges it solves and the innovative features it brings to the table.

Challenge No. 1: The burden of the synthetic data generation rule setup

Traditionally, setting up rules for synthetic data generation has been a laborious and time-consuming task. Imagine needing to define rules for each column across numerous tables—a process that can consume significant development time.

TDM Data Assistant Plus tackles this head on. We've significantly automated and accelerated this process by adopting a heuristic model. This intelligent approach allows for a high degree of accuracy in deriving appropriate data generation rules for tables and columns. The result? The ability to create consistent synthetic data while meticulously maintaining crucial key relationships (primary keys and foreign keys), ensuring data integrity.

Challenge No. 2: Crafting UI-workable scenarios with precision

Consider a scenario where a user interface (UI) requires specific data to function correctly. For instance, out of 1,000 tables, the first table might need one row, table two might need 150 rows, and other tables may require varying numbers of rows to make the UI work as expected. Manually ensuring the right amount of data in each table for a specific UI scenario is incredibly complex and error prone.

TDM Data Assistant Plus, with its innovative data clone-based approach, elegantly solves this. It can intelligently select a specific table and column for cloning, then generate data for that table with appropriate rules. This capability empowers you to overcome the challenge of precisely populating tables to ensure your UI functions seamlessly.

Challenges No. 3 and No. 4: Streamlining data cloning and slicing

Beyond synthetic data generation, TDM Data Assistant Plus extends its capabilities to data cloning and slicing. We have taken data cloning a step further by making it accessible via the TDM Portal, streamlining the process significantly. Data slicing refers to the process of creating smaller, more manageable subsets of production data for testing purposes. Working with a smaller subset of data, rather than the entire production database, significantly reduces storage requirements and the time needed to refresh data for testing.

Furthermore, using the Clone-based approach, data subsetting can also be performed directly from the TDM Portal. This addresses a traditional challenge: running a subset generator from the portal can be incredibly expensive and time-consuming, especially when dealing with scenarios involving 15-20 levels of joins. TDM Data Assistant Plus mitigates this by offering a more efficient and less resource-intensive method.

Introducing CRAWLER: Navigating complex data models

To further enhance data management for highly intricate systems, we've integrated CRAWLER into Data Assistant Plus. While not entirely new to TDM (it's also present in GTSubset and Javellen), its inclusion here is pivotal.

CRAWLER is designed to excel in working with complex data models, particularly those with 8-15 levels of joins. It helps in understanding and extracting data from these deeply intertwined structures. Importantly, using Data Assistant Plus with CRAWLER allows these operations to be performed conveniently via the TDM Portal.

Data Assistant Plus Templates: Simplify and standardize

TDM Data Assistant Plus leverages templates to further simplify and standardize data generation and management. We offer two primary types:

• Datatype-based template: This template considers the underlying data types, providing a foundation for consistent data generation.
  • Clone: For creating copies of existing data.
  • CloneEncrypt: For secure cloning with encryption.
  • CRAWLER: For navigating complex data models.
• Tag-based template: This allows for more flexible and business-oriented tagging of data elements.

It's important to differentiate between Clone and CRAWLER. While Clone focuses on replicating data, CRAWLER is specifically employed when dealing with highly complex data models that require deep traversal and understanding of relationships.

Conclusion

TDM Data Assistant Plus marks a significant leap forward in test data management. By leveraging a heuristic-based model, offering intelligent cloning and subsetting capabilities, and integrating the powerful CRAWLER functionality, it empowers teams to generate high-quality, relevant test data with unprecedented efficiency and accuracy. This translates to faster development cycles, more robust testing, and ultimately, higher quality software.

References

https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-11/provisioning-test-data/generate-synthetic-test-data.html

https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-11/provisioning-test-data/data-assistant-templates.html