Better Together: Optimize Test Data Management for Distributed Applications with Service Virtualization

Introduction 

Recently, my Broadcom colleague Sankar Natarajan wrote an excellent blog: Turbo-Charging Your Virtual Services by Making Them Data Driven. Sankar’s blog describes how Service Virtualization (SV) can become more powerful by enhancing virtual services with additional data.

This blog on the other hand discusses how we can optimize Test Data Management (TDM) for distributed applications in conjunction with use of service virtualization. 

This integrated approach provides multiple benefits, such as simplification of test data management efforts leading to cost and time savings. In addition, it provides all the usual benefits of service virtualization.    

Key challenges associated with implementing TDM alone for distributed applications

In a previous blog I wrote on microservices testing, I discussed challenges posed for TDM by distributed applications, and why Service Virtualization plays a key role in addressing those challenges. 

Some of the key challenges we see with implementing TDM alone (without SV) include the following: 

1. Test data management effort for dependent applications/components consumes a significant amount of time and effort.
   
   Testing an application with dependent applications/components requires that we provision and deploy test data for the dependent applications as well as provision the test data for the application under test (AUT), The need to provision all the data scenarios is amplified during  system or integration tests which cross application boundaries.
   
   For example, consider the scenario (see Figure below) where the AUT depends on two other applications. Testing the AUT requires that we set up test environments for both of these dependent services – along with their respective test data. This means that the testing team responsible for testing the AUT also has to spend time and effort setting up the test data for the dependent services. This adds to the TDM efforts, and slows down the testing of the AUT/CUT.
   
2. Testing with unavailable dependencies cannot be solved by TDM alone.
   
   This is a frequent problem with distributed applications where the application or component under test  has dependencies on other systems  that are not available for various reasons. For example, the dependent system may not exist, is down for maintenance,  is not easily accessible (such as mainframe systems), or is expensive to access (such as metered third party services). This impedes testing of the AUT/CUT and cannot be solved by use of TDM alone.
3. It is difficult to scale use of classic TDM approaches for hundreds of microservices. 
   
   As a corollary of point 1, it becomes more difficult to scale  using TDM when we have hundreds of microservices with a complex dependency map. See Figure below. This slows down the agility of micro-services delivery.
   
4. It is more challenging to support Continuous Integration(CI)/Continuous Delivery (CD) lifecycle automation with TDM alone. 
   
   As discussed in my previous blog on Continuous Test Data Management, enterprises are challenged to minimize Lead Time for Changes (LTC). The greater the TDM burden for testing in the CD part of the lifecycle, the greater the impediment to Lead Time for Change. Even with highly automated TDM processes,  the many required TDM activities can still impact  the lead time for change, and is often cited as the biggest bottleneck in the CD process. See figure below.
   
5. Miss out on other benefits of using Service Virtualization.
   
   Using TDM alone obviously forfeits all the benefits from using Service Virtualization, such as faster test environment setup, reduction of infrastructure costs, shift left of performance testing, etc.

Key use cases and benefits for using TDM with SV

In this section we will discuss the key uses for augmenting the use of TDM with SV.

1. Making testing possible with unavailable dependencies

This is one of the most important use cases for use of SV that cannot be solved by using TDM alone, since it is practically impossible to fully test the AUT/CUT that has dependent applications that are unavailable.

SV helps to solve this problem by creating virtual services that stand-in for the unavailable dependent applications (see Figure below). Such virtual services may either be created synthetically (for example from the API specification of an unavailable service using synthetic request-response pairs), or may be recorded from an existing application.

Service Virtualization provides many ways to create simple test data for such virtual services (see the next section) – especially for tests in the CI process. If needed, TDM may be used to create robust test data which can then be injected into the virtual service. TDM is also useful to synchronize test data across multiple dependent services, especially for integration/system tests in the CD process.

2. SV reduces the TDM burden for dependent applications

SV provides a variety of mechanisms to create test data for virtual services (that stand in for dependent applications) that are less burdensome than creating test data using traditional TDM approaches. These include:

• Test data from service recordings. Virtual services are created with data inferred from request response pairs that were recorded. SV supports desensitization of such data if needed. The virtual services can also be deployed in Learning Mode that allows such a service to continuously add new functionality (and underlying data) by connecting it to an existing (real) service.
• Magic strings and dates. These enable virtual services to dynamically return meaningful results for request parameters that were never recorded.
• Test data from production logs (or other sources). Virtual service test data can be created from RR-pairs recorded from production environment (and desensitized if needed), or by using a synthetic data generator.

All of these approaches provide a more efficient means to generate test data for dependent applications than using TDM alone – especially for shift-left tests in the CI process. For example, creating a data-rich virtual service typically takes a few minutes, while generating test data sets for dependent applications may take hours (depending on the complexity of the application). In addition, SV provides predictable responses to same request so not have to reset data (between test runs) and can also be used across teams.

Reducing the TDM burden for dependent apps helps focus our TDM efforts on the AUT/CUT, thereby improving test coverage and quality of the AUT.

As before, TDM can be used to inject additional robust data into the virtual services, as well as synchronize of the test data across multiple services. See Figure below.

3. SV enables scaling of TDM for microservices applications

In the section above, we discussed how SV helps to reduce the TDM burden on dependent apps. As the number of dependencies increase, it becomes more and more difficult to scale the use of traditional TDM techniques. This becomes a key issue in typical microservices architecture -- with hundreds of components with complex dependencies between each other – where agility of delivery is paramount.

SV is a natural fit in the context of development and testing of microservices applications. When coupled with SV, TDM can more easily support the various testing needs of micro-services – unit/component testing, API testing, contract testing, x-services systems testing etc. This integration approach for test data management for microservices testing is described in detail in my earlier blog, so I won’t repeat it here.

4. Use of SV with TDM simplifies test environment management for distributed applications

Provisioning all the dependent applications (and their accompanying test data) is one of the key challenges in setting up test environments in the CI/CD pipeline. Depending on the size and complexity of these applications, correct test environment setup is often onerous and prone to errors. In addition, deploying large applications consumes significant infrastructure resources despite being able  to bede-provisioned on demand. High environment setup time often has an adverse impact on lead time for change.

Use of SV as stand-in for these dependent applications helps to significantly simplify test environments and their setup. Virtual services are a small fraction of the size of applications, and can be packaged into containers (or VMs) for easy deployment.

To adequately meet the needs of different types of tests along the CI/CD lifecycle, the same virtual services can be progressively enriched with robust (and realistic) test data as we move from left to right along the CI/CD pipeline. See Figure below.

5. Better support for DevOps lifecycle automation in CI/CD pipelines

By virtue of the fact that data rich virtual services are easier to deploy compared to the full applications they represent in the CI/CD lifecycle, allows us to better automate the pipeline processes. Virtual services provisioning and deployment can be easily and efficiently automated using common CI/CD pipeline management tools – compared to that for real applications, since complex applications often need customized approaches. SV also provides rich APIs for integration with CI/CD engines. This helps to significantly reduce Lead Time for Change.

6. TDM enables better data-rich virtual services

We mentioned data driven virtual services in the introduction. One of the best ways to create data rich virtual services is by using TDM to generate the data. TDM provides the best mechanism to create robust test data. Data can be generated synthetically or by sub-setting/masking from production, or a combination of both. In addition, TDM can also help to synchronize data across multiple virtual services and test data stores automatically and efficiently.

Cases in Point

One of our customers in the financial services industry used SV and TDM together and achieved significant savings. Injecting synchronized test data into virtual services for the dependent applications and test data store (for the AUT) manually proved both slow and erroneous. Using TDM, they were able to not only automate the data generation, but also automate the data injection. This reduced their test data management effort by > 85% and made the process error free.

Other customers have similarly used SV along with TDM for testing against mainframe systems. Mainframe test environments (and their corresponding test data) are difficult and expensive to set up, often resulting in weeks of delays in LTC. Using SV to virtualize the mainframe and injecting test data (from TDM) into the virtual service significantly speeds up the process and cuts down LTC by over 80%.

How do we go about integrating SV into TDM implementations

The following figure summarizes how you can start to integrate use of SV into your TDM implementations.

The key steps of this process includes:

1. Identify and virtualize critical dependencies. Start by focusing on the AUT dependencies whose test environments take the most time (or are error prone) to configure, provision and deploy – both from an application complexity and test data perspective. Virtualize those dependencies, and replace those dependencies with virtual services.
2. Integrate TDM with SV. Start by leveraging the built in test data management capabilities of SV (as described in Section III.2 above) for populating the test data in the virtual services – specially for the CI tests. For the CD tests, progressively enhance these virtual services with additional test data from TDM by leveraging the built-in integration between SV and TDM or the Service Virtualization data-driven capability.
3. Optimize use of test data by leveraging change impact techniques. To further optimize your TDM efforts, focus on execution of tests that have been impacted by changes (either in code or requirements). And update test data (for the AUT and dependent applications) only as and when needed. See my prior blog on the different types of change impact analysis techniques and how they can be leveraged for TDM.
4. Integrate with CI/CD engines. Arguably, this could be Step 3 in your process , especially if you have a well-defined CI/CD engine and focus on reducing LTC.  Regardless, focus on integrating your TDM+SV implementations with your CI/CD process/tools, and adopt the Continuous TDM practices as described in my prior blog.

Summary and Conclusion

Extending Test Data Manager practices with Service Virtualization practices,   can yield significant benefits including decrease in effort, reduction of errors, decrease in Lead Time for Change, and predictability of responses.

TDM and SV complement each other – and hence better together. This is true especially in the context of modern applications such as microservices and when leveraging cloud architectures where the focus is on automation and agility.