Continuous Test Data Management for Microservices, Part 1: Key Approaches

Introduction 

In my previous blog posts on test data management (TDM), I have discussed principles for continuous TDM and outlined how TDM for microservices-based applications differs from traditional applications. 

In this blog post, the first in a two-part series, I will combine both of these concepts to discuss key approaches for applying continuous TDM to microservices. In my second post, Continuous Test Data Management for Microservices, Part 2, I detail the key steps for applying continuous TDM in various phases of the delivery lifecycle.

Revisiting the Continuous TDM Approach in the Context of Microservices Applications

As I have discussed in my previous blog posts, applying TDM to microservices is quite challenging. This is due to the fact that an application may have many services, each with its own underlying diverse data store. Also, there can be intricate dependencies between these services (see the “spaghetti architecture” depicted below). 

For these systems, TDM for end-to-end system tests can be quite complex. However, it lends itself very well to the continuous TDM approach. As part of this approach, it is key to align TDM with the test pyramid concept (see figure below).

Let’s look at the TDM approaches for tests in the various layers of the pyramid. 

1. TDM Approach for Supporting Microservices Unit Tests 

Unit tests test the code within the microservice and at the lowest level of granularity. This is typically at a function or method level within a class or object. This is no different than how we do unit testing for other types of applications. Most test data for such tests should be synthetic. Such data is typically created by the developer or software development engineer in test (SDET), who uses “as-code” algorithmic techniques, such as combinatorial. Through this approach, teams can establish a high level of test data coverage. While running unit tests, we recommend that all dependencies outside the component (or even the function being tested) are stubbed out using mocks or virtual services. See figure below. 

2. TDM Approach for Supporting Microservices Component or API Tests

This step is key for TDM of microservices, since the other tests in the stack depend on it.  In these tests, we prepare the test data for testing the microservice or component as a whole via its API.

There are various ways of doing this (see figure below), depending on the context: 

1. Generate simple synthetic test data based on the API specs. This is typically used for property-based testing or unit testing of the API.
2. Generate more robust synthetic test data from API models, for example, by using a test modeling tool like Broadcom Agile Requirements Designer. This enables us to do more rigorous API testing, for example for regression tests.
3. Generate test data by traffic sniffing a production instance of the service, for example, by using a tool like Wireshark. This helps us create more production-like data. This approach is very useful if for some reason it isn’t possible to take a subset of data from production instances. 
4. Generate test data by sub-setting and masking test data from a production instance of the service, or by using data virtualization. Note that many microservice architectures do not allow direct access to the data store, so we may need special data access APIs to create such test data.  

Regardless of the approach we take, our view is that in most cases (especially approaches b through d above) test data fabrication for a microservice must be prepared by the developer or producer of the microservice, and made available as part of service definition. Specifically, additional APIs should be provided to set up the test data for that component. This is necessary to allow for data encapsulation within a microservice. It is also required because different microservices may have various types of data stores, often with no direct access to the data. 

This also allows the TDM of microservices applications to re-use test data, which enables teams to scale tests at higher layers of the pyramid. For example, a system or end-to-end test may span hundreds of microservices, with each having its own unique encapsulated data storage. It would be very difficult to build test data for tests that span different microservices using traditional approaches.   

Again, for a single component API test, it is recommended that all dependencies from the component be virtualized to reduce the TDM burden placed on dependent systems. See figure below. 

3. TDM Approach for Supporting Microservices Integration and Contract Tests

These tests validate the interaction between microservices based on behaviors defined in their API specifications (see figure below). 

The TDM principles used for such testing are generally the same as for the process for API testing described previously. The process goes as follows (see figure below): 

1. For contract definition, we recommend using synthetic test data, for example, based on the API specs, to define the tests for the provider component. 
2. The validated contract should be a recorded virtual service based on the provider service. This virtual service can then be used for consumer tests. Note that in this case, a virtual service recording forms the basis of the test data for the consumer test. 

4. TDM Approach for Supporting a Cross-service System Test or Transaction Test at the API Level 

In this type of test, we have to support a chain of API calls across multiple services. For example, this type of test may involve invoking services A, B, and C in succession (see figure below).

The TDM approach for supporting this type of test is essentially the same as that for a single API test described above—except that we need to set up the test data for each of the services involved in the transaction, for example, for services A, B, C, and D. 

However, an additional complexity is that we also need to ensure that the test data setup for each of these services (and the underlying services they depend on) are aligned, so the test can be successfully executed. Data synchronization across microservices is largely a data management issue, not specific to TDM per-se, so we need to ensure that our microservices architecture sufficiently addresses this requirement. There are many approaches available to address this (see here for one example). 

Assuming data synchronization between microservices is in place, we recommend the following approaches to make test management easier: 

1. As mentioned before, use model-based testing to describe the cross-service system tests. This allows us to specify test data constraints for the test uniformly across affected services, so that that initial setup of test data is correct. This is done using the test data setup APIs we discussed above.
2. Since setting up test data definition across services is more time consuming, we recommend minimizing the number of cross-service tests, based on change impact testing. We run transaction tests only if the transaction, or any of the underlying components of the transaction, have changed. Again, this is a key principle of continuous testing that’s aligned with the test pyramid. 
3. If there have been no changes to a participating component or underlying sub-component, we recommend using a virtual service representation of that component, for example, services C and D referenced above. This will help to further reduce the TDM burden for that component. 

5. TDM Approach for Supporting End-to-End Business Process or User Acceptance Tests 

The TDM approach for these tests is similar to that for system tests described above, since user actions map to underlying API calls. Such tests are likely to span more components (see figure below). 

Many customers prefer to use real components, rather than virtual services, for user acceptance testing, which means that the TDM burden can be significant. As before, the key to reducing TDM complexity for such tests is to reduce the number of tests to the bare minimum, using techniques like change-impact testing, which was discussed above. We also recommend you use the change-impact approach to decide whether to use real components or their virtual services counterparts. If a set of components has changed as part of the release or deployment, it makes sense to use the actual components. However, if any dependent components are unchanged, and their test data has not been refreshed or is not readily available, then virtual services can be considered. 

Conclusion

In this post, I’ve detailed approaches for applying TDM for microservices testing in the various layers of the pyramid. In my next post, Continuous Test Data Management for Microservices, Part 2, I’ll examine the key steps for applying continuous TDM across the delivery lifecycle.